2024-03-18T15:53:31.426 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T15:53:31.426 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908 --name pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --description fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/orig.config.yaml 2024-03-18T15:53:31.471 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T15:53:31.557 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908 branch: wip-batrick-testing-20240314.183243 description: fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} email: pdonnell@redhat.com first_in_suite: false job_id: '7608908' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mds beacon grace: 60 mds heartbeat grace: 60 ms die on skipped message: false osd heartbeat grace: 60 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds valgrind exit: true mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon osd crush smoke test: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - out of quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow request - SLOW_OPS - MON_DOWN sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes watchdog: daemon_restart: normal ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: valgrind: - --tool=memcheck - --leak-check=full - --show-reachable=yes check-counter: counters: mds: - mds.exported - mds.imported install: ceph: debuginfo: true flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCIVywnGKMz4dbv6DuAZKnwUEr2ps9arusWyrnGMF6hbhtInfttAU9qNgdINIypmvSPQBuV58PzDs5kqECCYahA= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUN4kDHCkrh0k3Tv2BDt0sRWbFTJMwli02mlbrynLiEHbZLewUIAqhOS4zfPYrE85fDkkH6Malb9EVm7+BSETI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_14:19:46 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T15:53:31.557 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa; will attempt to use it 2024-03-18T15:53:31.557 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks 2024-03-18T15:53:31.558 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T15:53:31.559 INFO:teuthology.task.internal:Checking packages... 2024-03-18T15:53:31.583 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-18T15:53:31.583 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T15:53:31.583 INFO:teuthology.packaging:ref: None 2024-03-18T15:53:31.583 INFO:teuthology.packaging:tag: None 2024-03-18T15:53:31.583 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-18T15:53:31.583 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:53:31.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-18T15:53:31.798 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-18T15:53:31.799 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T15:53:31.806 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T15:53:31.806 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T15:53:31.836 INFO:teuthology.task.internal:Saving configuration 2024-03-18T15:53:31.853 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T15:53:31.859 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T15:53:31.883 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:48:14.324969', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCIVywnGKMz4dbv6DuAZKnwUEr2ps9arusWyrnGMF6hbhtInfttAU9qNgdINIypmvSPQBuV58PzDs5kqECCYahA='} 2024-03-18T15:53:31.904 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:48:14.326898', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUN4kDHCkrh0k3Tv2BDt0sRWbFTJMwli02mlbrynLiEHbZLewUIAqhOS4zfPYrE85fDkkH6Malb9EVm7+BSETI='} 2024-03-18T15:53:31.904 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T15:53:31.910 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-18T15:53:31.910 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-18T15:53:31.910 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T15:53:31.993 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4252df5160>, signals=[15]) 2024-03-18T15:53:31.993 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T15:53:32.000 INFO:teuthology.task.internal:Opening connections... 2024-03-18T15:53:32.000 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T15:53:32.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:53:32.089 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T15:53:32.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:53:32.177 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T15:53:32.183 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-03-18T15:53:32.200 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-03-18T15:53:32.200 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-03-18T15:53:32.255 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-03-18T15:53:32.255 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T15:53:32.256 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T15:53:32.257 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-03-18T15:53:32.279 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-03-18T15:53:32.295 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-03-18T15:53:32.296 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-03-18T15:53:32.352 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-03-18T15:53:32.352 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-03-18T15:53:32.352 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T15:53:32.353 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T15:53:32.354 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-03-18T15:53:32.379 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T15:53:32.388 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T15:53:32.428 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T15:53:32.428 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-03-18T15:53:32.431 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-03-18T15:53:32.448 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T15:53:32.453 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T15:53:32.453 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-03-18T15:53:32.487 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-03-18T15:53:32.622 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T15:53:32.679 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T15:53:32.699 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-18T15:53:32.700 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-18T15:53:32.700 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-18T15:53:32.700 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T15:53:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:53:32.716 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-18T15:53:33.020 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T15:53:33.020 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3845b7ac715a3ff198582638d199a90e30e254da"... 2024-03-18T15:53:33.020 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-18T15:53:33.037 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-361.el9.x86_64 2024-03-18T15:53:33.037 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T15:53:33.037 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-18T15:53:33.038 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum upgrade -y linux-firmware 2024-03-18T15:53:35.175 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-18T15:53:36.834 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-18T15:53:37.666 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-18T15:53:39.870 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-18T15:53:43.735 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-18T15:53:45.302 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-18T15:53:47.048 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-18T15:53:52.179 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-03-18T15:53:54.513 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-18T15:53:54.525 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-18T15:53:54.525 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-18T15:53:54.525 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-18T15:53:54.525 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-03-18T15:53:54.525 INFO:teuthology.orchestra.run.smithi102.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-18T15:53:54.525 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T15:53:54.526 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-18T15:53:54.526 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-18T15:53:54.526 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 1 Package 2024-03-18T15:53:54.526 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T15:53:54.526 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 375 M 2024-03-18T15:53:54.526 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-18T15:54:10.937 INFO:teuthology.orchestra.run.smithi102.stdout:linux-firmware-20240219-143.el9.noarch.rpm 23 MB/s | 375 MB 00:16 2024-03-18T15:54:10.938 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:54:10.939 INFO:teuthology.orchestra.run.smithi102.stdout:Total 23 MB/s | 375 MB 00:16 2024-03-18T15:54:10.939 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-18T15:54:11.022 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-18T15:54:11.022 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-18T15:54:13.415 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-18T15:54:13.416 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-18T15:54:19.711 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-18T15:54:19.978 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-18T15:54:20.180 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T15:54:21.237 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T15:54:21.238 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-18T15:54:21.645 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T15:54:21.645 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T15:54:21.645 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-03-18T15:54:21.645 INFO:teuthology.orchestra.run.smithi102.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-18T15:54:21.645 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T15:54:21.645 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-18T15:54:21.949 INFO:teuthology.task.kernel:Downloading kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-18T15:54:21.950 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T15:54:22.174 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ 2024-03-18T15:54:22.174 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-18T15:54:22.176 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi102.front.sepia.ceph.com to complete... 2024-03-18T15:54:35.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18 15:54:35 URL:https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm [336075563/336075563] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-18T15:54:35.192 INFO:teuthology.orchestra.run.smithi102.stderr:FINISHED --2024-03-18 15:54:35-- 2024-03-18T15:54:35.192 INFO:teuthology.orchestra.run.smithi102.stderr:Total wall clock time: 13s 2024-03-18T15:54:35.193 INFO:teuthology.orchestra.run.smithi102.stderr:Downloaded: 1 files, 321M in 13s (25.4 MB/s) 2024-03-18T15:54:35.195 INFO:teuthology.task.kernel:Installing kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-18T15:54:35.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-18T15:54:37.117 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying... ######################################## 2024-03-18T15:54:37.165 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing... ######################################## 2024-03-18T15:54:37.243 INFO:teuthology.orchestra.run.smithi102.stdout:Updating / installing... 2024-03-18T15:54:44.755 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-6.8.0_rc7_g3845b7ac715a-1 ######################################## 2024-03-18T15:55:01.165 INFO:teuthology.orchestra.run.smithi102.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T15:55:23.517 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-18T15:55:23.517 DEBUG:teuthology.orchestra.run.smithi102:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-18T15:55:23.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a 2024-03-18T15:55:23.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/System.map 2024-03-18T15:55:23.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/config 2024-03-18T15:55:23.588 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-18T15:55:23.589 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-18T15:55:23.590 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ansi_cprng.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/anubis.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/arc4.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_memcpy.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_pq.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_tx.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_xor.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/raid6test.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authenc.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authencesn.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blake2b_generic.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_common.ko 2024-03-18T15:55:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_generic.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/camellia_generic.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast5_generic.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast6_generic.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast_common.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ccm.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cmac.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crc32_generic.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_engine.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_user.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cts.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/des_generic.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/dh_generic.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/echainiv.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/essiv.ko 2024-03-18T15:55:23.592 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/fcrypt.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/khazad.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4hc.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/md4.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/michael_mic.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcbc.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcrypt.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/rmd160.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/seed.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/serpent_generic.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tcrypt.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tea.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_common.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_generic.ko 2024-03-18T15:55:23.593 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/vmac.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/wp512.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xcbc.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xor.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xxhash_generic.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_pad.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/custom_method.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/ec_sys.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/platform_profile.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbs.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbshc.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/video.ko 2024-03-18T15:55:23.594 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/acard-ahci.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ahci_platform.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ata_generic.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/libahci_platform.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_acpi.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ali.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_amd.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_artop.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atiixp.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atp867x.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd640.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd64x.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cypress.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_efar.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt366.ko 2024-03-18T15:55:23.595 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt37x.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it8213.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it821x.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_jmicron.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_marvell.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_mpiix.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_netcell.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ninja32.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87410.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87415.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_oldpiix.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_opti.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_optidma.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pcmcia.ko 2024-03-18T15:55:23.596 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_piccolo.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_rdc.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sch.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_serverworks.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sil680.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sis.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sl82c105.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_triflex.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_via.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pdc_adma.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_inic162x.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_mv.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_nv.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_promise.ko 2024-03-18T15:55:23.597 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_qstor.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil24.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sis.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_svw.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sx4.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_uli.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_via.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_vsc.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay/ks0108.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-18T15:55:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma/bcma.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe/aoe.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/brd.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd/drbd.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/floppy.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/loop.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/nbd.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk/null_blk.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/pktcdvd.ko 2024-03-18T15:55:23.599 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/rbd.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/virtio_blk.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkfront.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram/zram.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hangcheck-timer.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/via-rng.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi 2024-03-18T15:55:23.600 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/lp.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave/mwave.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ppdev.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tlclk.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-18T15:55:23.601 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/uv_mmtimer.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/virtio_console.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp.ko 2024-03-18T15:55:23.602 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-aes.ko 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-sha.ko 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca/dca.ko 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw 2024-03-18T15:55:23.603 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/amd64_edac.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/e752x_edac.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/edac_mce_amd.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3000_edac.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3200_edac.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5100_edac.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5400_edac.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7300_edac.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7core_edac.ko 2024-03-18T15:55:23.604 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i82975x_edac.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/ie31200_edac.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/sb_edac.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/x38_edac.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/edd.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-ich.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-appleir.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-aureal.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-axff.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-dr.ko 2024-03-18T15:55:23.605 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elecom.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elo.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-emsff.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gaff.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gt683r.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gyration.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtekff.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-hyperv.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-icade.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-keytouch.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-kye.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lcpower.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-led.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lenovo.ko 2024-03-18T15:55:23.606 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-multitouch.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-ortek.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-penmount.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-petalynx.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-picolcd.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-pl.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-plantronics.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-primax.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-rmi.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-common.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-18T15:55:23.607 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-saitek.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-samsung.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sjoy.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sony.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-speedlink.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-steelseries.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sunplus.ko 2024-03-18T15:55:23.608 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tivo.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tmff.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-topseed.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-twinhan.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-uclogic.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-waltop.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-wiimote.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-xinmo.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zpff.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zydacron.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/uhid.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/wacom.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_balloon.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_utils.ko 2024-03-18T15:55:23.609 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_vmbus.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru3.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7414.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7418.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adc128d818.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1025.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1026.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1029.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1031.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm9240.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ads7828.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7410.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7411.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7462.ko 2024-03-18T15:55:23.610 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7470.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7475.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7x10.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/amc6821.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/applesmc.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asb100.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asc7621.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/atxp1.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/coretemp.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dme1737.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds1621.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds620.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc1403.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc6w201.ko 2024-03-18T15:55:23.611 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71805f.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71882fg.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f75375s.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fam15h_power.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fschmd.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g760a.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g762.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl518sm.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl520sm.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5500_temp.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5k_amb.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmaem.ko 2024-03-18T15:55:23.612 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmpex.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina209.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina2xx.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/it87.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k10temp.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k8temp.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lineage-pem.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm63.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm73.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm75.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm77.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm78.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm80.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm83.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm85.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm87.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm90.ko 2024-03-18T15:55:23.613 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm92.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm93.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95234.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95241.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95245.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc2945.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4151.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4215.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4222.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4245.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4260.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4261.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max16065.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1619.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1668.ko 2024-03-18T15:55:23.614 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max197.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6639.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6650.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6697.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/mcp3021.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6683.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775-core.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct7802.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87360.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87427.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pcf8591.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-18T15:55:23.615 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/powr1220.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5627.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5636.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht15.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht21.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/shtc1.ko 2024-03-18T15:55:23.616 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sis5595.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47b397.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m1.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m192.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/thmc50.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp102.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp103.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp401.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp421.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via-cputemp.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via686a.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt1211.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt8231.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627ehf.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627hf.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83781d.ko 2024-03-18T15:55:23.617 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83791d.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83792d.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83793.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83795.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l785ts.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l786ng.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-18T15:55:23.618 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-18T15:55:23.619 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-dev.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-mux.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-smbus.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-stub.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_core.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-18T15:55:23.620 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert 2024-03-18T15:55:23.621 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/gameport.ko 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/joydev.ko 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard 2024-03-18T15:55:23.622 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-18T15:55:23.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse 2024-03-18T15:55:23.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/appletouch.ko 2024-03-18T15:55:23.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/bcm5974.ko 2024-03-18T15:55:23.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/cyapatp.ko 2024-03-18T15:55:23.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-18T15:55:23.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/sermouse.ko 2024-03-18T15:55:23.623 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/altera_ps2.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/arc_ps2.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/serio_raw.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/sparse-keymap.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-blinkm.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lm3530.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lp3944.ko 2024-03-18T15:55:23.624 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-ss4200.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache/bcache.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-bio-prison.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache-smq.ko 2024-03-18T15:55:23.625 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-crypt.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-delay.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-flakey.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-log-userspace.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-multipath.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-queue-length.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-raid.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-round-robin.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-service-time.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-switch.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-thin-pool.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-verity.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid0.ko 2024-03-18T15:55:23.626 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid1.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid10.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid456.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptbase.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptctl.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptfc.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptlan.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptsas.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptscsih.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptspi.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_ich.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_sch.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/mfd-core.ko 2024-03-18T15:55:23.627 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/sm501.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/viperboard.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/vx855.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/wl1273-core.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds9802als.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds990x.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/bh1770glc.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710/cb710.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/at24.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/max6875.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/enclosure.ko 2024-03-18T15:55:23.628 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/hpilo.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29003.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29020.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-me.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-txe.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-18T15:55:23.629 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_7xx1.ko 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_core.ko 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tsl2550.ko 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_balloon.ko 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding/bonding.ko 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-18T15:55:23.630 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dummy.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/eql.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-18T15:55:23.631 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx 2024-03-18T15:55:23.632 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-18T15:55:23.633 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-18T15:55:23.634 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-18T15:55:23.635 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dnet.ko 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ethoc.ko 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/fealnx.ko 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e 2024-03-18T15:55:23.636 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/jme.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-18T15:55:23.637 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-18T15:55:23.638 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-18T15:55:23.639 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-18T15:55:23.640 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-18T15:55:23.641 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/geneve.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ifb.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvlan.ko 2024-03-18T15:55:23.642 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvtap.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mii.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/net_failover.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/netconsole.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/nlmon.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/amd.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-18T15:55:23.643 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm87xx.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/broadcom.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/cicada.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/davicom.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/dp83640.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/et1011c.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/icplus.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/lxt.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/marvell.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/micrel.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/national.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/phylink.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/qsemi.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/realtek.ko 2024-03-18T15:55:23.644 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/smsc.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/ste10Xp.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/vitesse.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_async.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppoe.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppox.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pptp.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slhc.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slip.ko 2024-03-18T15:55:23.645 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/sungem_phy.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tap.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_random.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tun.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/veth.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/virtio_net.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan 2024-03-18T15:55:23.646 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan/vxlan.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netfront.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb/ntb.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme-core.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_cs.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_pc.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_serial.ko 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug 2024-03-18T15:55:23.647 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/xen-pcifront.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/i82092.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/pd6729.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acerhdf.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-18T15:55:23.648 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/hdaps.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel_ips.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-18T15:55:23.649 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps_parport.ko 2024-03-18T15:55:23.650 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/pps_core.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp_kvm.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-em3027.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-18T15:55:23.651 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-max6900.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-18T15:55:23.652 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-x1205.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-9xxx.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-sas.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-xxxx.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/BusLogic.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/a100u2w.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/advansys.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/am53c974.ko 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr 2024-03-18T15:55:23.653 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/atp870u.ko 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa/bfa.ko 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ch.ko 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-18T15:55:23.654 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dc395x.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dmx3191d.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esp_scsi.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-18T15:55:23.655 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic/fnic.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hpsa.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hptiop.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hv_storvsc.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/initio.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ipr.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ips.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci/isci.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc/libfc.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas 2024-03-18T15:55:23.656 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas/libsas.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvumi.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pmcraid.ko 2024-03-18T15:55:23.657 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla1280.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/raid_class.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_debug.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ses.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/st.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/stex.ko 2024-03-18T15:55:23.658 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/virtio_scsi.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/wd719x.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb/ssb.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_file.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_iblock.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_mod.ko 2024-03-18T15:55:23.659 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_pscsi.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_user.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-18T15:55:23.660 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_gsm.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_hdlc.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/nozomi.ko 2024-03-18T15:55:23.661 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/arc_uart.ko 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio.ko 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_aec.ko 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_cif.ko 2024-03-18T15:55:23.662 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_pci_generic.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_sercos3.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-acm.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usblp.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usbtmc.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/mdc800.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/microtek.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/adutux.ko 2024-03-18T15:55:23.663 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/appledisplay.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi26.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi62.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ezusb.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/idmouse.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/iowarrior.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ldusb.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/legousbtower.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usb3503.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usblcd.ko 2024-03-18T15:55:23.664 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/uss720.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/yurex.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy/phy-generic.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/aircable.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ark3116.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ch341.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cp210x.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cyberjack.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/empeg.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-18T15:55:23.665 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_ti.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipaq.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipw.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ir-usb.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mct_u232.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7720.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7840.ko 2024-03-18T15:55:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/navman.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/omninet.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/opticon.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/option.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/oti6858.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/pl2303.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcaux.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcserial.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/quatech2.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/safe_serial.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/sierra.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ssu100.ko 2024-03-18T15:55:23.667 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/symbolserial.ko 2024-03-18T15:55:23.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-18T15:55:23.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-18T15:55:23.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_debug.ko 2024-03-18T15:55:23.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-18T15:55:23.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/visor.ko 2024-03-18T15:55:23.668 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/whiteheat.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/uas.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-karma.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-18T15:55:23.669 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/usb-storage.ko 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio.ko 2024-03-18T15:55:23.670 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_net.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_scsi.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/apple_bl.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/lcd.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_balloon.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_mmio.ko 2024-03-18T15:55:23.671 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_ring.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2482.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2490.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-18T15:55:23.672 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/wire.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/hpwdt.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/i6300esb.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-18T15:55:23.673 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ib700wdt.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ibmasr.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it87_wdt.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/machzwd.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/nv_tco.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/softdog.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-18T15:55:23.674 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/via_wdt.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/wdt_pci.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/xen_wdt.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-evtchn.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntalloc.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntdev.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-privcmd.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs 2024-03-18T15:55:23.675 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p/9p.ko 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs/affs.ko 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs/befs.ko 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/binfmt_misc.ko 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs/btrfs.ko 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles/cachefiles.ko 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph/ceph.ko 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda 2024-03-18T15:55:23.676 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda/coda.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs/cramfs.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm/dlm.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs/f2fs.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/fat.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/msdos.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/vfat.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/cuse.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/fuse.ko 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2 2024-03-18T15:55:23.677 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2/gfs2.ko 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs/hfs.ko 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus/hfsplus.ko 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs/isofs.ko 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs/jfs.ko 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd/lockd.ko 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix/minix.ko 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs 2024-03-18T15:55:23.678 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs/netfs.ko 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfs.ko 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv3.ko 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv4.ko 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/grace.ko 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/nfs_acl.ko 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd 2024-03-18T15:55:23.679 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd/nfsd.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2/nilfs2.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-celtic.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-centeuro.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-croatian.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-cyrillic.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-gaelic.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-greek.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-iceland.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-inuit.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-roman.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-romanian.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-turkish.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1250.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1251.ko 2024-03-18T15:55:23.680 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1255.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp737.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp775.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp850.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp852.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp855.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp857.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp860.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp861.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp862.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp863.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp864.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp865.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp866.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp869.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp874.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp932.ko 2024-03-18T15:55:23.681 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp936.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp949.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp950.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_euc-jp.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-1.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-13.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-14.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-15.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-2.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-3.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-4.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-5.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-6.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-7.ko 2024-03-18T15:55:23.682 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-9.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-r.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-ru.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-u.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_utf8.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-18T15:55:23.683 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs/overlay.ko 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore/ramoops.ko 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs/reiserfs.ko 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs/romfs.ko 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client/cifs.ko 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_arc4.ko 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_md4.ko 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs/squashfs.ko 2024-03-18T15:55:23.684 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv/sysv.ko 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf/udf.ko 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs/ufs.ko 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs/xfs.ko 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu/rcutorture.ko 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/torture.ko 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/asn1_encoder.ko 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc-itu-t.ko 2024-03-18T15:55:23.685 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc8.ko 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libarc4.ko 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libdes.ko 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/libcrc32c.ko 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lru_cache.ko 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4_compress.ko 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4hc_compress.ko 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math/cordic.ko 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6/raid6_pq.ko 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon 2024-03-18T15:55:23.686 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_bm.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_fsm.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_kmp.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd/zstd_compress.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm/hwpoison-inject.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/6lowpan.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_dest.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_fragment.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_hop.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_mobility.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_routing.ko 2024-03-18T15:55:23.687 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_udp.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/garp.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/mrp.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/p8022.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/psnap.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/stp.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q/8021q.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_fd.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_rdma.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_virtio.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv/batman-adv.ko 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge 2024-03-18T15:55:23.688 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/br_netfilter.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/bridge.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-18T15:55:23.689 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtables.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph/libceph.ko 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core 2024-03-18T15:55:23.690 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/failover.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/pktgen.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_diag.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv4.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv6.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver/dns_resolver.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/dsa_core.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_brcm.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_dsa.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_none.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_trailer.ko 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan 2024-03-18T15:55:23.691 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ah4.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/esp4.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/fou.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/gre.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/inet_diag.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_gre.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_tunnel.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_vti.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipcomp.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipip.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-18T15:55:23.692 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-18T15:55:23.693 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_bic.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_dctcp.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_diag.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_highspeed.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_htcp.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_hybla.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_illinois.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_lp.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_scalable.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_vegas.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_veno.ko 2024-03-18T15:55:23.694 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_westwood.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_yeah.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tunnel4.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_diag.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_tunnel.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ah6.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/esp6.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/fou6.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_tunnel.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_vti.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ipcomp6.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-18T15:55:23.695 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-18T15:55:23.696 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/sit.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/tunnel6.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key/af_key.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_core.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_eth.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip6.ko 2024-03-18T15:55:23.697 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_netlink.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ppp.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc/llc.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154/mac802154.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls/mpls_gso.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-18T15:55:23.698 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-18T15:55:23.699 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conncount.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-18T15:55:23.700 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_log_syslog.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-18T15:55:23.701 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_irc.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_sip.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_tables.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_log.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_chain_nat.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_compat.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_ct.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-18T15:55:23.702 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fib.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_hash.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_limit.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_log.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_masq.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_nat.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_numgen.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_queue.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_quota.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_redir.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject_inet.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_socket.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_tunnel.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_AUDIT.ko 2024-03-18T15:55:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CT.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_DSCP.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HL.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HMARK.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LED.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LOG.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NETMAP.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFLOG.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_RATEEST.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_SECMARK.ko 2024-03-18T15:55:23.704 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TEE.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TPROXY.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TRACE.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_addrtype.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_bpf.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cgroup.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cluster.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_comment.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connbytes.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlabel.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlimit.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connmark.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_conntrack.ko 2024-03-18T15:55:23.705 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cpu.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dccp.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_devgroup.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dscp.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ecn.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_esp.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hashlimit.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_helper.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hl.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipcomp.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_iprange.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipvs.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_l2tp.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_length.ko 2024-03-18T15:55:23.706 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_limit.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mac.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mark.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_multiport.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nat.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nfacct.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_osf.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_owner.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_physdev.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_pkttype.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_policy.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_quota.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_rateest.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_realm.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_recent.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_sctp.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_set.ko 2024-03-18T15:55:23.707 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_socket.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_state.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_statistic.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_string.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_tcpmss.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_time.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_u32.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink/netlink_diag.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh/nsh.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/openvswitch.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-geneve.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-gre.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-vxlan.ko 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet 2024-03-18T15:55:23.708 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet/af_packet_diag.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_rdma.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_tcp.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_csum.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_gact.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_mirred.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_nat.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_pedit.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_police.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_simple.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_skbedit.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_vlan.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_basic.ko 2024-03-18T15:55:23.709 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_bpf.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_flow.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_fw.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_route.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_u32.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_cmp.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_ipset.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_meta.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_nbyte.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_text.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_u32.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_choke.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_codel.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_drr.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_fq.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_gred.ko 2024-03-18T15:55:23.710 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hfsc.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hhf.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_htb.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_ingress.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio_lib.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_multiq.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_netem.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_pie.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_plug.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_prio.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_qfq.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_red.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfb.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfq.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_tbf.ko 2024-03-18T15:55:23.711 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_teql.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp_diag.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/sunrpc.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/diag.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/tipc.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix/unix_diag.ko 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock 2024-03-18T15:55:23.712 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock.ko 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys/trusted.ko 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt 2024-03-18T15:55:23.713 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib/irqbypass.ko 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias.bin 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.alias.bin 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.bin 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.modinfo 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep.bin 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.devname 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.order 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.softdep 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols.bin 2024-03-18T15:55:23.714 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-18T15:55:23.718 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:55:23.718 DEBUG:teuthology.orchestra.run.smithi102:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout:postinstall scriptlet (using /bin/sh): 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout: /usr/bin/kernel-install add 6.8.0-rc7-g3845b7ac715a /lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout:for file in vmlinuz System.map config; do 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a"; then 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout: cp "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a" 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout: fi 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout:done 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi102.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-18T15:55:23.741 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-18T15:55:23.741 INFO:teuthology.orchestra.run.smithi102.stdout:new-kernel-pkg --remove 6.8.0-rc7-g3845b7ac715a --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-g3845b7ac715a.img 2024-03-18T15:55:23.741 INFO:teuthology.orchestra.run.smithi102.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-18T15:55:23.741 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-install remove 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:55:23.741 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-03-18T15:55:23.741 INFO:teuthology.orchestra.run.smithi102.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-18T15:55:23.741 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-18T15:55:23.741 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:55:23.741 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-03-18T15:55:23.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-03-18T15:55:23.816 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-03-18T15:55:23.816 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 68.el9 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8264254 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-03-18T15:55:23.817 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T15:55:23.818 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T15:55:23.818 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T15:55:23.818 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T15:55:23.818 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-03-18T15:55:23.818 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T15:55:23.818 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T15:55:23.818 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T15:55:23.819 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:55:23.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T15:55:24.526 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-03-18T15:55:25.907 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T15:55:25.934 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-03-18T15:55:25.945 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-18T15:55:25.959 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.aF18c4Pmqe 2024-03-18T15:55:25.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aF18c4Pmqe 2024-03-18T15:55:26.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.aF18c4Pmqe 2024-03-18T15:55:26.171 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.aF18c4Pmqe is 6KB 2024-03-18T15:55:26.219 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.aF18c4Pmqe 2024-03-18T15:55:26.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T15:55:26.298 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T15:55:26.298 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T15:55:26.298 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T15:55:26.298 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a.conf 2024-03-18T15:55:26.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a 2024-03-18T15:55:26.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-03-18T15:55:56.470 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-18T15:55:56.470 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-18T15:55:56.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:56:15.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-18T15:56:24.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-18T15:56:24.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:56:27.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-18T15:56:39.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-18T15:56:39.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:57:39.074 DEBUG:teuthology.orchestra.remote:timed out 2024-03-18T15:57:54.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-18T15:57:54.076 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:57:54.451 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-18T15:57:55.109 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-18T15:57:55.109 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-g3845b7ac715a"... 2024-03-18T15:57:55.110 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-18T15:57:55.161 INFO:teuthology.orchestra.run.smithi102.stdout:6.8.0-rc7-g3845b7ac715a 2024-03-18T15:57:55.162 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-g3845b7ac715a vs 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:57:55.162 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T15:57:55.162 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T15:57:56.166 DEBUG:teuthology.parallel:result is None 2024-03-18T15:57:56.166 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T15:57:56.176 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T15:57:56.177 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T15:57:56.184 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T15:57:56.233 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T15:57:56.240 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T15:57:56.274 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T15:57:56.274 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T15:57:56.281 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T15:57:56.380 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T15:57:56.387 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T15:57:56.387 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T15:57:56.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:57:56.404 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T15:57:56.412 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T15:57:56.481 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:57:56.509 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:57:56.544 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:57:56.613 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:57:56.618 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T15:57:56.625 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T15:57:56.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T15:57:56.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T15:57:56.719 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T15:57:56.729 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T15:57:56.729 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T15:57:56.737 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T15:57:56.784 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:57:56.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:57:56.963 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:57:57.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:57:57.143 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T15:57:57.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T15:57:57.235 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:57:57.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:57:57.338 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:57:57.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:57:57.470 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T15:57:57.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T15:57:57.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-03-18T15:57:57.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-03-18T15:57:57.626 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:57:57.662 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:57:58.119 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T15:57:58.127 INFO:teuthology.task.internal:Starting timer... 2024-03-18T15:57:58.127 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T15:57:58.174 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T15:57:58.222 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-18T15:57:58.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-03-18T15:57:58.343 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-03-18T15:57:58.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-03-18T15:57:58.416 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-03-18T15:57:58.418 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T15:57:58.418 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-03-18T15:57:58.469 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-03-18T15:57:58.469 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-03-18T15:57:58.491 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-03-18T15:57:58.491 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive'} 2024-03-18T15:57:58.492 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:57:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:57:58.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:57:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:57:58.591 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T15:57:58.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-03-18T15:57:58.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-03-18T15:57:58.718 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T15:57:58.729 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T15:57:58.745 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T15:57:58.746 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T16:03:47.788 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2024-03-18T16:03:47.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-18T16:03:47.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T16:03:47.930 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-18T16:03:48.076 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-18T16:03:48.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-18T16:03:48.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T16:03:48.171 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-03-18T16:03:48.241 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-18T16:03:48.242 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T16:03:48.253 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T16:03:48.253 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T16:03:48.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:03:48.260 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T16:03:48.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:03:48.315 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T16:03:48.332 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T16:03:48.357 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T16:03:48.364 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-03-18T16:03:48.379 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-03-18T16:03:48.396 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T16:03:48.408 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T16:03:48.413 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T16:03:48.458 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:03:48.461 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:03:48.461 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-03-18T16:03:48.461 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:03:48.461 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:03:48.461 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:03:48.461 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:03:48.502 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-03-18T16:03:48.544 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-03-18T16:03:48.589 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T16:03:48.641 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T16:03:48.767 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:03:48.774 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:03:48.774 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-18T16:03:48.774 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:03:48.774 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:03:48.774 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:03:48.774 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:03:48.775 INFO:teuthology.run_tasks:Running task install... 2024-03-18T16:03:48.784 DEBUG:teuthology.task.install:project ceph 2024-03-18T16:03:48.784 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-18T16:03:48.784 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T16:03:48.784 INFO:teuthology.task.install:Using flavor: default 2024-03-18T16:03:48.790 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-18T16:03:48.790 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T16:03:48.790 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T16:03:48.791 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T16:03:48.793 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T16:03:48.793 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T16:03:49.048 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T16:03:49.048 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T16:03:49.055 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T16:03:49.056 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T16:03:49.337 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T16:03:49.337 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:03:49.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T16:03:49.405 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T16:03:49.405 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:03:49.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T16:03:49.430 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T16:03:49.430 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T16:03:49.433 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T16:03:49.433 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T16:03:49.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T16:03:49.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T16:03:49.581 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T16:03:49.647 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-03-18T16:03:49.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-03-18T16:03:49.700 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T16:03:49.786 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-03-18T16:03:49.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-03-18T16:03:49.886 INFO:teuthology.orchestra.run.smithi154.stdout:74 files removed 2024-03-18T16:03:49.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T16:03:50.154 INFO:teuthology.orchestra.run.smithi102.stdout:74 files removed 2024-03-18T16:03:50.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T16:03:51.103 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 92 kB/s | 83 kB 00:00 2024-03-18T16:03:51.675 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 84 kB/s | 83 kB 00:00 2024-03-18T16:03:51.845 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-18T16:03:52.501 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-18T16:03:52.663 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-18T16:03:53.057 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-18T16:03:53.335 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-18T16:03:53.930 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-18T16:03:54.591 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-18T16:03:55.647 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-18T16:03:56.626 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-18T16:03:59.569 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-18T16:04:03.870 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-18T16:04:05.371 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 16 kB 00:00 2024-03-18T16:04:08.664 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 21 MB 00:03 2024-03-18T16:04:14.464 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-18T16:04:17.237 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T16:04:17.239 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T16:04:17.288 INFO:teuthology.orchestra.run.smithi102.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T16:04:17.289 INFO:teuthology.orchestra.run.smithi102.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T16:04:17.291 INFO:teuthology.orchestra.run.smithi102.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T16:04:17.292 INFO:teuthology.orchestra.run.smithi102.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T16:04:17.294 INFO:teuthology.orchestra.run.smithi102.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T16:04:17.297 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T16:04:17.299 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T16:04:17.389 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout:==================================================================================================== 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout:==================================================================================================== 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 69 M 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 443 M 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.1 M 2024-03-18T16:04:17.424 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 54 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 20 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 290 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 178 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 679 M 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T16:04:17.425 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.5 M 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 109 k 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 M 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 52 M 2024-03-18T16:04:17.426 INFO:teuthology.orchestra.run.smithi102.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 615 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 770 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.1 M 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 337 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T16:04:17.427 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 516 k 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 52 M 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.3 M 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T16:04:17.428 INFO:teuthology.orchestra.run.smithi102.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T16:04:17.429 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 23 M 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T16:04:17.430 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T16:04:17.431 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T16:04:17.432 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T16:04:17.433 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T16:04:17.434 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T16:04:17.435 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T16:04:17.436 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout:==================================================================================================== 2024-03-18T16:04:17.437 INFO:teuthology.orchestra.run.smithi102.stdout:Install 172 Packages 2024-03-18T16:04:17.438 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 24 Packages 2024-03-18T16:04:17.438 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:04:17.438 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 2.2 G 2024-03-18T16:04:17.438 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-18T16:04:18.474 INFO:teuthology.orchestra.run.smithi102.stdout:(1/196): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-03-18T16:04:18.933 INFO:teuthology.orchestra.run.smithi102.stdout:(2/196): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 7.5 MB/s | 5.4 MB 00:00 2024-03-18T16:04:22.299 INFO:teuthology.orchestra.run.smithi102.stdout:(3/196): ceph-common-19.0.0-2162.g3d78f5ba.el9. 5.4 MB/s | 21 MB 00:03 2024-03-18T16:04:27.399 INFO:teuthology.orchestra.run.smithi102.stdout:(4/196): ceph-debuginfo-19.0.0-2162.g3d78f5ba.e 4.5 MB/s | 23 MB 00:05 2024-03-18T16:04:27.749 INFO:teuthology.orchestra.run.smithi102.stdout:(5/196): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.5 MB/s | 890 kB 00:00 2024-03-18T16:04:29.382 INFO:teuthology.orchestra.run.smithi102.stdout:(6/196): ceph-fuse-debuginfo-19.0.0-2162.g3d78f 3.7 MB/s | 6.1 MB 00:01 2024-03-18T16:04:29.516 INFO:teuthology.orchestra.run.smithi102.stdout:(7/196): ceph-immutable-object-cache-19.0.0-216 1.1 MB/s | 145 kB 00:00 2024-03-18T16:04:29.875 INFO:teuthology.orchestra.run.smithi102.stdout:(8/196): ceph-immutable-object-cache-debuginfo- 4.1 MB/s | 1.5 MB 00:00 2024-03-18T16:04:30.448 INFO:teuthology.orchestra.run.smithi102.stdout:(9/196): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 4.1 MB/s | 2.4 MB 00:00 2024-03-18T16:04:33.634 INFO:teuthology.orchestra.run.smithi102.stdout:(10/196): ceph-base-debuginfo-19.0.0-2162.g3d78 4.5 MB/s | 69 MB 00:15 2024-03-18T16:04:34.200 INFO:teuthology.orchestra.run.smithi102.stdout:(11/196): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x8 2.6 MB/s | 1.5 MB 00:00 2024-03-18T16:04:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:(12/196): ceph-mgr-debuginfo-19.0.0-2162.g3d78f 4.8 MB/s | 20 MB 00:04 2024-03-18T16:04:39.309 INFO:teuthology.orchestra.run.smithi102.stdout:(13/196): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x8 4.9 MB/s | 4.7 MB 00:00 2024-03-18T16:04:41.151 INFO:teuthology.orchestra.run.smithi102.stdout:(14/196): ceph-mds-debuginfo-19.0.0-2162.g3d78f 5.0 MB/s | 54 MB 00:10 2024-03-18T16:04:45.559 INFO:teuthology.orchestra.run.smithi102.stdout:(15/196): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 3.8 MB/s | 17 MB 00:04 2024-03-18T16:04:49.651 INFO:teuthology.orchestra.run.smithi102.stdout:(16/196): ceph-common-debuginfo-19.0.0-2162.g3d 14 MB/s | 443 MB 00:30 2024-03-18T16:04:50.501 INFO:teuthology.orchestra.run.smithi102.stdout:(17/196): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 12 MB/s | 9.8 MB 00:00 2024-03-18T16:04:54.901 INFO:teuthology.orchestra.run.smithi102.stdout:(18/196): ceph-mon-debuginfo-19.0.0-2162.g3d78f 6.4 MB/s | 99 MB 00:15 2024-03-18T16:04:54.951 INFO:teuthology.orchestra.run.smithi102.stdout:(19/196): ceph-selinux-19.0.0-2162.g3d78f5ba.el 516 kB/s | 25 kB 00:00 2024-03-18T16:05:08.367 INFO:teuthology.orchestra.run.smithi102.stdout:(20/196): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 3.5 MB/s | 48 MB 00:13 2024-03-18T16:05:10.909 INFO:teuthology.orchestra.run.smithi102.stdout:(21/196): ceph-radosgw-debuginfo-19.0.0-2162.g3 8.7 MB/s | 178 MB 00:20 2024-03-18T16:05:11.026 INFO:teuthology.orchestra.run.smithi102.stdout:(22/196): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 1.9 MB/s | 221 kB 00:00 2024-03-18T16:05:11.093 INFO:teuthology.orchestra.run.smithi102.stdout:(23/196): cephfs-shell-19.0.0-2162.g3d78f5ba.el 410 kB/s | 27 kB 00:00 2024-03-18T16:05:11.168 INFO:teuthology.orchestra.run.smithi102.stdout:(24/196): libcephfs-devel-19.0.0-2162.g3d78f5ba 428 kB/s | 32 kB 00:00 2024-03-18T16:05:11.352 INFO:teuthology.orchestra.run.smithi102.stdout:(25/196): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 4.0 MB/s | 747 kB 00:00 2024-03-18T16:05:11.893 INFO:teuthology.orchestra.run.smithi102.stdout:(26/196): libcephfs2-debuginfo-19.0.0-2162.g3d7 10 MB/s | 5.5 MB 00:00 2024-03-18T16:05:11.969 INFO:teuthology.orchestra.run.smithi102.stdout:(27/196): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.2 MB/s | 163 kB 00:00 2024-03-18T16:05:12.044 INFO:teuthology.orchestra.run.smithi102.stdout:(28/196): librados-devel-19.0.0-2162.g3d78f5ba. 1.7 MB/s | 127 kB 00:00 2024-03-18T16:05:12.128 INFO:teuthology.orchestra.run.smithi102.stdout:(29/196): librados-devel-debuginfo-19.0.0-2162. 1.3 MB/s | 109 kB 00:00 2024-03-18T16:05:22.236 INFO:teuthology.orchestra.run.smithi102.stdout:(30/196): librados2-debuginfo-19.0.0-2162.g3d78 4.5 MB/s | 45 MB 00:10 2024-03-18T16:05:22.319 INFO:teuthology.orchestra.run.smithi102.stdout:(31/196): libradosstriper1-19.0.0-2162.g3d78f5b 6.0 MB/s | 501 kB 00:00 2024-03-18T16:05:26.603 INFO:teuthology.orchestra.run.smithi102.stdout:(32/196): librbd1-debuginfo-19.0.0-2162.g3d78f5 12 MB/s | 52 MB 00:04 2024-03-18T16:05:27.094 INFO:teuthology.orchestra.run.smithi102.stdout:(33/196): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-18T16:05:27.161 INFO:teuthology.orchestra.run.smithi102.stdout:(34/196): python3-ceph-argparse-19.0.0-2162.g3d 688 kB/s | 45 kB 00:00 2024-03-18T16:05:27.245 INFO:teuthology.orchestra.run.smithi102.stdout:(35/196): python3-ceph-common-19.0.0-2162.g3d78 1.6 MB/s | 131 kB 00:00 2024-03-18T16:05:27.361 INFO:teuthology.orchestra.run.smithi102.stdout:(36/196): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.4 MB/s | 162 kB 00:00 2024-03-18T16:05:27.554 INFO:teuthology.orchestra.run.smithi102.stdout:(37/196): python3-cephfs-debuginfo-19.0.0-2162. 3.1 MB/s | 615 kB 00:00 2024-03-18T16:05:27.654 INFO:teuthology.orchestra.run.smithi102.stdout:(38/196): python3-rados-19.0.0-2162.g3d78f5ba.e 3.2 MB/s | 320 kB 00:00 2024-03-18T16:05:27.813 INFO:teuthology.orchestra.run.smithi102.stdout:(39/196): python3-rados-debuginfo-19.0.0-2162.g 4.8 MB/s | 770 kB 00:00 2024-03-18T16:05:27.896 INFO:teuthology.orchestra.run.smithi102.stdout:(40/196): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.5 MB/s | 298 kB 00:00 2024-03-18T16:05:28.074 INFO:teuthology.orchestra.run.smithi102.stdout:(41/196): python3-rbd-debuginfo-19.0.0-2162.g3d 6.5 MB/s | 1.1 MB 00:00 2024-03-18T16:05:28.192 INFO:teuthology.orchestra.run.smithi102.stdout:(42/196): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.3 MB/s | 99 kB 00:00 2024-03-18T16:05:28.280 INFO:teuthology.orchestra.run.smithi102.stdout:(43/196): python3-rgw-debuginfo-19.0.0-2162.g3d 2.5 MB/s | 337 kB 00:00 2024-03-18T16:05:28.373 INFO:teuthology.orchestra.run.smithi102.stdout:(44/196): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 937 kB/s | 85 kB 00:00 2024-03-18T16:05:28.540 INFO:teuthology.orchestra.run.smithi102.stdout:(45/196): rbd-fuse-debuginfo-19.0.0-2162.g3d78f 3.0 MB/s | 516 kB 00:00 2024-03-18T16:05:28.873 INFO:teuthology.orchestra.run.smithi102.stdout:(46/196): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 9.3 MB/s | 3.1 MB 00:00 2024-03-18T16:05:34.148 INFO:teuthology.orchestra.run.smithi102.stdout:(47/196): rbd-mirror-debuginfo-19.0.0-2162.g3d7 9.8 MB/s | 52 MB 00:05 2024-03-18T16:05:34.207 INFO:teuthology.orchestra.run.smithi102.stdout:(48/196): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-18T16:05:34.424 INFO:teuthology.orchestra.run.smithi102.stdout:(49/196): rbd-nbd-debuginfo-19.0.0-2162.g3d78f5 6.0 MB/s | 1.3 MB 00:00 2024-03-18T16:05:34.491 INFO:teuthology.orchestra.run.smithi102.stdout:(50/196): ceph-grafana-dashboards-19.0.0-2162.g 517 kB/s | 34 kB 00:00 2024-03-18T16:05:34.574 INFO:teuthology.orchestra.run.smithi102.stdout:(51/196): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.6 MB/s | 139 kB 00:00 2024-03-18T16:05:34.775 INFO:teuthology.orchestra.run.smithi102.stdout:(52/196): ceph-mgr-dashboard-19.0.0-2162.g3d78f 9.0 MB/s | 1.8 MB 00:00 2024-03-18T16:05:35.933 INFO:teuthology.orchestra.run.smithi102.stdout:(53/196): ceph-mgr-diskprediction-local-19.0.0- 6.4 MB/s | 7.4 MB 00:01 2024-03-18T16:05:36.025 INFO:teuthology.orchestra.run.smithi102.stdout:(54/196): ceph-mgr-modules-core-19.0.0-2162.g3d 2.7 MB/s | 248 kB 00:00 2024-03-18T16:05:36.892 INFO:teuthology.orchestra.run.smithi102.stdout:(55/196): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 57 kB/s | 49 kB 00:00 2024-03-18T16:05:36.950 INFO:teuthology.orchestra.run.smithi102.stdout:(56/196): ceph-prometheus-alerts-19.0.0-2162.g3 287 kB/s | 16 kB 00:00 2024-03-18T16:05:37.151 INFO:teuthology.orchestra.run.smithi102.stdout:(57/196): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.3 MB/s | 264 kB 00:00 2024-03-18T16:05:37.318 INFO:teuthology.orchestra.run.smithi102.stdout:(58/196): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-18T16:05:37.393 INFO:teuthology.orchestra.run.smithi102.stdout:(59/196): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 246 kB/s | 18 kB 00:00 2024-03-18T16:05:37.593 INFO:teuthology.orchestra.run.smithi102.stdout:(60/196): automake-1.16.2-8.el9.noarch.rpm 3.3 MB/s | 662 kB 00:00 2024-03-18T16:05:37.702 INFO:teuthology.orchestra.run.smithi102.stdout:(61/196): bison-3.7.4-5.el9.x86_64.rpm 8.4 MB/s | 920 kB 00:00 2024-03-18T16:05:37.752 INFO:teuthology.orchestra.run.smithi102.stdout:(62/196): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-18T16:05:37.811 INFO:teuthology.orchestra.run.smithi102.stdout:(63/196): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-18T16:05:37.861 INFO:teuthology.orchestra.run.smithi102.stdout:(64/196): flexiblas-3.0.4-8.el9.x86_64.rpm 642 kB/s | 32 kB 00:00 2024-03-18T16:05:38.037 INFO:teuthology.orchestra.run.smithi102.stdout:(65/196): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-18T16:05:38.095 INFO:teuthology.orchestra.run.smithi102.stdout:(66/196): flexiblas-openblas-openmp-3.0.4-8.el9 299 kB/s | 17 kB 00:00 2024-03-18T16:05:38.162 INFO:teuthology.orchestra.run.smithi102.stdout:(67/196): libaio-devel-0.3.111-13.el9.x86_64.rp 153 kB/s | 10 kB 00:00 2024-03-18T16:05:38.229 INFO:teuthology.orchestra.run.smithi102.stdout:(68/196): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-03-18T16:05:38.288 INFO:teuthology.orchestra.run.smithi102.stdout:(69/196): librabbitmq-0.11.0-7.el9.x86_64.rpm 768 kB/s | 44 kB 00:00 2024-03-18T16:05:38.371 INFO:teuthology.orchestra.run.smithi102.stdout:(70/196): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-18T16:05:38.422 INFO:teuthology.orchestra.run.smithi102.stdout:(71/196): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-18T16:05:38.447 INFO:teuthology.orchestra.run.smithi102.stdout:(72/196): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.9 MB/s | 46 kB 00:00 2024-03-18T16:05:38.489 INFO:teuthology.orchestra.run.smithi102.stdout:(73/196): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 282 kB 00:00 2024-03-18T16:05:38.531 INFO:teuthology.orchestra.run.smithi102.stdout:(74/196): lua-5.4.4-4.el9.x86_64.rpm 4.5 MB/s | 187 kB 00:00 2024-03-18T16:05:38.573 INFO:teuthology.orchestra.run.smithi102.stdout:(75/196): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.2 MB/s | 217 kB 00:00 2024-03-18T16:05:38.598 INFO:teuthology.orchestra.run.smithi102.stdout:(76/196): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-03-18T16:05:38.765 INFO:teuthology.orchestra.run.smithi102.stdout:(77/196): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2024-03-18T16:05:38.841 INFO:teuthology.orchestra.run.smithi102.stdout:(78/196): postgresql-13.11-1.el9.x86_64.rpm 21 MB/s | 1.5 MB 00:00 2024-03-18T16:05:38.893 INFO:teuthology.orchestra.run.smithi102.stdout:(79/196): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-18T16:05:38.941 INFO:teuthology.orchestra.run.smithi102.stdout:(80/196): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-18T16:05:39.100 INFO:teuthology.orchestra.run.smithi102.stdout:(81/196): postgresql-server-13.11-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-03-18T16:05:39.167 INFO:teuthology.orchestra.run.smithi102.stdout:(82/196): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-18T16:05:39.209 INFO:teuthology.orchestra.run.smithi102.stdout:(83/196): python3-attrs-20.3.0-7.el9.noarch.rpm 1.9 MB/s | 78 kB 00:00 2024-03-18T16:05:39.376 INFO:teuthology.orchestra.run.smithi102.stdout:(84/196): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-18T16:05:39.409 INFO:teuthology.orchestra.run.smithi102.stdout:(85/196): python3-jsonpatch-1.21-16.el9.noarch. 759 kB/s | 24 kB 00:00 2024-03-18T16:05:39.434 INFO:teuthology.orchestra.run.smithi102.stdout:(86/196): python3-jsonpointer-2.0-4.el9.noarch. 751 kB/s | 18 kB 00:00 2024-03-18T16:05:39.476 INFO:teuthology.orchestra.run.smithi102.stdout:(87/196): python3-mako-1.1.4-6.el9.noarch.rpm 3.5 MB/s | 144 kB 00:00 2024-03-18T16:05:39.502 INFO:teuthology.orchestra.run.smithi102.stdout:(88/196): python3-markupsafe-1.1.1-12.el9.x86_6 1.3 MB/s | 32 kB 00:00 2024-03-18T16:05:39.652 INFO:teuthology.orchestra.run.smithi102.stdout:(89/196): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-18T16:05:39.694 INFO:teuthology.orchestra.run.smithi102.stdout:(90/196): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.2 MB/s | 217 kB 00:00 2024-03-18T16:05:39.736 INFO:teuthology.orchestra.run.smithi102.stdout:(91/196): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-18T16:05:39.778 INFO:teuthology.orchestra.run.smithi102.stdout:(92/196): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-18T16:05:39.812 INFO:teuthology.orchestra.run.smithi102.stdout:(93/196): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-18T16:05:39.887 INFO:teuthology.orchestra.run.smithi102.stdout:(94/196): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-18T16:05:39.929 INFO:teuthology.orchestra.run.smithi102.stdout:(95/196): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2024-03-18T16:05:39.971 INFO:teuthology.orchestra.run.smithi102.stdout:(96/196): python3-pyasn1-modules-0.4.8-6.el9.no 5.0 MB/s | 211 kB 00:00 2024-03-18T16:05:40.013 INFO:teuthology.orchestra.run.smithi102.stdout:(97/196): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-18T16:05:40.047 INFO:teuthology.orchestra.run.smithi102.stdout:(98/196): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-18T16:05:40.489 INFO:teuthology.orchestra.run.smithi102.stdout:(99/196): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-18T16:05:40.533 INFO:teuthology.orchestra.run.smithi102.stdout:(100/196): python3-toml-0.10.2-6.el9.noarch.rpm 956 kB/s | 39 kB 00:00 2024-03-18T16:05:40.572 INFO:teuthology.orchestra.run.smithi102.stdout:(101/196): python3-wcwidth-0.2.5-8.el9.noarch.r 1.0 MB/s | 41 kB 00:00 2024-03-18T16:05:40.614 INFO:teuthology.orchestra.run.smithi102.stdout:(102/196): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2024-03-18T16:05:40.656 INFO:teuthology.orchestra.run.smithi102.stdout:(103/196): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-18T16:05:40.698 INFO:teuthology.orchestra.run.smithi102.stdout:(104/196): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-18T16:05:40.740 INFO:teuthology.orchestra.run.smithi102.stdout:(105/196): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-18T16:05:40.782 INFO:teuthology.orchestra.run.smithi102.stdout:(106/196): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-18T16:05:40.824 INFO:teuthology.orchestra.run.smithi102.stdout:(107/196): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.8 MB/s | 241 kB 00:00 2024-03-18T16:05:40.891 INFO:teuthology.orchestra.run.smithi102.stdout:(108/196): python3-cryptography-36.0.1-4.el9.x8 17 MB/s | 1.1 MB 00:00 2024-03-18T16:05:40.933 INFO:teuthology.orchestra.run.smithi102.stdout:(109/196): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 103 kB 00:00 2024-03-18T16:05:40.975 INFO:teuthology.orchestra.run.smithi102.stdout:(110/196): python3-pycparser-2.20-6.el9.noarch. 3.0 MB/s | 124 kB 00:00 2024-03-18T16:05:41.025 INFO:teuthology.orchestra.run.smithi102.stdout:(111/196): python3-urllib3-1.26.5-5.el9.noarch. 3.8 MB/s | 187 kB 00:00 2024-03-18T16:05:41.401 INFO:teuthology.orchestra.run.smithi102.stdout:(112/196): jq-1.6-16.el9.x86_64.rpm 501 kB/s | 187 kB 00:00 2024-03-18T16:05:41.509 INFO:teuthology.orchestra.run.smithi102.stdout:(113/196): libgfortran-11.4.1-3.el9.x86_64.rpm 7.3 MB/s | 803 kB 00:00 2024-03-18T16:05:41.618 INFO:teuthology.orchestra.run.smithi102.stdout:(114/196): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-03-18T16:05:41.744 INFO:teuthology.orchestra.run.smithi102.stdout:(115/196): python3-requests-2.25.1-8.el9.noarch 1.0 MB/s | 125 kB 00:00 2024-03-18T16:05:41.794 INFO:teuthology.orchestra.run.smithi102.stdout:(116/196): elfutils-libelf-devel-0.190-2.el9.x8 438 kB/s | 21 kB 00:00 2024-03-18T16:05:41.836 INFO:teuthology.orchestra.run.smithi102.stdout:(117/196): libuuid-devel-2.37.4-18.el9.x86_64.r 624 kB/s | 25 kB 00:00 2024-03-18T16:05:42.086 INFO:teuthology.orchestra.run.smithi102.stdout:(118/196): openssl-devel-3.0.7-27.el9.x86_64.rp 17 MB/s | 4.1 MB 00:00 2024-03-18T16:05:42.170 INFO:teuthology.orchestra.run.smithi102.stdout:(119/196): python3-devel-3.9.17-2.el9.x86_64.rp 2.9 MB/s | 245 kB 00:00 2024-03-18T16:05:42.253 INFO:teuthology.orchestra.run.smithi102.stdout:(120/196): python3-jinja2-2.11.3-5.el9.noarch.r 2.9 MB/s | 248 kB 00:00 2024-03-18T16:05:42.329 INFO:teuthology.orchestra.run.smithi102.stdout:(121/196): qatlib-23.11.0-1.el9.x86_64.rpm 2.9 MB/s | 216 kB 00:00 2024-03-18T16:05:42.379 INFO:teuthology.orchestra.run.smithi102.stdout:(122/196): qatlib-service-23.11.0-1.el9.x86_64. 708 kB/s | 35 kB 00:00 2024-03-18T16:05:42.437 INFO:teuthology.orchestra.run.smithi102.stdout:(123/196): qatzip-libs-1.1.2-2.el9.x86_64.rpm 774 kB/s | 44 kB 00:00 2024-03-18T16:05:42.530 INFO:teuthology.orchestra.run.smithi102.stdout:(124/196): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.0 MB/s | 94 kB 00:00 2024-03-18T16:05:51.068 INFO:teuthology.orchestra.run.smithi102.stdout:(125/196): ceph-osd-debuginfo-19.0.0-2162.g3d78 4.4 MB/s | 290 MB 01:05 2024-03-18T16:06:03.234 INFO:teuthology.orchestra.run.smithi102.stdout:(126/196): ceph-test-debuginfo-19.0.0-2162.g3d7 12 MB/s | 679 MB 00:54 2024-03-18T16:06:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:(127/196): abseil-cpp-20211102.0-3.el9.x86_64.r 3.2 MB/s | 548 kB 00:00 2024-03-18T16:06:03.476 INFO:teuthology.orchestra.run.smithi102.stdout:(128/196): gperftools-libs-2.9.1-2.el9.x86_64.r 4.0 MB/s | 309 kB 00:00 2024-03-18T16:06:03.510 INFO:teuthology.orchestra.run.smithi102.stdout:(129/196): grpc-data-1.46.7-10.el9.noarch.rpm 595 kB/s | 19 kB 00:00 2024-03-18T16:06:03.768 INFO:teuthology.orchestra.run.smithi102.stdout:(130/196): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-18T16:06:03.844 INFO:teuthology.orchestra.run.smithi102.stdout:(131/196): libarrow-doc-9.0.0-10.el9.noarch.rpm 348 kB/s | 26 kB 00:00 2024-03-18T16:06:03.877 INFO:teuthology.orchestra.run.smithi102.stdout:(132/196): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-18T16:06:03.911 INFO:teuthology.orchestra.run.smithi102.stdout:(133/196): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-18T16:06:03.961 INFO:teuthology.orchestra.run.smithi102.stdout:(134/196): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-03-18T16:06:04.054 INFO:teuthology.orchestra.run.smithi102.stdout:(135/196): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 838 kB 00:00 2024-03-18T16:06:04.112 INFO:teuthology.orchestra.run.smithi102.stdout:(136/196): python3-asyncssh-2.13.2-4.el9.noarch 9.3 MB/s | 548 kB 00:00 2024-03-18T16:06:04.154 INFO:teuthology.orchestra.run.smithi102.stdout:(137/196): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-03-18T16:06:04.204 INFO:teuthology.orchestra.run.smithi102.stdout:(138/196): python3-cachetools-4.2.4-1.el9.noarc 650 kB/s | 32 kB 00:00 2024-03-18T16:06:04.254 INFO:teuthology.orchestra.run.smithi102.stdout:(139/196): python3-certifi-2021.10.8-2.el9.noar 304 kB/s | 15 kB 00:00 2024-03-18T16:06:04.321 INFO:teuthology.orchestra.run.smithi102.stdout:(140/196): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-18T16:06:04.371 INFO:teuthology.orchestra.run.smithi102.stdout:(141/196): python3-cherrypy-18.6.1-2.el9.noarch 7.1 MB/s | 358 kB 00:00 2024-03-18T16:06:04.430 INFO:teuthology.orchestra.run.smithi102.stdout:(142/196): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-03-18T16:06:04.472 INFO:teuthology.orchestra.run.smithi102.stdout:(143/196): python3-colorama-0.4.4-12.el9.noarch 850 kB/s | 35 kB 00:00 2024-03-18T16:06:04.522 INFO:teuthology.orchestra.run.smithi102.stdout:(144/196): python3-google-auth-2.28.0-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-03-18T16:06:04.642 INFO:teuthology.orchestra.run.smithi102.stdout:(145/196): python3-grpcio-1.46.7-10.el9.x86_64. 18 MB/s | 2.0 MB 00:00 2024-03-18T16:06:04.690 INFO:teuthology.orchestra.run.smithi102.stdout:(146/196): python3-grpcio-tools-1.46.7-10.el9.x 2.9 MB/s | 144 kB 00:00 2024-03-18T16:06:04.732 INFO:teuthology.orchestra.run.smithi102.stdout:(147/196): python3-jaraco-8.2.1-3.el9.noarch.rp 260 kB/s | 11 kB 00:00 2024-03-18T16:06:04.815 INFO:teuthology.orchestra.run.smithi102.stdout:(148/196): python3-jaraco-classes-3.2.1-5.el9.n 213 kB/s | 18 kB 00:00 2024-03-18T16:06:04.857 INFO:teuthology.orchestra.run.smithi102.stdout:(149/196): python3-jaraco-collections-3.0.0-8.e 565 kB/s | 23 kB 00:00 2024-03-18T16:06:04.899 INFO:teuthology.orchestra.run.smithi102.stdout:(150/196): python3-jaraco-functools-3.5.0-2.el9 468 kB/s | 19 kB 00:00 2024-03-18T16:06:04.949 INFO:teuthology.orchestra.run.smithi102.stdout:(151/196): python3-jaraco-text-3.2.0-6.el9.noar 397 kB/s | 20 kB 00:00 2024-03-18T16:06:05.041 INFO:teuthology.orchestra.run.smithi102.stdout:(152/196): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-03-18T16:06:05.083 INFO:teuthology.orchestra.run.smithi102.stdout:(153/196): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-03-18T16:06:05.133 INFO:teuthology.orchestra.run.smithi102.stdout:(154/196): python3-more-itertools-8.12.0-2.el9. 1.6 MB/s | 79 kB 00:00 2024-03-18T16:06:05.183 INFO:teuthology.orchestra.run.smithi102.stdout:(155/196): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-18T16:06:05.234 INFO:teuthology.orchestra.run.smithi102.stdout:(156/196): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-18T16:06:05.276 INFO:teuthology.orchestra.run.smithi102.stdout:(157/196): python3-portend-3.1.0-2.el9.noarch.r 400 kB/s | 16 kB 00:00 2024-03-18T16:06:05.310 INFO:teuthology.orchestra.run.smithi102.stdout:(158/196): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.6 MB/s | 90 kB 00:00 2024-03-18T16:06:05.393 INFO:teuthology.orchestra.run.smithi102.stdout:(159/196): python3-pyperclip-1.8.2-2.el9.noarch 340 kB/s | 28 kB 00:00 2024-03-18T16:06:05.443 INFO:teuthology.orchestra.run.smithi102.stdout:(160/196): python3-repoze-lru-0.7-16.el9.noarch 623 kB/s | 31 kB 00:00 2024-03-18T16:06:05.502 INFO:teuthology.orchestra.run.smithi102.stdout:(161/196): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-03-18T16:06:05.561 INFO:teuthology.orchestra.run.smithi102.stdout:(162/196): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-18T16:06:05.653 INFO:teuthology.orchestra.run.smithi102.stdout:(163/196): python3-tempora-5.0.0-2.el9.noarch.r 392 kB/s | 36 kB 00:00 2024-03-18T16:06:05.686 INFO:teuthology.orchestra.run.smithi102.stdout:(164/196): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-03-18T16:06:05.753 INFO:teuthology.orchestra.run.smithi102.stdout:(165/196): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-18T16:06:05.803 INFO:teuthology.orchestra.run.smithi102.stdout:(166/196): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-03-18T16:06:05.870 INFO:teuthology.orchestra.run.smithi102.stdout:(167/196): python3-werkzeug-2.0.3-3.el9.1.noarc 6.3 MB/s | 427 kB 00:00 2024-03-18T16:06:05.929 INFO:teuthology.orchestra.run.smithi102.stdout:(168/196): python3-zc-lockfile-2.0-10.el9.noarc 343 kB/s | 20 kB 00:00 2024-03-18T16:06:05.971 INFO:teuthology.orchestra.run.smithi102.stdout:(169/196): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-03-18T16:06:06.079 INFO:teuthology.orchestra.run.smithi102.stdout:(170/196): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-18T16:06:06.388 INFO:teuthology.orchestra.run.smithi102.stdout:(171/196): librados2-19.0.0-2162.g3d78f5ba.el9. 11 MB/s | 3.4 MB 00:00 2024-03-18T16:06:06.589 INFO:teuthology.orchestra.run.smithi102.stdout:(172/196): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 16 MB/s | 3.1 MB 00:00 2024-03-18T16:06:06.639 INFO:teuthology.orchestra.run.smithi102.stdout:(173/196): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-18T16:06:06.823 INFO:teuthology.orchestra.run.smithi102.stdout:(174/196): NetworkManager-1.46.0-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-03-18T16:06:06.991 INFO:teuthology.orchestra.run.smithi102.stdout:(175/196): NetworkManager-libnm-1.46.0-1.el9.x8 11 MB/s | 1.8 MB 00:00 2024-03-18T16:06:07.075 INFO:teuthology.orchestra.run.smithi102.stdout:(176/196): NetworkManager-team-1.46.0-1.el9.x86 512 kB/s | 42 kB 00:00 2024-03-18T16:06:07.158 INFO:teuthology.orchestra.run.smithi102.stdout:(177/196): NetworkManager-tui-1.46.0-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-18T16:06:07.217 INFO:teuthology.orchestra.run.smithi102.stdout:(178/196): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-18T16:06:07.276 INFO:teuthology.orchestra.run.smithi102.stdout:(179/196): elfutils-debuginfod-client-0.190-2.e 631 kB/s | 36 kB 00:00 2024-03-18T16:06:07.351 INFO:teuthology.orchestra.run.smithi102.stdout:(180/196): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-18T16:06:07.426 INFO:teuthology.orchestra.run.smithi102.stdout:(181/196): elfutils-libs-0.190-2.el9.x86_64.rpm 3.4 MB/s | 256 kB 00:00 2024-03-18T16:06:07.476 INFO:teuthology.orchestra.run.smithi102.stdout:(182/196): libacl-2.3.1-4.el9.x86_64.rpm 465 kB/s | 23 kB 00:00 2024-03-18T16:06:07.543 INFO:teuthology.orchestra.run.smithi102.stdout:(183/196): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-18T16:06:07.619 INFO:teuthology.orchestra.run.smithi102.stdout:(184/196): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-18T16:06:07.694 INFO:teuthology.orchestra.run.smithi102.stdout:(185/196): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-18T16:06:07.761 INFO:teuthology.orchestra.run.smithi102.stdout:(186/196): libsmartcols-2.37.4-18.el9.x86_64.rp 967 kB/s | 64 kB 00:00 2024-03-18T16:06:07.811 INFO:teuthology.orchestra.run.smithi102.stdout:(187/196): libuuid-2.37.4-18.el9.x86_64.rpm 580 kB/s | 29 kB 00:00 2024-03-18T16:06:07.945 INFO:teuthology.orchestra.run.smithi102.stdout:(188/196): openssl-3.0.7-27.el9.x86_64.rpm 9.1 MB/s | 1.2 MB 00:00 2024-03-18T16:06:08.113 INFO:teuthology.orchestra.run.smithi102.stdout:(189/196): openssl-libs-3.0.7-27.el9.x86_64.rpm 13 MB/s | 2.2 MB 00:00 2024-03-18T16:06:08.287 INFO:teuthology.orchestra.run.smithi102.stdout:(190/196): util-linux-2.37.4-18.el9.x86_64.rpm 13 MB/s | 2.3 MB 00:00 2024-03-18T16:06:08.379 INFO:teuthology.orchestra.run.smithi102.stdout:(191/196): util-linux-core-2.37.4-18.el9.x86_64 5.0 MB/s | 465 kB 00:00 2024-03-18T16:06:08.429 INFO:teuthology.orchestra.run.smithi102.stdout:(192/196): util-linux-user-2.37.4-18.el9.x86_64 633 kB/s | 31 kB 00:00 2024-03-18T16:06:08.554 INFO:teuthology.orchestra.run.smithi102.stdout:(193/196): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-18T16:06:08.613 INFO:teuthology.orchestra.run.smithi102.stdout:(194/196): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 74 kB 00:00 2024-03-18T16:07:57.202 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-18T16:07:57.492 INFO:teuthology.orchestra.run.smithi102.stdout:(195/196): lua-devel-5.4.4-4.el9.x86_64.rpm 169 B/s | 22 kB 02:14 2024-03-18T16:08:04.999 INFO:teuthology.orchestra.run.smithi102.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-18T16:08:05.900 INFO:teuthology.orchestra.run.smithi102.stdout:(196/196): protobuf-compiler-3.14.0-13.el9.x86_ 6.4 kB/s | 863 kB 02:14 2024-03-18T16:08:05.904 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-18T16:08:05.904 INFO:teuthology.orchestra.run.smithi102.stdout:Total 9.8 MB/s | 2.2 GB 03:48 2024-03-18T16:08:07.350 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-18T16:08:07.737 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-18T16:08:07.737 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-18T16:08:22.244 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-18T16:08:22.245 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-18T16:08:23.415 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 31 kB/s | 8.0 MB 04:28 2024-03-18T16:08:42.176 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-18T16:08:44.949 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/220 2024-03-18T16:08:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_6 2/220 2024-03-18T16:08:45.038 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 3/220 2024-03-18T16:08:45.082 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-18T16:08:45.116 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-18T16:08:45.203 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/220 2024-03-18T16:08:45.297 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/220 2024-03-18T16:08:45.347 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/220 2024-03-18T16:08:45.372 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/220 2024-03-18T16:08:45.448 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/220 2024-03-18T16:08:45.872 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/220 2024-03-18T16:08:45.945 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/220 2024-03-18T16:08:46.123 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/220 2024-03-18T16:08:46.155 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/220 2024-03-18T16:08:46.283 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/220 2024-03-18T16:08:46.323 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/220 2024-03-18T16:08:46.383 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/220 2024-03-18T16:08:46.677 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 15/220 2024-03-18T16:08:46.725 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-13.11-1.el9.x86_64 16/220 2024-03-18T16:08:46.752 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/220 2024-03-18T16:08:46.906 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/220 2024-03-18T16:08:46.944 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 18/220 2024-03-18T16:08:47.036 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 18/220 2024-03-18T16:08:47.098 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 19/220 2024-03-18T16:08:47.123 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-18T16:08:47.186 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-18T16:08:47.475 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-18T16:08:47.522 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-18T16:08:47.565 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 22/220 2024-03-18T16:08:47.847 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 23/220 2024-03-18T16:08:47.997 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 24/220 2024-03-18T16:08:48.094 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 24/220 2024-03-18T16:08:48.276 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 25/220 2024-03-18T16:08:48.373 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 25/220 2024-03-18T16:08:48.688 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 25/220 2024-03-18T16:08:48.866 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 26/220 2024-03-18T16:08:49.027 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 27/220 2024-03-18T16:08:49.072 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 28/220 2024-03-18T16:08:49.110 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 29/220 2024-03-18T16:08:49.169 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/220 2024-03-18T16:08:49.277 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/220 2024-03-18T16:08:49.346 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/220 2024-03-18T16:08:49.389 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/220 2024-03-18T16:08:49.480 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 34/220 2024-03-18T16:08:49.536 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 35/220 2024-03-18T16:08:49.609 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 36/220 2024-03-18T16:08:49.659 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 37/220 2024-03-18T16:08:49.698 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-5.4.4-4.el9.x86_64 38/220 2024-03-18T16:08:49.816 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 39/220 2024-03-18T16:08:49.860 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 40/220 2024-03-18T16:08:49.922 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 41/220 2024-03-18T16:08:49.959 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 42/220 2024-03-18T16:08:50.078 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 43/220 2024-03-18T16:08:50.135 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : luarocks-3.9.2-1.el9.noarch 44/220 2024-03-18T16:08:50.234 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 45/220 2024-03-18T16:08:50.423 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 46/220 2024-03-18T16:08:50.497 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 47/220 2024-03-18T16:08:51.191 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/220 2024-03-18T16:08:51.302 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 49/220 2024-03-18T16:08:51.446 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 50/220 2024-03-18T16:08:51.512 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-18T16:08:51.540 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-18T16:08:51.592 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 52/220 2024-03-18T16:08:51.667 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 53/220 2024-03-18T16:08:51.722 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 54/220 2024-03-18T16:08:51.762 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 55/220 2024-03-18T16:08:52.196 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/220 2024-03-18T16:08:52.405 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/220 2024-03-18T16:08:52.502 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/220 2024-03-18T16:08:52.579 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/220 2024-03-18T16:08:52.789 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/220 2024-03-18T16:08:52.839 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/220 2024-03-18T16:08:52.910 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/220 2024-03-18T16:08:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/220 2024-03-18T16:08:53.071 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/220 2024-03-18T16:08:53.119 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/220 2024-03-18T16:08:53.205 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/220 2024-03-18T16:08:53.245 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/220 2024-03-18T16:08:53.309 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/220 2024-03-18T16:08:53.375 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/220 2024-03-18T16:08:53.563 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/220 2024-03-18T16:08:53.617 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/220 2024-03-18T16:08:53.679 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/220 2024-03-18T16:08:53.728 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/220 2024-03-18T16:08:53.867 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/220 2024-03-18T16:08:53.912 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/220 2024-03-18T16:08:53.958 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/220 2024-03-18T16:08:53.995 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/220 2024-03-18T16:08:54.028 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/220 2024-03-18T16:08:54.195 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/220 2024-03-18T16:08:54.399 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/220 2024-03-18T16:08:54.533 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/220 2024-03-18T16:08:54.581 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/220 2024-03-18T16:08:54.652 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/220 2024-03-18T16:08:54.750 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/220 2024-03-18T16:08:55.001 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/220 2024-03-18T16:08:55.052 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/220 2024-03-18T16:08:55.227 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/220 2024-03-18T16:08:55.299 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/220 2024-03-18T16:08:55.340 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/220 2024-03-18T16:08:55.377 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-15.el9.x86_64 90/220 2024-03-18T16:08:55.418 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 91/220 2024-03-18T16:08:55.465 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 92/220 2024-03-18T16:08:55.532 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : uuid-1.6.2-55.el9.x86_64 93/220 2024-03-18T16:08:55.583 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 94/220 2024-03-18T16:08:55.631 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 95/220 2024-03-18T16:08:55.757 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 96/220 2024-03-18T16:08:55.810 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 97/220 2024-03-18T16:08:55.870 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 98/220 2024-03-18T16:08:55.939 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 99/220 2024-03-18T16:08:55.983 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 100/220 2024-03-18T16:08:56.129 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 101/220 2024-03-18T16:08:56.247 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 102/220 2024-03-18T16:08:56.303 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 103/220 2024-03-18T16:08:56.339 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 104/220 2024-03-18T16:08:56.396 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 105/220 2024-03-18T16:08:56.553 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-packaging-20.9-5.el9.noarch 106/220 2024-03-18T16:08:56.622 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 107/220 2024-03-18T16:08:57.585 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 108/220 2024-03-18T16:08:57.646 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 109/220 2024-03-18T16:08:57.679 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 110/220 2024-03-18T16:08:57.736 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 111/220 2024-03-18T16:08:57.855 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 112/220 2024-03-18T16:08:57.905 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 113/220 2024-03-18T16:08:58.136 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 114/220 2024-03-18T16:08:58.168 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 115/220 2024-03-18T16:08:58.268 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 116/220 2024-03-18T16:08:59.068 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 117/220 2024-03-18T16:08:59.205 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 118/220 2024-03-18T16:09:00.747 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 119/220 2024-03-18T16:09:00.822 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 120/220 2024-03-18T16:09:00.860 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 121/220 2024-03-18T16:09:00.891 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.6-16.el9.x86_64 122/220 2024-03-18T16:09:00.930 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/220 2024-03-18T16:09:01.304 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/220 2024-03-18T16:09:01.330 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/220 2024-03-18T16:09:01.451 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/220 2024-03-18T16:09:01.486 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 126/220 2024-03-18T16:09:01.540 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/220 2024-03-18T16:09:02.125 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/220 2024-03-18T16:09:02.177 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/220 2024-03-18T16:09:02.275 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/220 2024-03-18T16:09:02.302 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/220 2024-03-18T16:09:05.091 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/220 2024-03-18T16:09:05.140 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/220 2024-03-18T16:09:05.791 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/220 2024-03-18T16:09:05.835 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/220 2024-03-18T16:09:06.003 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/220 2024-03-18T16:09:06.003 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T16:09:06.004 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T16:09:06.004 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:06.069 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 133/220 2024-03-18T16:09:19.747 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 133/220 2024-03-18T16:09:19.748 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-03-18T16:09:19.748 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-03-18T16:09:19.748 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-03-18T16:09:19.748 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-03-18T16:09:19.748 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-03-18T16:09:19.748 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-03-18T16:09:19.748 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-03-18T16:09:19.748 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:19.975 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 134/220 2024-03-18T16:09:20.025 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 134/220 2024-03-18T16:09:20.025 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:09:20.025 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T16:09:20.025 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:21.628 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/220 2024-03-18T16:09:21.710 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/220 2024-03-18T16:09:21.710 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:09:21.710 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T16:09:21.710 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T16:09:21.710 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T16:09:21.710 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:21.953 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/220 2024-03-18T16:09:22.022 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/220 2024-03-18T16:09:22.022 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:09:22.022 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T16:09:22.022 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T16:09:22.022 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T16:09:22.023 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:22.496 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 137/220 2024-03-18T16:09:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 137/220 2024-03-18T16:09:22.572 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:09:22.572 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T16:09:22.572 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T16:09:22.572 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T16:09:22.572 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:22.711 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : automake-1.16.2-8.el9.noarch 138/220 2024-03-18T16:09:23.271 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 139/220 2024-03-18T16:09:23.324 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 139/220 2024-03-18T16:09:23.376 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 140/220 2024-03-18T16:09:23.478 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 141/220 2024-03-18T16:09:23.519 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 142/220 2024-03-18T16:09:23.855 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 142/220 2024-03-18T16:09:23.912 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 143/220 2024-03-18T16:09:25.037 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 143/220 2024-03-18T16:09:25.070 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 144/220 2024-03-18T16:09:25.213 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 144/220 2024-03-18T16:09:25.384 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 145/220 2024-03-18T16:09:25.422 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 146/220 2024-03-18T16:09:25.489 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 146/220 2024-03-18T16:09:25.489 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:09:25.489 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T16:09:25.489 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T16:09:25.490 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T16:09:25.490 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:25.557 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 147/220 2024-03-18T16:09:25.596 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 147/220 2024-03-18T16:09:25.678 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 148/220 2024-03-18T16:09:26.628 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/220 2024-03-18T16:09:26.667 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 150/220 2024-03-18T16:09:26.752 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 150/220 2024-03-18T16:09:26.752 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:09:26.752 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T16:09:26.752 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T16:09:26.753 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T16:09:26.753 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:26.809 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 151/220 2024-03-18T16:09:26.888 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 151/220 2024-03-18T16:09:26.888 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:09:26.888 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T16:09:26.888 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:26.969 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/220 2024-03-18T16:09:27.046 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/220 2024-03-18T16:09:27.046 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:09:27.046 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T16:09:27.046 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T16:09:27.046 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T16:09:27.046 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:27.436 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/220 2024-03-18T16:09:27.503 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/220 2024-03-18T16:09:27.503 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:09:27.503 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T16:09:27.503 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T16:09:27.503 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T16:09:27.503 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:09:32.317 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/220 2024-03-18T16:09:32.370 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/220 2024-03-18T16:09:32.407 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 156/220 2024-03-18T16:09:32.445 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/220 2024-03-18T16:09:32.724 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 158/220 2024-03-18T16:09:32.853 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/220 2024-03-18T16:09:32.916 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 160/220 2024-03-18T16:09:32.968 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/220 2024-03-18T16:09:33.033 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/220 2024-03-18T16:09:33.103 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/220 2024-03-18T16:09:33.147 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 164/220 2024-03-18T16:09:33.192 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/220 2024-03-18T16:09:33.243 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 166/220 2024-03-18T16:09:33.328 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 167/220 2024-03-18T16:09:33.459 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 168/220 2024-03-18T16:09:33.500 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/220 2024-03-18T16:09:34.373 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-18T16:09:34.416 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-18T16:09:34.472 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-18T16:09:42.136 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 171/220 2024-03-18T16:10:26.946 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 172/220 2024-03-18T16:10:27.617 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2162.g3d78f5ba.el 173/220 2024-03-18T16:10:27.800 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 174/220 2024-03-18T16:10:33.021 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 175/220 2024-03-18T16:10:35.227 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 176/220 2024-03-18T16:10:45.568 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 177/220 2024-03-18T16:11:15.467 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 178/220 2024-03-18T16:11:34.045 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 179/220 2024-03-18T16:12:46.596 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2162.g3d78f5ba.e 180/220 2024-03-18T16:12:47.200 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 181/220 2024-03-18T16:12:47.247 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9 182/220 2024-03-18T16:12:52.400 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2162.g3d78f5ba 183/220 2024-03-18T16:12:58.522 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 184/220 2024-03-18T16:12:58.612 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librbd1-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x8 185/220 2024-03-18T16:12:58.711 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2162.g3d78f5ba 186/220 2024-03-18T16:12:58.857 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2162.g3d78f5ba. 187/220 2024-03-18T16:12:58.919 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el 188/220 2024-03-18T16:12:59.003 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2162.g3d78f5ba.el 189/220 2024-03-18T16:13:01.454 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 70 kB/s | 19 MB 04:36 2024-03-18T16:13:05.030 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 190/220 2024-03-18T16:13:05.190 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2162.g3d78f5ba.el9 191/220 2024-03-18T16:13:05.963 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-18T16:13:06.430 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x8 192/220 2024-03-18T16:13:06.519 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 193/220 2024-03-18T16:13:06.572 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 194/220 2024-03-18T16:13:06.702 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flex-2.6.4-9.el9.x86_64 195/220 2024-03-18T16:13:06.739 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : bison-3.7.4-5.el9.x86_64 196/220 2024-03-18T16:13:06.772 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 197/220 2024-03-18T16:13:06.806 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 198/220 2024-03-18T16:13:06.855 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 199/220 2024-03-18T16:13:06.855 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-18T16:13:07.027 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-18T16:13:07.069 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 201/220 2024-03-18T16:13:07.069 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-18T16:13:07.190 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-18T16:13:07.233 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 203/220 2024-03-18T16:13:07.280 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 204/220 2024-03-18T16:13:07.314 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 205/220 2024-03-18T16:13:07.355 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 206/220 2024-03-18T16:13:07.435 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 207/220 2024-03-18T16:13:07.511 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 208/220 2024-03-18T16:13:07.576 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 209/220 2024-03-18T16:13:07.609 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 210/220 2024-03-18T16:13:07.643 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 211/220 2024-03-18T16:13:07.685 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 212/220 2024-03-18T16:13:07.718 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 213/220 2024-03-18T16:13:07.754 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-18T16:13:07.755 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-18T16:13:07.764 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-18T16:13:08.068 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-18T16:13:08.069 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-18T16:13:08.225 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-18T16:13:08.255 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 216/220 2024-03-18T16:13:08.288 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 217/220 2024-03-18T16:13:08.345 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 218/220 2024-03-18T16:13:08.381 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 219/220 2024-03-18T16:13:08.510 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-18T16:13:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-18T16:13:08.757 INFO:teuthology.orchestra.run.smithi102.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-18T16:13:08.757 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:13:08.899 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-18T16:13:13.832 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-18T16:13:16.320 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T16:13:16.321 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T16:13:16.353 INFO:teuthology.orchestra.run.smithi154.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T16:13:16.354 INFO:teuthology.orchestra.run.smithi154.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T16:13:16.355 INFO:teuthology.orchestra.run.smithi154.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T16:13:16.356 INFO:teuthology.orchestra.run.smithi154.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T16:13:16.358 INFO:teuthology.orchestra.run.smithi154.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T16:13:16.361 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T16:13:16.362 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T16:13:16.445 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-03-18T16:13:16.467 INFO:teuthology.orchestra.run.smithi154.stdout:==================================================================================================== 2024-03-18T16:13:16.467 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-03-18T16:13:16.467 INFO:teuthology.orchestra.run.smithi154.stdout:==================================================================================================== 2024-03-18T16:13:16.467 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-03-18T16:13:16.467 INFO:teuthology.orchestra.run.smithi154.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T16:13:16.467 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T16:13:16.467 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T16:13:16.467 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 69 M 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 443 M 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.1 M 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 54 M 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 20 M 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T16:13:16.468 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 M 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 290 M 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 178 M 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 679 M 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T16:13:16.469 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.5 M 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 109 k 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 M 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 52 M 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T16:13:16.470 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 615 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 770 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.1 M 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 337 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 516 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 52 M 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T16:13:16.471 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.3 M 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T16:13:16.472 INFO:teuthology.orchestra.run.smithi154.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-03-18T16:13:16.473 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-debuginfo x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 23 M 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T16:13:16.474 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T16:13:16.475 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T16:13:16.476 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T16:13:16.477 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T16:13:16.478 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T16:13:16.479 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T16:13:16.480 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-03-18T16:13:16.481 INFO:teuthology.orchestra.run.smithi154.stdout:==================================================================================================== 2024-03-18T16:13:16.482 INFO:teuthology.orchestra.run.smithi154.stdout:Install 172 Packages 2024-03-18T16:13:16.482 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 24 Packages 2024-03-18T16:13:16.482 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:13:16.482 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.2 G 2024-03-18T16:13:16.482 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-03-18T16:13:17.483 INFO:teuthology.orchestra.run.smithi154.stdout:(1/196): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-18T16:13:17.952 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 3/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2162.g3d78f5ba.el 5/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_6 6/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 8/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 9/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 10/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 12/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 14/220 2024-03-18T16:13:17.953 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 15/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 16/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 18/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 19/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2162.g3d78f5ba.e 20/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 21/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 22/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 23/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 26/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9 28/220 2024-03-18T16:13:17.954 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/220 2024-03-18T16:13:17.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 30/220 2024-03-18T16:13:17.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2162.g3d78f5ba 31/220 2024-03-18T16:13:17.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 32/220 2024-03-18T16:13:17.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 33/220 2024-03-18T16:13:17.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x8 34/220 2024-03-18T16:13:17.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 35/220 2024-03-18T16:13:17.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 36/220 2024-03-18T16:13:17.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 37/220 2024-03-18T16:13:17.973 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 38/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2162.g3d78f5ba 39/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 40/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2162.g3d78f5ba. 41/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 42/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el 43/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 44/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2162.g3d78f5ba.el 45/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 46/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 47/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 48/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2162.g3d78f5ba.el9 49/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 50/220 2024-03-18T16:13:17.974 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x8 51/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 52/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 53/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 54/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 55/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 56/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 57/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 58/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 59/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 60/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 61/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : automake-1.16.2-8.el9.noarch 62/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : bison-3.7.4-5.el9.x86_64 63/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 64/220 2024-03-18T16:13:17.975 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flex-2.6.4-9.el9.x86_64 65/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 66/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 67/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 68/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 69/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 73/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 74/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 75/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-5.4.4-4.el9.x86_64 76/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 77/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 78/220 2024-03-18T16:13:17.976 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 79/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-13.11-1.el9.x86_64 80/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 81/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 82/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 83/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 84/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 85/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 86/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 87/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 88/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 89/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 90/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 91/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 93/220 2024-03-18T16:13:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 94/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 95/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 96/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 97/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 98/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 99/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 100/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 101/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 102/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 103/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 104/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 105/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 106/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 107/220 2024-03-18T16:13:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 108/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 109/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 110/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 111/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 112/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 113/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.6-16.el9.x86_64 114/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 115/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 116/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 117/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 118/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 119/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 120/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 121/220 2024-03-18T16:13:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 122/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 123/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 124/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 125/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 126/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 127/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 128/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 129/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 130/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 131/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 132/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 133/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 134/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 135/220 2024-03-18T16:13:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 136/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 137/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 138/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 139/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 140/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 141/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 142/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 143/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 144/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 145/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 146/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 147/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 148/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 149/220 2024-03-18T16:13:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 150/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 151/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 152/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 153/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 154/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 155/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 156/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 157/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 158/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 159/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 160/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 161/220 2024-03-18T16:13:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 162/220 2024-03-18T16:13:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 163/220 2024-03-18T16:13:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 164/220 2024-03-18T16:13:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 165/220 2024-03-18T16:13:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 166/220 2024-03-18T16:13:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 167/220 2024-03-18T16:13:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 168/220 2024-03-18T16:13:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 169/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 170/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 171/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 172/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 173/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 174/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 175/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 176/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 179/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 180/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 181/220 2024-03-18T16:13:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 182/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 183/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 184/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 185/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 186/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : acl-2.3.1-4.el9.x86_64 187/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : acl-2.3.1-3.el9.x86_64 188/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 189/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 190/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 191/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 192/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 193/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 194/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 195/220 2024-03-18T16:13:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 196/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 197/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 198/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 199/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 200/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 201/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 202/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 203/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 204/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 205/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 206/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 207/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 208/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 209/220 2024-03-18T16:13:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 210/220 2024-03-18T16:13:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 211/220 2024-03-18T16:13:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 212/220 2024-03-18T16:13:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 213/220 2024-03-18T16:13:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 214/220 2024-03-18T16:13:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 215/220 2024-03-18T16:13:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 216/220 2024-03-18T16:13:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 217/220 2024-03-18T16:13:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 218/220 2024-03-18T16:13:17.989 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 219/220 2024-03-18T16:13:18.050 INFO:teuthology.orchestra.run.smithi154.stdout:(2/196): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 7.3 MB/s | 5.4 MB 00:00 2024-03-18T16:13:19.559 INFO:teuthology.orchestra.run.smithi154.stdout:(3/196): ceph-common-19.0.0-2162.g3d78f5ba.el9. 9.9 MB/s | 21 MB 00:02 2024-03-18T16:13:20.738 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 220/220 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T16:13:20.739 INFO:teuthology.orchestra.run.smithi102.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T16:13:20.740 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.741 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.742 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.743 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: jq-1.6-16.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.744 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T16:13:20.745 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T16:13:20.746 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T16:13:20.747 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T16:13:20.748 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T16:13:20.749 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T16:13:20.750 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T16:13:20.751 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T16:13:20.752 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T16:13:20.753 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T16:13:20.753 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T16:13:20.753 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:13:20.753 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-18T16:13:21.734 INFO:teuthology.orchestra.run.smithi154.stdout:(4/196): ceph-debuginfo-19.0.0-2162.g3d78f5ba.e 11 MB/s | 23 MB 00:02 2024-03-18T16:13:21.875 INFO:teuthology.orchestra.run.smithi154.stdout:(5/196): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 6.1 MB/s | 890 kB 00:00 2024-03-18T16:13:22.094 DEBUG:teuthology.parallel:result is None 2024-03-18T16:13:28.975 INFO:teuthology.orchestra.run.smithi154.stdout:(6/196): ceph-fuse-debuginfo-19.0.0-2162.g3d78f 878 kB/s | 6.1 MB 00:07 2024-03-18T16:13:29.091 INFO:teuthology.orchestra.run.smithi154.stdout:(7/196): ceph-immutable-object-cache-19.0.0-216 1.2 MB/s | 145 kB 00:00 2024-03-18T16:13:29.317 INFO:teuthology.orchestra.run.smithi154.stdout:(8/196): ceph-immutable-object-cache-debuginfo- 6.5 MB/s | 1.5 MB 00:00 2024-03-18T16:13:29.592 INFO:teuthology.orchestra.run.smithi154.stdout:(9/196): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 8.6 MB/s | 2.4 MB 00:00 2024-03-18T16:13:40.516 INFO:teuthology.orchestra.run.smithi154.stdout:(10/196): ceph-mds-debuginfo-19.0.0-2162.g3d78f 4.9 MB/s | 54 MB 00:10 2024-03-18T16:13:40.716 INFO:teuthology.orchestra.run.smithi154.stdout:(11/196): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x8 7.5 MB/s | 1.5 MB 00:00 2024-03-18T16:13:42.483 INFO:teuthology.orchestra.run.smithi154.stdout:(12/196): ceph-base-debuginfo-19.0.0-2162.g3d78 2.7 MB/s | 69 MB 00:25 2024-03-18T16:13:42.883 INFO:teuthology.orchestra.run.smithi154.stdout:(13/196): ceph-mgr-debuginfo-19.0.0-2162.g3d78f 9.3 MB/s | 20 MB 00:02 2024-03-18T16:13:43.267 INFO:teuthology.orchestra.run.smithi154.stdout:(14/196): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x8 6.0 MB/s | 4.7 MB 00:00 2024-03-18T16:13:45.800 INFO:teuthology.orchestra.run.smithi154.stdout:(15/196): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 6.6 MB/s | 17 MB 00:02 2024-03-18T16:13:53.200 INFO:teuthology.orchestra.run.smithi154.stdout:(16/196): ceph-mon-debuginfo-19.0.0-2162.g3d78f 9.6 MB/s | 99 MB 00:10 2024-03-18T16:13:54.349 INFO:teuthology.orchestra.run.smithi154.stdout:(17/196): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 8.6 MB/s | 9.8 MB 00:01 2024-03-18T16:14:11.891 INFO:teuthology.orchestra.run.smithi154.stdout:(18/196): ceph-radosgw-debuginfo-19.0.0-2162.g3 10 MB/s | 178 MB 00:17 2024-03-18T16:14:11.958 INFO:teuthology.orchestra.run.smithi154.stdout:(19/196): ceph-selinux-19.0.0-2162.g3d78f5ba.el 379 kB/s | 25 kB 00:00 2024-03-18T16:14:18.350 INFO:teuthology.orchestra.run.smithi154.stdout:(20/196): ceph-common-debuginfo-19.0.0-2162.g3d 7.4 MB/s | 443 MB 01:00 2024-03-18T16:14:20.316 INFO:teuthology.orchestra.run.smithi154.stdout:(21/196): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 5.7 MB/s | 48 MB 00:08 2024-03-18T16:14:20.366 INFO:teuthology.orchestra.run.smithi154.stdout:(22/196): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 4.4 MB/s | 221 kB 00:00 2024-03-18T16:14:20.441 INFO:teuthology.orchestra.run.smithi154.stdout:(23/196): cephfs-shell-19.0.0-2162.g3d78f5ba.el 362 kB/s | 27 kB 00:00 2024-03-18T16:14:20.533 INFO:teuthology.orchestra.run.smithi154.stdout:(24/196): libcephfs-devel-19.0.0-2162.g3d78f5ba 346 kB/s | 32 kB 00:00 2024-03-18T16:14:20.692 INFO:teuthology.orchestra.run.smithi154.stdout:(25/196): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 4.6 MB/s | 747 kB 00:00 2024-03-18T16:14:21.409 INFO:teuthology.orchestra.run.smithi154.stdout:(26/196): libcephfs2-debuginfo-19.0.0-2162.g3d7 7.6 MB/s | 5.5 MB 00:00 2024-03-18T16:14:21.501 INFO:teuthology.orchestra.run.smithi154.stdout:(27/196): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.7 MB/s | 163 kB 00:00 2024-03-18T16:14:21.576 INFO:teuthology.orchestra.run.smithi154.stdout:(28/196): librados-devel-19.0.0-2162.g3d78f5ba. 1.7 MB/s | 127 kB 00:00 2024-03-18T16:14:21.660 INFO:teuthology.orchestra.run.smithi154.stdout:(29/196): librados-devel-debuginfo-19.0.0-2162. 1.3 MB/s | 109 kB 00:00 2024-03-18T16:14:26.910 INFO:teuthology.orchestra.run.smithi154.stdout:(30/196): librados2-debuginfo-19.0.0-2162.g3d78 8.6 MB/s | 45 MB 00:05 2024-03-18T16:14:27.018 INFO:teuthology.orchestra.run.smithi154.stdout:(31/196): libradosstriper1-19.0.0-2162.g3d78f5b 4.5 MB/s | 501 kB 00:00 2024-03-18T16:14:29.385 INFO:teuthology.orchestra.run.smithi154.stdout:(32/196): ceph-osd-debuginfo-19.0.0-2162.g3d78f 6.6 MB/s | 290 MB 00:43 2024-03-18T16:14:30.268 INFO:teuthology.orchestra.run.smithi154.stdout:(33/196): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 5.8 MB/s | 5.1 MB 00:00 2024-03-18T16:14:30.393 INFO:teuthology.orchestra.run.smithi154.stdout:(34/196): python3-ceph-argparse-19.0.0-2162.g3d 361 kB/s | 45 kB 00:00 2024-03-18T16:14:30.519 INFO:teuthology.orchestra.run.smithi154.stdout:(35/196): python3-ceph-common-19.0.0-2162.g3d78 1.0 MB/s | 131 kB 00:00 2024-03-18T16:14:30.627 INFO:teuthology.orchestra.run.smithi154.stdout:(36/196): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.5 MB/s | 162 kB 00:00 2024-03-18T16:14:30.769 INFO:teuthology.orchestra.run.smithi154.stdout:(37/196): python3-cephfs-debuginfo-19.0.0-2162. 4.2 MB/s | 615 kB 00:00 2024-03-18T16:14:30.878 INFO:teuthology.orchestra.run.smithi154.stdout:(38/196): python3-rados-19.0.0-2162.g3d78f5ba.e 2.9 MB/s | 320 kB 00:00 2024-03-18T16:14:31.078 INFO:teuthology.orchestra.run.smithi154.stdout:(39/196): python3-rados-debuginfo-19.0.0-2162.g 3.8 MB/s | 770 kB 00:00 2024-03-18T16:14:31.187 INFO:teuthology.orchestra.run.smithi154.stdout:(40/196): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.7 MB/s | 298 kB 00:00 2024-03-18T16:14:31.404 INFO:teuthology.orchestra.run.smithi154.stdout:(41/196): python3-rbd-debuginfo-19.0.0-2162.g3d 5.2 MB/s | 1.1 MB 00:00 2024-03-18T16:14:31.504 INFO:teuthology.orchestra.run.smithi154.stdout:(42/196): python3-rgw-19.0.0-2162.g3d78f5ba.el9 996 kB/s | 99 kB 00:00 2024-03-18T16:14:31.637 INFO:teuthology.orchestra.run.smithi154.stdout:(43/196): python3-rgw-debuginfo-19.0.0-2162.g3d 2.5 MB/s | 337 kB 00:00 2024-03-18T16:14:31.738 INFO:teuthology.orchestra.run.smithi154.stdout:(44/196): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 853 kB/s | 85 kB 00:00 2024-03-18T16:14:31.905 INFO:teuthology.orchestra.run.smithi154.stdout:(45/196): rbd-fuse-debuginfo-19.0.0-2162.g3d78f 3.0 MB/s | 516 kB 00:00 2024-03-18T16:14:32.397 INFO:teuthology.orchestra.run.smithi154.stdout:(46/196): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 6.3 MB/s | 3.1 MB 00:00 2024-03-18T16:14:33.581 INFO:teuthology.orchestra.run.smithi154.stdout:(47/196): librbd1-debuginfo-19.0.0-2162.g3d78f5 8.0 MB/s | 52 MB 00:06 2024-03-18T16:14:33.631 INFO:teuthology.orchestra.run.smithi154.stdout:(48/196): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 3.4 MB/s | 170 kB 00:00 2024-03-18T16:14:33.898 INFO:teuthology.orchestra.run.smithi154.stdout:(49/196): rbd-nbd-debuginfo-19.0.0-2162.g3d78f5 4.8 MB/s | 1.3 MB 00:00 2024-03-18T16:14:33.990 INFO:teuthology.orchestra.run.smithi154.stdout:(50/196): ceph-grafana-dashboards-19.0.0-2162.g 371 kB/s | 34 kB 00:00 2024-03-18T16:14:34.073 INFO:teuthology.orchestra.run.smithi154.stdout:(51/196): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.6 MB/s | 139 kB 00:00 2024-03-18T16:14:34.365 INFO:teuthology.orchestra.run.smithi154.stdout:(52/196): ceph-mgr-dashboard-19.0.0-2162.g3d78f 6.2 MB/s | 1.8 MB 00:00 2024-03-18T16:14:35.207 INFO:teuthology.orchestra.run.smithi154.stdout:(53/196): ceph-mgr-diskprediction-local-19.0.0- 8.8 MB/s | 7.4 MB 00:00 2024-03-18T16:14:35.291 INFO:teuthology.orchestra.run.smithi154.stdout:(54/196): ceph-mgr-modules-core-19.0.0-2162.g3d 2.9 MB/s | 248 kB 00:00 2024-03-18T16:14:35.391 INFO:teuthology.orchestra.run.smithi154.stdout:(55/196): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 494 kB/s | 49 kB 00:00 2024-03-18T16:14:35.458 INFO:teuthology.orchestra.run.smithi154.stdout:(56/196): ceph-prometheus-alerts-19.0.0-2162.g3 246 kB/s | 16 kB 00:00 2024-03-18T16:14:35.542 INFO:teuthology.orchestra.run.smithi154.stdout:(57/196): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.1 MB/s | 264 kB 00:00 2024-03-18T16:14:35.842 INFO:teuthology.orchestra.run.smithi154.stdout:(58/196): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 2.4 MB/s | 748 kB 00:00 2024-03-18T16:14:35.925 INFO:teuthology.orchestra.run.smithi154.stdout:(59/196): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 220 kB/s | 18 kB 00:00 2024-03-18T16:14:36.118 INFO:teuthology.orchestra.run.smithi154.stdout:(60/196): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-18T16:14:36.193 INFO:teuthology.orchestra.run.smithi154.stdout:(61/196): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 920 kB 00:00 2024-03-18T16:14:36.235 INFO:teuthology.orchestra.run.smithi154.stdout:(62/196): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-18T16:14:36.285 INFO:teuthology.orchestra.run.smithi154.stdout:(63/196): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-18T16:14:36.327 INFO:teuthology.orchestra.run.smithi154.stdout:(64/196): flexiblas-3.0.4-8.el9.x86_64.rpm 758 kB/s | 32 kB 00:00 2024-03-18T16:14:36.439 INFO:teuthology.orchestra.run.smithi154.stdout:(65/196): flexiblas-netlib-3.0.4-8.el9.x86_64.r 27 MB/s | 3.0 MB 00:00 2024-03-18T16:14:36.473 INFO:teuthology.orchestra.run.smithi154.stdout:(66/196): flexiblas-openblas-openmp-3.0.4-8.el9 513 kB/s | 17 kB 00:00 2024-03-18T16:14:36.506 INFO:teuthology.orchestra.run.smithi154.stdout:(67/196): libaio-devel-0.3.111-13.el9.x86_64.rp 304 kB/s | 10 kB 00:00 2024-03-18T16:14:36.557 INFO:teuthology.orchestra.run.smithi154.stdout:(68/196): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-18T16:14:36.590 INFO:teuthology.orchestra.run.smithi154.stdout:(69/196): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T16:14:36.641 INFO:teuthology.orchestra.run.smithi154.stdout:(70/196): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T16:14:36.691 INFO:teuthology.orchestra.run.smithi154.stdout:(71/196): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-18T16:14:36.724 INFO:teuthology.orchestra.run.smithi154.stdout:(72/196): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-18T16:14:36.766 INFO:teuthology.orchestra.run.smithi154.stdout:(73/196): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-18T16:14:36.800 INFO:teuthology.orchestra.run.smithi154.stdout:(74/196): lua-5.4.4-4.el9.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2024-03-18T16:14:36.842 INFO:teuthology.orchestra.run.smithi154.stdout:(75/196): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-18T16:14:36.884 INFO:teuthology.orchestra.run.smithi154.stdout:(76/196): openblas-0.3.21-2.el9.x86_64.rpm 818 kB/s | 34 kB 00:00 2024-03-18T16:14:37.035 INFO:teuthology.orchestra.run.smithi154.stdout:(77/196): openblas-openmp-0.3.21-2.el9.x86_64.r 32 MB/s | 4.7 MB 00:00 2024-03-18T16:14:37.102 INFO:teuthology.orchestra.run.smithi154.stdout:(78/196): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-18T16:14:37.152 INFO:teuthology.orchestra.run.smithi154.stdout:(79/196): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-18T16:14:37.194 INFO:teuthology.orchestra.run.smithi154.stdout:(80/196): postgresql-private-libs-13.11-1.el9.x 3.1 MB/s | 134 kB 00:00 2024-03-18T16:14:37.386 INFO:teuthology.orchestra.run.smithi154.stdout:(81/196): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-18T16:14:37.445 INFO:teuthology.orchestra.run.smithi154.stdout:(82/196): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-03-18T16:14:37.486 INFO:teuthology.orchestra.run.smithi154.stdout:(83/196): python3-attrs-20.3.0-7.el9.noarch.rpm 1.8 MB/s | 78 kB 00:00 2024-03-18T16:14:37.653 INFO:teuthology.orchestra.run.smithi154.stdout:(84/196): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-18T16:14:37.695 INFO:teuthology.orchestra.run.smithi154.stdout:(85/196): python3-jsonpatch-1.21-16.el9.noarch. 586 kB/s | 24 kB 00:00 2024-03-18T16:14:37.737 INFO:teuthology.orchestra.run.smithi154.stdout:(86/196): python3-jsonpointer-2.0-4.el9.noarch. 434 kB/s | 18 kB 00:00 2024-03-18T16:14:37.779 INFO:teuthology.orchestra.run.smithi154.stdout:(87/196): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-18T16:14:37.821 INFO:teuthology.orchestra.run.smithi154.stdout:(88/196): python3-markupsafe-1.1.1-12.el9.x86_6 755 kB/s | 32 kB 00:00 2024-03-18T16:14:37.963 INFO:teuthology.orchestra.run.smithi154.stdout:(89/196): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-18T16:14:38.005 INFO:teuthology.orchestra.run.smithi154.stdout:(90/196): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-18T16:14:38.047 INFO:teuthology.orchestra.run.smithi154.stdout:(91/196): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-18T16:14:38.089 INFO:teuthology.orchestra.run.smithi154.stdout:(92/196): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-18T16:14:38.123 INFO:teuthology.orchestra.run.smithi154.stdout:(93/196): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-18T16:14:38.167 INFO:teuthology.orchestra.run.smithi154.stdout:(94/196): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 238 kB 00:00 2024-03-18T16:14:38.209 INFO:teuthology.orchestra.run.smithi154.stdout:(95/196): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2024-03-18T16:14:38.251 INFO:teuthology.orchestra.run.smithi154.stdout:(96/196): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-18T16:14:38.293 INFO:teuthology.orchestra.run.smithi154.stdout:(97/196): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-18T16:14:38.335 INFO:teuthology.orchestra.run.smithi154.stdout:(98/196): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-18T16:14:38.760 INFO:teuthology.orchestra.run.smithi154.stdout:(99/196): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-18T16:14:38.794 INFO:teuthology.orchestra.run.smithi154.stdout:(100/196): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-18T16:14:38.827 INFO:teuthology.orchestra.run.smithi154.stdout:(101/196): python3-wcwidth-0.2.5-8.el9.noarch.r 1.2 MB/s | 41 kB 00:00 2024-03-18T16:14:38.869 INFO:teuthology.orchestra.run.smithi154.stdout:(102/196): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-18T16:14:38.903 INFO:teuthology.orchestra.run.smithi154.stdout:(103/196): uuid-1.6.2-55.el9.x86_64.rpm 1.6 MB/s | 56 kB 00:00 2024-03-18T16:14:38.937 INFO:teuthology.orchestra.run.smithi154.stdout:(104/196): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-18T16:14:38.970 INFO:teuthology.orchestra.run.smithi154.stdout:(105/196): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T16:14:38.995 INFO:teuthology.orchestra.run.smithi154.stdout:(106/196): fuse-2.9.9-15.el9.x86_64.rpm 3.1 MB/s | 79 kB 00:00 2024-03-18T16:14:39.037 INFO:teuthology.orchestra.run.smithi154.stdout:(107/196): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-18T16:14:39.105 INFO:teuthology.orchestra.run.smithi154.stdout:(108/196): python3-cryptography-36.0.1-4.el9.x8 17 MB/s | 1.1 MB 00:00 2024-03-18T16:14:39.146 INFO:teuthology.orchestra.run.smithi154.stdout:(109/196): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-18T16:14:39.188 INFO:teuthology.orchestra.run.smithi154.stdout:(110/196): python3-pycparser-2.20-6.el9.noarch. 2.9 MB/s | 124 kB 00:00 2024-03-18T16:14:39.230 INFO:teuthology.orchestra.run.smithi154.stdout:(111/196): python3-urllib3-1.26.5-5.el9.noarch. 4.4 MB/s | 187 kB 00:00 2024-03-18T16:14:39.789 INFO:teuthology.orchestra.run.smithi154.stdout:(112/196): jq-1.6-16.el9.x86_64.rpm 335 kB/s | 187 kB 00:00 2024-03-18T16:14:40.131 INFO:teuthology.orchestra.run.smithi154.stdout:(113/196): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-18T16:14:40.298 INFO:teuthology.orchestra.run.smithi154.stdout:(114/196): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-18T16:14:40.465 INFO:teuthology.orchestra.run.smithi154.stdout:(115/196): python3-requests-2.25.1-8.el9.noarch 750 kB/s | 125 kB 00:00 2024-03-18T16:14:41.640 INFO:teuthology.orchestra.run.smithi154.stdout:(116/196): elfutils-libelf-devel-0.190-2.el9.x8 18 kB/s | 21 kB 00:01 2024-03-18T16:14:41.707 INFO:teuthology.orchestra.run.smithi154.stdout:(117/196): libuuid-devel-2.37.4-18.el9.x86_64.r 382 kB/s | 25 kB 00:00 2024-03-18T16:14:42.307 INFO:teuthology.orchestra.run.smithi154.stdout:(118/196): rbd-mirror-debuginfo-19.0.0-2162.g3d 5.2 MB/s | 52 MB 00:09 2024-03-18T16:14:42.499 INFO:teuthology.orchestra.run.smithi154.stdout:(119/196): openssl-devel-3.0.7-27.el9.x86_64.rp 5.2 MB/s | 4.1 MB 00:00 2024-03-18T16:14:42.524 INFO:teuthology.orchestra.run.smithi154.stdout:(120/196): python3-devel-3.9.17-2.el9.x86_64.rp 1.1 MB/s | 245 kB 00:00 2024-03-18T16:14:42.667 INFO:teuthology.orchestra.run.smithi154.stdout:(121/196): python3-jinja2-2.11.3-5.el9.noarch.r 1.5 MB/s | 248 kB 00:00 2024-03-18T16:14:42.692 INFO:teuthology.orchestra.run.smithi154.stdout:(122/196): qatlib-23.11.0-1.el9.x86_64.rpm 1.3 MB/s | 216 kB 00:00 2024-03-18T16:14:42.750 INFO:teuthology.orchestra.run.smithi154.stdout:(123/196): qatlib-service-23.11.0-1.el9.x86_64. 418 kB/s | 35 kB 00:00 2024-03-18T16:14:42.801 INFO:teuthology.orchestra.run.smithi154.stdout:(124/196): qatzip-libs-1.1.2-2.el9.x86_64.rpm 410 kB/s | 44 kB 00:00 2024-03-18T16:14:42.918 INFO:teuthology.orchestra.run.smithi154.stdout:(125/196): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 562 kB/s | 94 kB 00:00 2024-03-18T16:14:42.943 INFO:teuthology.orchestra.run.smithi154.stdout:(126/196): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-18T16:14:43.085 INFO:teuthology.orchestra.run.smithi154.stdout:(127/196): protobuf-compiler-3.14.0-13.el9.x86_ 5.1 MB/s | 863 kB 00:00 2024-03-18T16:14:43.135 INFO:teuthology.orchestra.run.smithi154.stdout:(128/196): abseil-cpp-20211102.0-3.el9.x86_64.r 2.8 MB/s | 548 kB 00:00 2024-03-18T16:14:43.177 INFO:teuthology.orchestra.run.smithi154.stdout:(129/196): grpc-data-1.46.7-10.el9.noarch.rpm 467 kB/s | 19 kB 00:00 2024-03-18T16:14:43.236 INFO:teuthology.orchestra.run.smithi154.stdout:(130/196): gperftools-libs-2.9.1-2.el9.x86_64.r 2.0 MB/s | 309 kB 00:00 2024-03-18T16:14:43.277 INFO:teuthology.orchestra.run.smithi154.stdout:(131/196): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-03-18T16:14:43.319 INFO:teuthology.orchestra.run.smithi154.stdout:(132/196): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-18T16:14:43.386 INFO:teuthology.orchestra.run.smithi154.stdout:(133/196): libarrow-9.0.0-10.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-03-18T16:14:43.412 INFO:teuthology.orchestra.run.smithi154.stdout:(134/196): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-03-18T16:14:43.437 INFO:teuthology.orchestra.run.smithi154.stdout:(135/196): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-03-18T16:14:43.495 INFO:teuthology.orchestra.run.smithi154.stdout:(136/196): python3-asyncssh-2.13.2-4.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-03-18T16:14:43.529 INFO:teuthology.orchestra.run.smithi154.stdout:(137/196): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-18T16:14:43.554 INFO:teuthology.orchestra.run.smithi154.stdout:(138/196): python3-bcrypt-3.2.2-1.el9.x86_64.rp 742 kB/s | 43 kB 00:00 2024-03-18T16:14:43.580 INFO:teuthology.orchestra.run.smithi154.stdout:(139/196): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-18T16:14:43.596 INFO:teuthology.orchestra.run.smithi154.stdout:(140/196): python3-certifi-2021.10.8-2.el9.noar 358 kB/s | 15 kB 00:00 2024-03-18T16:14:43.622 INFO:teuthology.orchestra.run.smithi154.stdout:(141/196): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-03-18T16:14:43.647 INFO:teuthology.orchestra.run.smithi154.stdout:(142/196): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-18T16:14:43.672 INFO:teuthology.orchestra.run.smithi154.stdout:(143/196): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-03-18T16:14:43.689 INFO:teuthology.orchestra.run.smithi154.stdout:(144/196): python3-colorama-0.4.4-12.el9.noarch 846 kB/s | 35 kB 00:00 2024-03-18T16:14:43.714 INFO:teuthology.orchestra.run.smithi154.stdout:(145/196): python3-google-auth-2.28.0-1.el9.noa 4.7 MB/s | 203 kB 00:00 2024-03-18T16:14:43.765 INFO:teuthology.orchestra.run.smithi154.stdout:(146/196): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-03-18T16:14:43.815 INFO:teuthology.orchestra.run.smithi154.stdout:(147/196): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-03-18T16:14:43.832 INFO:teuthology.orchestra.run.smithi154.stdout:(148/196): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-18T16:14:43.849 INFO:teuthology.orchestra.run.smithi154.stdout:(149/196): python3-jaraco-classes-3.2.1-5.el9.n 532 kB/s | 18 kB 00:00 2024-03-18T16:14:43.866 INFO:teuthology.orchestra.run.smithi154.stdout:(150/196): python3-jaraco-collections-3.0.0-8.e 692 kB/s | 23 kB 00:00 2024-03-18T16:14:43.883 INFO:teuthology.orchestra.run.smithi154.stdout:(151/196): python3-jaraco-functools-3.5.0-2.el9 574 kB/s | 19 kB 00:00 2024-03-18T16:14:43.908 INFO:teuthology.orchestra.run.smithi154.stdout:(152/196): python3-jaraco-text-3.2.0-6.el9.noar 465 kB/s | 20 kB 00:00 2024-03-18T16:14:43.942 INFO:teuthology.orchestra.run.smithi154.stdout:(153/196): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-03-18T16:14:43.959 INFO:teuthology.orchestra.run.smithi154.stdout:(154/196): python3-logutils-0.3.5-21.el9.noarch 924 kB/s | 46 kB 00:00 2024-03-18T16:14:43.992 INFO:teuthology.orchestra.run.smithi154.stdout:(155/196): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-18T16:14:44.010 INFO:teuthology.orchestra.run.smithi154.stdout:(156/196): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-18T16:14:44.035 INFO:teuthology.orchestra.run.smithi154.stdout:(157/196): python3-pecan-1.4.2-3.el9.noarch.rpm 6.3 MB/s | 272 kB 00:00 2024-03-18T16:14:44.060 INFO:teuthology.orchestra.run.smithi154.stdout:(158/196): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-03-18T16:14:44.085 INFO:teuthology.orchestra.run.smithi154.stdout:(159/196): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-18T16:14:44.111 INFO:teuthology.orchestra.run.smithi154.stdout:(160/196): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-03-18T16:14:44.136 INFO:teuthology.orchestra.run.smithi154.stdout:(161/196): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-03-18T16:14:44.161 INFO:teuthology.orchestra.run.smithi154.stdout:(162/196): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-03-18T16:14:44.186 INFO:teuthology.orchestra.run.smithi154.stdout:(163/196): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-03-18T16:14:44.212 INFO:teuthology.orchestra.run.smithi154.stdout:(164/196): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-03-18T16:14:44.237 INFO:teuthology.orchestra.run.smithi154.stdout:(165/196): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-18T16:14:44.262 INFO:teuthology.orchestra.run.smithi154.stdout:(166/196): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-18T16:14:44.287 INFO:teuthology.orchestra.run.smithi154.stdout:(167/196): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-18T16:14:44.313 INFO:teuthology.orchestra.run.smithi154.stdout:(168/196): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-03-18T16:14:44.338 INFO:teuthology.orchestra.run.smithi154.stdout:(169/196): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-03-18T16:14:44.363 INFO:teuthology.orchestra.run.smithi154.stdout:(170/196): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-18T16:14:44.422 INFO:teuthology.orchestra.run.smithi154.stdout:(171/196): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-18T16:14:45.314 INFO:teuthology.orchestra.run.smithi154.stdout:(172/196): librados2-19.0.0-2162.g3d78f5ba.el9. 3.6 MB/s | 3.4 MB 00:00 2024-03-18T16:14:45.364 INFO:teuthology.orchestra.run.smithi154.stdout:(173/196): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-18T16:14:46.215 INFO:teuthology.orchestra.run.smithi154.stdout:(174/196): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 1.8 MB/s | 3.1 MB 00:01 2024-03-18T16:14:46.665 INFO:teuthology.orchestra.run.smithi154.stdout:(175/196): NetworkManager-libnm-1.46.0-1.el9.x8 4.0 MB/s | 1.8 MB 00:00 2024-03-18T16:14:46.740 INFO:teuthology.orchestra.run.smithi154.stdout:(176/196): NetworkManager-1.46.0-1.el9.x86_64.r 1.7 MB/s | 2.3 MB 00:01 2024-03-18T16:14:46.756 INFO:teuthology.orchestra.run.smithi154.stdout:(177/196): NetworkManager-team-1.46.0-1.el9.x86 462 kB/s | 42 kB 00:00 2024-03-18T16:14:46.882 INFO:teuthology.orchestra.run.smithi154.stdout:(178/196): NetworkManager-tui-1.46.0-1.el9.x86_ 1.7 MB/s | 248 kB 00:00 2024-03-18T16:14:46.907 INFO:teuthology.orchestra.run.smithi154.stdout:(179/196): acl-2.3.1-4.el9.x86_64.rpm 475 kB/s | 71 kB 00:00 2024-03-18T16:14:46.966 INFO:teuthology.orchestra.run.smithi154.stdout:(180/196): elfutils-debuginfod-client-0.190-2.e 436 kB/s | 36 kB 00:00 2024-03-18T16:14:47.016 INFO:teuthology.orchestra.run.smithi154.stdout:(181/196): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-18T16:14:47.117 INFO:teuthology.orchestra.run.smithi154.stdout:(182/196): elfutils-libs-0.190-2.el9.x86_64.rpm 1.7 MB/s | 256 kB 00:00 2024-03-18T16:14:47.133 INFO:teuthology.orchestra.run.smithi154.stdout:(183/196): libacl-2.3.1-4.el9.x86_64.rpm 197 kB/s | 23 kB 00:00 2024-03-18T16:14:47.209 INFO:teuthology.orchestra.run.smithi154.stdout:(184/196): libblkid-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 108 kB 00:00 2024-03-18T16:14:47.234 INFO:teuthology.orchestra.run.smithi154.stdout:(185/196): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-18T16:14:47.301 INFO:teuthology.orchestra.run.smithi154.stdout:(186/196): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-18T16:14:47.326 INFO:teuthology.orchestra.run.smithi154.stdout:(187/196): libsmartcols-2.37.4-18.el9.x86_64.rp 695 kB/s | 64 kB 00:00 2024-03-18T16:14:47.394 INFO:teuthology.orchestra.run.smithi154.stdout:(188/196): libuuid-2.37.4-18.el9.x86_64.rpm 312 kB/s | 29 kB 00:00 2024-03-18T16:14:47.443 INFO:teuthology.orchestra.run.smithi154.stdout:(189/196): openssl-3.0.7-27.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2024-03-18T16:14:47.603 INFO:teuthology.orchestra.run.smithi154.stdout:(190/196): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-03-18T16:14:47.711 INFO:teuthology.orchestra.run.smithi154.stdout:(191/196): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-18T16:14:47.803 INFO:teuthology.orchestra.run.smithi154.stdout:(192/196): util-linux-user-2.37.4-18.el9.x86_64 340 kB/s | 31 kB 00:00 2024-03-18T16:14:47.853 INFO:teuthology.orchestra.run.smithi154.stdout:(193/196): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.7 MB/s | 2.2 MB 00:00 2024-03-18T16:14:47.937 INFO:teuthology.orchestra.run.smithi154.stdout:(194/196): libacl-devel-2.3.1-4.el9.x86_64.rpm 895 kB/s | 74 kB 00:00 2024-03-18T16:14:47.995 INFO:teuthology.orchestra.run.smithi154.stdout:(195/196): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.6 MB/s | 1.1 MB 00:00 2024-03-18T16:14:56.911 INFO:teuthology.orchestra.run.smithi154.stdout:(196/196): ceph-test-debuginfo-19.0.0-2162.g3d7 18 MB/s | 679 MB 00:38 2024-03-18T16:14:56.918 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-03-18T16:14:56.918 INFO:teuthology.orchestra.run.smithi154.stdout:Total 22 MB/s | 2.2 GB 01:40 2024-03-18T16:14:57.368 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-03-18T16:14:57.648 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-03-18T16:14:57.648 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-03-18T16:15:11.220 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-03-18T16:15:11.220 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-03-18T16:15:30.488 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-03-18T16:15:33.077 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/220 2024-03-18T16:15:33.102 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_6 2/220 2024-03-18T16:15:33.127 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 3/220 2024-03-18T16:15:33.167 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-18T16:15:33.176 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-18T16:15:33.216 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/220 2024-03-18T16:15:33.269 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/220 2024-03-18T16:15:33.312 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/220 2024-03-18T16:15:33.338 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/220 2024-03-18T16:15:33.380 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/220 2024-03-18T16:15:33.702 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/220 2024-03-18T16:15:33.763 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/220 2024-03-18T16:15:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/220 2024-03-18T16:15:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/220 2024-03-18T16:15:33.989 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/220 2024-03-18T16:15:34.018 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/220 2024-03-18T16:15:34.065 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/220 2024-03-18T16:15:34.242 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 15/220 2024-03-18T16:15:34.291 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-13.11-1.el9.x86_64 16/220 2024-03-18T16:15:34.319 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/220 2024-03-18T16:15:34.423 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/220 2024-03-18T16:15:34.445 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 18/220 2024-03-18T16:15:34.492 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 18/220 2024-03-18T16:15:34.532 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 19/220 2024-03-18T16:15:34.555 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-18T16:15:34.570 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-18T16:15:34.755 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-18T16:15:34.785 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-18T16:15:34.817 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 22/220 2024-03-18T16:15:34.913 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 23/220 2024-03-18T16:15:34.995 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 24/220 2024-03-18T16:15:35.055 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 24/220 2024-03-18T16:15:35.120 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 25/220 2024-03-18T16:15:35.181 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 25/220 2024-03-18T16:15:35.377 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 25/220 2024-03-18T16:15:35.485 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : re2-1:20211101-3.el9.x86_64 26/220 2024-03-18T16:15:35.581 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 27/220 2024-03-18T16:15:35.610 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 28/220 2024-03-18T16:15:35.637 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 29/220 2024-03-18T16:15:35.673 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/220 2024-03-18T16:15:35.747 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/220 2024-03-18T16:15:35.819 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/220 2024-03-18T16:15:35.864 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/220 2024-03-18T16:15:35.929 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 34/220 2024-03-18T16:15:35.963 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 35/220 2024-03-18T16:15:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 36/220 2024-03-18T16:15:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 37/220 2024-03-18T16:15:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-5.4.4-4.el9.x86_64 38/220 2024-03-18T16:15:36.143 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 39/220 2024-03-18T16:15:36.180 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 40/220 2024-03-18T16:15:36.223 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 41/220 2024-03-18T16:15:36.252 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 42/220 2024-03-18T16:15:36.321 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 43/220 2024-03-18T16:15:36.367 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : luarocks-3.9.2-1.el9.noarch 44/220 2024-03-18T16:15:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 45/220 2024-03-18T16:15:36.582 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 46/220 2024-03-18T16:15:36.636 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 47/220 2024-03-18T16:15:37.154 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/220 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 49/220 2024-03-18T16:15:37.318 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 50/220 2024-03-18T16:15:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-18T16:15:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-18T16:15:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 52/220 2024-03-18T16:15:37.472 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 53/220 2024-03-18T16:15:37.511 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 54/220 2024-03-18T16:15:37.541 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 55/220 2024-03-18T16:15:37.768 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/220 2024-03-18T16:15:37.946 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/220 2024-03-18T16:15:38.001 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/220 2024-03-18T16:15:38.045 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/220 2024-03-18T16:15:38.177 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/220 2024-03-18T16:15:38.223 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/220 2024-03-18T16:15:38.277 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/220 2024-03-18T16:15:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/220 2024-03-18T16:15:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/220 2024-03-18T16:15:38.418 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/220 2024-03-18T16:15:38.465 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/220 2024-03-18T16:15:38.501 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/220 2024-03-18T16:15:38.543 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/220 2024-03-18T16:15:38.595 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/220 2024-03-18T16:15:38.729 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/220 2024-03-18T16:15:38.766 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/220 2024-03-18T16:15:38.797 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/220 2024-03-18T16:15:38.838 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/220 2024-03-18T16:15:38.932 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/220 2024-03-18T16:15:38.971 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/220 2024-03-18T16:15:39.020 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/220 2024-03-18T16:15:39.045 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/220 2024-03-18T16:15:39.070 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/220 2024-03-18T16:15:39.141 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/220 2024-03-18T16:15:39.271 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/220 2024-03-18T16:15:39.371 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/220 2024-03-18T16:15:39.420 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/220 2024-03-18T16:15:39.493 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/220 2024-03-18T16:15:39.548 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/220 2024-03-18T16:15:39.735 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/220 2024-03-18T16:15:39.782 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/220 2024-03-18T16:15:39.911 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/220 2024-03-18T16:15:39.947 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/220 2024-03-18T16:15:39.979 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/220 2024-03-18T16:15:40.010 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-15.el9.x86_64 90/220 2024-03-18T16:15:40.037 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 91/220 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 92/220 2024-03-18T16:15:40.111 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : uuid-1.6.2-55.el9.x86_64 93/220 2024-03-18T16:15:40.142 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 94/220 2024-03-18T16:15:40.173 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 95/220 2024-03-18T16:15:40.258 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 96/220 2024-03-18T16:15:40.291 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 97/220 2024-03-18T16:15:40.324 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 98/220 2024-03-18T16:15:40.366 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 99/220 2024-03-18T16:15:40.401 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 100/220 2024-03-18T16:15:40.490 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 101/220 2024-03-18T16:15:40.580 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 102/220 2024-03-18T16:15:40.623 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 103/220 2024-03-18T16:15:40.654 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 104/220 2024-03-18T16:15:40.702 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 105/220 2024-03-18T16:15:40.800 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-packaging-20.9-5.el9.noarch 106/220 2024-03-18T16:15:40.847 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 107/220 2024-03-18T16:15:41.580 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 108/220 2024-03-18T16:15:41.628 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 109/220 2024-03-18T16:15:41.661 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 110/220 2024-03-18T16:15:41.710 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 111/220 2024-03-18T16:15:41.800 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 112/220 2024-03-18T16:15:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 113/220 2024-03-18T16:15:42.021 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.21-2.el9.x86_64 114/220 2024-03-18T16:15:42.051 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 115/220 2024-03-18T16:15:42.128 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 116/220 2024-03-18T16:15:42.768 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 117/220 2024-03-18T16:15:42.882 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 118/220 2024-03-18T16:15:44.100 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 119/220 2024-03-18T16:15:44.185 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 120/220 2024-03-18T16:15:44.215 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 121/220 2024-03-18T16:15:44.247 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : jq-1.6-16.el9.x86_64 122/220 2024-03-18T16:15:44.274 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/220 2024-03-18T16:15:44.586 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/220 2024-03-18T16:15:44.612 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/220 2024-03-18T16:15:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/220 2024-03-18T16:15:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 126/220 2024-03-18T16:15:44.761 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/220 2024-03-18T16:15:45.219 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/220 2024-03-18T16:15:45.249 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/220 2024-03-18T16:15:45.303 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/220 2024-03-18T16:15:45.325 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/220 2024-03-18T16:15:47.568 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/220 2024-03-18T16:15:47.605 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/220 2024-03-18T16:15:48.143 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/220 2024-03-18T16:15:48.191 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/220 2024-03-18T16:15:48.256 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/220 2024-03-18T16:15:48.256 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T16:15:48.256 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T16:15:48.257 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:15:48.284 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 133/220 2024-03-18T16:16:01.265 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 133/220 2024-03-18T16:16:01.266 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-03-18T16:16:01.266 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-03-18T16:16:01.266 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-03-18T16:16:01.266 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-03-18T16:16:01.266 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-03-18T16:16:01.266 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2024-03-18T16:16:01.266 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-03-18T16:16:01.266 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:01.432 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 134/220 2024-03-18T16:16:01.447 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 134/220 2024-03-18T16:16:01.447 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:16:01.447 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T16:16:01.447 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:02.926 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/220 2024-03-18T16:16:02.952 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/220 2024-03-18T16:16:02.952 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:16:02.952 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T16:16:02.952 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T16:16:02.952 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T16:16:02.952 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:03.177 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/220 2024-03-18T16:16:03.199 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/220 2024-03-18T16:16:03.199 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:16:03.199 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T16:16:03.199 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T16:16:03.199 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T16:16:03.199 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:03.644 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 137/220 2024-03-18T16:16:03.666 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 137/220 2024-03-18T16:16:03.666 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:16:03.666 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T16:16:03.666 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T16:16:03.667 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T16:16:03.667 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : automake-1.16.2-8.el9.noarch 138/220 2024-03-18T16:16:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 139/220 2024-03-18T16:16:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 139/220 2024-03-18T16:16:04.259 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 140/220 2024-03-18T16:16:04.320 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 141/220 2024-03-18T16:16:04.342 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 142/220 2024-03-18T16:16:04.576 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 142/220 2024-03-18T16:16:04.610 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 143/220 2024-03-18T16:16:05.571 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 143/220 2024-03-18T16:16:05.601 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 144/220 2024-03-18T16:16:05.703 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 144/220 2024-03-18T16:16:05.864 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 145/220 2024-03-18T16:16:05.896 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 146/220 2024-03-18T16:16:05.919 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 146/220 2024-03-18T16:16:05.919 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:16:05.919 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T16:16:05.919 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T16:16:05.919 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T16:16:05.919 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:05.963 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 147/220 2024-03-18T16:16:05.973 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 147/220 2024-03-18T16:16:06.018 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 148/220 2024-03-18T16:16:06.893 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/220 2024-03-18T16:16:06.923 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 150/220 2024-03-18T16:16:06.945 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 150/220 2024-03-18T16:16:06.945 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:16:06.945 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T16:16:06.946 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T16:16:06.946 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T16:16:06.946 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:06.999 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 151/220 2024-03-18T16:16:07.020 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 151/220 2024-03-18T16:16:07.020 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:16:07.021 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T16:16:07.021 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:07.074 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/220 2024-03-18T16:16:07.096 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/220 2024-03-18T16:16:07.096 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:16:07.097 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T16:16:07.097 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T16:16:07.097 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T16:16:07.097 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:07.408 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/220 2024-03-18T16:16:07.431 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/220 2024-03-18T16:16:07.431 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T16:16:07.431 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T16:16:07.431 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T16:16:07.431 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T16:16:07.431 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:16:11.871 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/220 2024-03-18T16:16:11.912 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/220 2024-03-18T16:16:11.943 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 156/220 2024-03-18T16:16:11.970 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/220 2024-03-18T16:16:12.154 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 158/220 2024-03-18T16:16:12.269 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/220 2024-03-18T16:16:12.310 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 160/220 2024-03-18T16:16:12.358 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/220 2024-03-18T16:16:12.427 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/220 2024-03-18T16:16:12.470 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/220 2024-03-18T16:16:12.503 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 164/220 2024-03-18T16:16:12.535 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/220 2024-03-18T16:16:12.564 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 166/220 2024-03-18T16:16:12.619 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 167/220 2024-03-18T16:16:12.680 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 168/220 2024-03-18T16:16:12.713 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/220 2024-03-18T16:16:13.285 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-18T16:16:13.327 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-18T16:16:13.347 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-18T16:16:20.496 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 171/220 2024-03-18T16:17:02.676 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 172/220 2024-03-18T16:17:03.314 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2162.g3d78f5ba.el 173/220 2024-03-18T16:17:03.482 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 174/220 2024-03-18T16:17:08.415 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 175/220 2024-03-18T16:17:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 176/220 2024-03-18T16:17:20.380 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 177/220 2024-03-18T16:17:48.611 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 178/220 2024-03-18T16:18:05.789 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 179/220 2024-03-18T16:19:13.616 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2162.g3d78f5ba.e 180/220 2024-03-18T16:19:14.251 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 181/220 2024-03-18T16:19:14.289 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9 182/220 2024-03-18T16:19:19.092 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2162.g3d78f5ba 183/220 2024-03-18T16:19:24.848 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 184/220 2024-03-18T16:19:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librbd1-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x8 185/220 2024-03-18T16:19:25.234 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2162.g3d78f5ba 186/220 2024-03-18T16:19:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2162.g3d78f5ba. 187/220 2024-03-18T16:19:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el 188/220 2024-03-18T16:19:25.508 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2162.g3d78f5ba.el 189/220 2024-03-18T16:19:31.137 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 190/220 2024-03-18T16:19:31.510 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2162.g3d78f5ba.el9 191/220 2024-03-18T16:19:32.593 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x8 192/220 2024-03-18T16:19:32.677 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 193/220 2024-03-18T16:19:32.717 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 194/220 2024-03-18T16:19:32.805 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flex-2.6.4-9.el9.x86_64 195/220 2024-03-18T16:19:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : bison-3.7.4-5.el9.x86_64 196/220 2024-03-18T16:19:32.864 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 197/220 2024-03-18T16:19:32.889 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 198/220 2024-03-18T16:19:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 199/220 2024-03-18T16:19:32.916 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-18T16:19:33.012 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-18T16:19:33.041 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 201/220 2024-03-18T16:19:33.042 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-18T16:19:33.126 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-18T16:19:33.152 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 203/220 2024-03-18T16:19:33.185 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 204/220 2024-03-18T16:19:33.211 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 205/220 2024-03-18T16:19:33.236 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 206/220 2024-03-18T16:19:33.277 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 207/220 2024-03-18T16:19:33.325 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 208/220 2024-03-18T16:19:33.357 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 209/220 2024-03-18T16:19:33.382 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 210/220 2024-03-18T16:19:33.408 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 211/220 2024-03-18T16:19:33.441 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 212/220 2024-03-18T16:19:33.467 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 213/220 2024-03-18T16:19:33.473 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-18T16:19:33.473 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-18T16:19:33.653 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-18T16:19:33.653 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-18T16:19:33.728 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-18T16:19:33.752 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 216/220 2024-03-18T16:19:33.778 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 217/220 2024-03-18T16:19:33.811 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 218/220 2024-03-18T16:19:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 219/220 2024-03-18T16:19:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-18T16:19:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-18T16:19:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/220 2024-03-18T16:19:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/220 2024-03-18T16:19:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 3/220 2024-03-18T16:19:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/220 2024-03-18T16:19:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2162.g3d78f5ba.el 5/220 2024-03-18T16:19:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_6 6/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 8/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 9/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 10/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 12/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 14/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 15/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 16/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 18/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 19/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2162.g3d78f5ba.e 20/220 2024-03-18T16:19:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 21/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 22/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 23/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 26/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9 28/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 30/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2162.g3d78f5ba 31/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9. 32/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 33/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x8 34/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 35/220 2024-03-18T16:19:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 36/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 37/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 38/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2162.g3d78f5ba 39/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 40/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2162.g3d78f5ba. 41/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 42/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el 43/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 44/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2162.g3d78f5ba.el 45/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 46/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x 47/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 48/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2162.g3d78f5ba.el9 49/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 50/220 2024-03-18T16:19:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x8 51/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 52/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 53/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 54/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 55/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 56/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 57/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 58/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 59/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 60/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 61/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : automake-1.16.2-8.el9.noarch 62/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : bison-3.7.4-5.el9.x86_64 63/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 64/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flex-2.6.4-9.el9.x86_64 65/220 2024-03-18T16:19:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 66/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 67/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 68/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 69/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 73/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 74/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 75/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-5.4.4-4.el9.x86_64 76/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 77/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 78/220 2024-03-18T16:19:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 79/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-13.11-1.el9.x86_64 80/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 81/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 82/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 83/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 84/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 85/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 86/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 87/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 88/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 89/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 90/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 91/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 93/220 2024-03-18T16:19:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 94/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 95/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 96/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 97/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 98/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 99/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 100/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 101/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 102/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 103/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 104/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 105/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 106/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 107/220 2024-03-18T16:19:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 108/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 109/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 110/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 111/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 112/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 113/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : jq-1.6-16.el9.x86_64 114/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 115/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 116/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 117/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 118/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 119/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 120/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 121/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 122/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 123/220 2024-03-18T16:19:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 124/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 125/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 126/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 127/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 128/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 129/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 130/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 131/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 132/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 133/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 134/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 135/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 136/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 137/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 138/220 2024-03-18T16:19:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 139/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 140/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 141/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 142/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 143/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 144/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 145/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 146/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 147/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 148/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 149/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 150/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 151/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 152/220 2024-03-18T16:19:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 153/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 154/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 155/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 156/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 157/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 158/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 159/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 160/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 161/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 162/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 163/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 164/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 165/220 2024-03-18T16:19:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 166/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 167/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 168/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 169/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 170/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : re2-1:20211101-3.el9.x86_64 171/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 172/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 173/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 174/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 175/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 176/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/220 2024-03-18T16:19:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 179/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 180/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 181/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 182/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 183/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 184/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 185/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 186/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : acl-2.3.1-4.el9.x86_64 187/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : acl-2.3.1-3.el9.x86_64 188/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 189/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 190/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 191/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 192/220 2024-03-18T16:19:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 193/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 194/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 195/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 196/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 197/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 198/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 199/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 200/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 201/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 202/220 2024-03-18T16:19:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 203/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 204/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 205/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 206/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 207/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 208/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 209/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 210/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 211/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 212/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 213/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 214/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 215/220 2024-03-18T16:19:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 216/220 2024-03-18T16:19:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 217/220 2024-03-18T16:19:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 218/220 2024-03-18T16:19:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 219/220 2024-03-18T16:19:46.513 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 220/220 2024-03-18T16:19:46.513 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:46.513 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-03-18T16:19:46.513 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T16:19:46.513 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.514 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T16:19:46.515 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.516 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.517 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T16:19:46.518 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: jq-1.6-16.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T16:19:46.519 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T16:19:46.520 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T16:19:46.521 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T16:19:46.522 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T16:19:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T16:19:46.524 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T16:19:46.525 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T16:19:46.526 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-debuginfo-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T16:19:46.527 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T16:19:46.528 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T16:19:46.528 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:46.528 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-03-18T16:19:47.214 DEBUG:teuthology.parallel:result is None 2024-03-18T16:19:47.215 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T16:19:47.357 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T16:19:47.459 INFO:teuthology.orchestra.run.smithi102.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T16:19:47.459 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T16:19:47.459 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T16:19:47.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T16:19:47.706 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T16:19:47.728 INFO:teuthology.orchestra.run.smithi154.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T16:19:47.751 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T16:19:47.751 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T16:19:47.754 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T16:19:47.754 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:47.754 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T16:19:47.855 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:19:47.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T16:19:47.887 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T16:19:47.887 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:47.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T16:19:47.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T16:19:48.076 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:19:48.189 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T16:19:48.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T16:19:48.287 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T16:19:48.287 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:48.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T16:19:48.376 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T16:19:48.501 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:19:48.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T16:19:48.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T16:19:48.599 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T16:19:48.600 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:48.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T16:19:48.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T16:19:48.777 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:19:48.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T16:19:48.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T16:19:48.879 INFO:teuthology.run_tasks:Running task ceph... 2024-03-18T16:19:48.958 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-18T16:19:48.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /var/log/ceph 2024-03-18T16:19:48.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /var/log/ceph 2024-03-18T16:19:49.032 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-18T16:19:49.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-18T16:19:49.036 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-18T16:19:49.104 INFO:tasks.ceph:Creating extra log directories... 2024-03-18T16:19:49.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-18T16:19:49.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-18T16:19:49.184 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-18T16:19:49.184 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-18T16:19:49.184 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-18T16:19:49.184 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:49.184 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-18T16:19:49.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-18T16:19:49.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-18T16:19:49.613 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:19:49.614 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-18T16:19:49.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-18T16:19:49.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-18T16:19:49.831 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-18T16:19:49.831 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'ms die on skipped message': False, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-18T16:19:49.831 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908', 'branch': 'wip-batrick-testing-20240314.183243', 'description': 'fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7608908', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240314.183243'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'ms die on skipped message': False, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'client.0': {'valgrind': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240314.183243', 'sha1': '4e19f704afd9957ea657ab858f9ecf76401fef6f'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 4770, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'sleep_before_teardown': 0, 'subset': '498/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240314.183243', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph', 'suite_sha1': '4e19f704afd9957ea657ab858f9ecf76401fef6f', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi102.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCIVywnGKMz4dbv6DuAZKnwUEr2ps9arusWyrnGMF6hbhtInfttAU9qNgdINIypmvSPQBuV58PzDs5kqECCYahA=', 'smithi154.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUN4kDHCkrh0k3Tv2BDt0sRWbFTJMwli02mlbrynLiEHbZLewUIAqhOS4zfPYrE85fDkkH6Malb9EVm7+BSETI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'ms die on skipped message': False, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}, 'cluster': 'ceph'}}, {'kclient': None}, {'workunit': {'clients': {'all': ['suites/dbench.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-18_14:19:46', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-18T16:19:49.832 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-18T16:19:49.838 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-18T16:19:49.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/run/ceph 2024-03-18T16:19:49.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/run/ceph 2024-03-18T16:19:50.013 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:50.014 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T16:19:50.062 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T16:19:50.165 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-03-18T16:19:50.219 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T16:19:50.220 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:19:50.220 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 633 Links: 1 2024-03-18T16:19:50.220 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:19:50.220 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:19:50.220 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-18 16:13:09.324634051 +0000 2024-03-18T16:19:50.220 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-18 16:03:16.983083596 +0000 2024-03-18T16:19:50.220 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-18 16:03:16.983083596 +0000 2024-03-18T16:19:50.220 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-18 16:03:16.983083596 +0000 2024-03-18T16:19:50.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T16:19:50.306 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-18T16:19:50.310 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-18T16:19:50.310 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000321551 s, 1.6 MB/s 2024-03-18T16:19:50.311 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T16:19:50.363 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-03-18T16:19:50.412 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T16:19:50.412 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:19:50.412 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-18T16:19:50.412 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:19:50.412 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:19:50.412 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-18 16:13:09.324634051 +0000 2024-03-18T16:19:50.412 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-18 16:03:17.562071443 +0000 2024-03-18T16:19:50.412 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-18 16:03:17.562071443 +0000 2024-03-18T16:19:50.412 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-18 16:03:17.562071443 +0000 2024-03-18T16:19:50.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T16:19:50.494 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-18T16:19:50.494 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-18T16:19:50.494 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000354721 s, 1.4 MB/s 2024-03-18T16:19:50.498 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T16:19:50.551 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-03-18T16:19:50.598 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T16:19:50.598 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:19:50.598 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-03-18T16:19:50.598 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:19:50.599 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:19:50.599 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-18 16:13:09.325634031 +0000 2024-03-18T16:19:50.599 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-18 16:03:18.131059499 +0000 2024-03-18T16:19:50.599 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-18 16:03:18.131059499 +0000 2024-03-18T16:19:50.599 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-18 16:03:18.131059499 +0000 2024-03-18T16:19:50.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T16:19:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-18T16:19:50.801 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-18T16:19:50.801 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000360639 s, 1.4 MB/s 2024-03-18T16:19:50.802 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T16:19:50.864 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-03-18T16:19:50.915 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T16:19:50.915 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:19:50.915 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-03-18T16:19:50.915 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:19:50.915 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:19:50.916 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-18 16:13:09.325634031 +0000 2024-03-18T16:19:50.916 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-18 16:03:18.704047472 +0000 2024-03-18T16:19:50.916 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-18 16:03:18.704047472 +0000 2024-03-18T16:19:50.916 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-18 16:03:18.704047472 +0000 2024-03-18T16:19:50.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T16:19:51.002 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-18T16:19:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-18T16:19:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000297367 s, 1.7 MB/s 2024-03-18T16:19:51.014 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T16:19:51.073 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T16:19:51.073 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:19:51.073 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T16:19:51.091 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T16:19:51.092 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-03-18T16:19:51.149 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T16:19:51.150 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:19:51.150 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-18T16:19:51.150 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:19:51.150 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:19:51.150 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-18 16:19:34.251838973 +0000 2024-03-18T16:19:51.150 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-18 16:02:07.688899299 +0000 2024-03-18T16:19:51.150 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-18 16:02:07.688899299 +0000 2024-03-18T16:19:51.150 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-18 16:02:07.688899299 +0000 2024-03-18T16:19:51.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T16:19:51.218 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-18T16:19:51.218 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-18T16:19:51.218 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000220512 s, 2.3 MB/s 2024-03-18T16:19:51.220 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T16:19:51.277 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-03-18T16:19:51.334 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T16:19:51.334 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:19:51.334 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-18T16:19:51.334 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:19:51.334 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:19:51.335 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-18 16:19:34.251838973 +0000 2024-03-18T16:19:51.335 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-18 16:02:08.028889664 +0000 2024-03-18T16:19:51.335 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-18 16:02:08.028889664 +0000 2024-03-18T16:19:51.335 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-18 16:02:08.028889664 +0000 2024-03-18T16:19:51.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T16:19:51.402 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-18T16:19:51.402 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-18T16:19:51.402 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000219405 s, 2.3 MB/s 2024-03-18T16:19:51.403 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T16:19:51.460 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-03-18T16:19:51.515 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T16:19:51.515 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:19:51.515 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-18T16:19:51.515 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:19:51.515 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:19:51.515 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-18 16:19:34.251838973 +0000 2024-03-18T16:19:51.515 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-18 16:02:08.366880087 +0000 2024-03-18T16:19:51.515 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-18 16:02:08.366880087 +0000 2024-03-18T16:19:51.516 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-18 16:02:08.366880087 +0000 2024-03-18T16:19:51.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T16:19:51.581 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-18T16:19:51.606 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-18T16:19:51.607 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.00018813 s, 2.7 MB/s 2024-03-18T16:19:51.608 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T16:19:51.664 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-03-18T16:19:51.719 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T16:19:51.719 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:19:51.719 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-18T16:19:51.719 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:19:51.719 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:19:51.719 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-18 16:19:34.252838946 +0000 2024-03-18T16:19:51.719 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-18 16:02:08.686871019 +0000 2024-03-18T16:19:51.719 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-18 16:02:08.686871019 +0000 2024-03-18T16:19:51.719 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-03-18 16:02:08.686871019 +0000 2024-03-18T16:19:51.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T16:19:51.785 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-18T16:19:51.785 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-18T16:19:51.785 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.00019505 s, 2.6 MB/s 2024-03-18T16:19:51.786 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T16:19:51.841 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T16:19:51.842 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-18T16:19:51.842 INFO:tasks.ceph:Generating config... 2024-03-18T16:19:51.843 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-18T16:19:51.843 INFO:tasks.ceph:[client] debug client = 20 2024-03-18T16:19:51.843 INFO:tasks.ceph:[client] debug ms = 1 2024-03-18T16:19:51.843 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-18T16:19:51.843 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-18T16:19:51.843 INFO:tasks.ceph:[global] mds beacon grace = 60 2024-03-18T16:19:51.843 INFO:tasks.ceph:[global] mds heartbeat grace = 60 2024-03-18T16:19:51.843 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-18T16:19:51.844 INFO:tasks.ceph:[global] osd heartbeat grace = 60 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] mds valgrind exit = True 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-18T16:19:51.844 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-18T16:19:51.845 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-18T16:19:51.845 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-18T16:19:51.845 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-18T16:19:51.845 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-18T16:19:51.845 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-18T16:19:51.846 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-18T16:19:51.846 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-18T16:19:51.846 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-18T16:19:51.846 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-18T16:19:51.846 INFO:tasks.ceph:Setting up mon.a... 2024-03-18T16:19:51.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-18T16:19:51.990 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.keyring 2024-03-18T16:19:51.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-18T16:19:52.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-18T16:19:52.224 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.102'), ('mon.b', '172.21.15.154'), ('mon.c', '[v2:172.21.15.154:3301,v1:172.21.15.154:6790]')] 2024-03-18T16:19:52.224 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.102,172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790]', 'mds beacon grace': 60, 'mds heartbeat grace': 60, 'ms die on skipped message': False, 'osd heartbeat grace': 60}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-18T16:19:52.225 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:52.225 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-18T16:19:52.279 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.102 --add b 172.21.15.154 --addv c '[v2:172.21.15.154:3301,v1:172.21.15.154:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:19:52.435 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:19:52.436 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: generated fsid 0179d728-7c99-4c29-b345-3950a2c4b164 2024-03-18T16:19:52.436 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = quincy 2024-03-18T16:19:52.436 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 0 2024-03-18T16:19:52.436 INFO:teuthology.orchestra.run.smithi102.stdout:fsid 0179d728-7c99-4c29-b345-3950a2c4b164 2024-03-18T16:19:52.436 INFO:teuthology.orchestra.run.smithi102.stdout:last_changed 2024-03-18T16:19:52.434521+0000 2024-03-18T16:19:52.436 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-18T16:19:52.434521+0000 2024-03-18T16:19:52.436 INFO:teuthology.orchestra.run.smithi102.stdout:min_mon_release 17 (quincy) 2024-03-18T16:19:52.436 INFO:teuthology.orchestra.run.smithi102.stdout:election_strategy: 1 2024-03-18T16:19:52.437 INFO:teuthology.orchestra.run.smithi102.stdout:0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-03-18T16:19:52.437 INFO:teuthology.orchestra.run.smithi102.stdout:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-03-18T16:19:52.437 INFO:teuthology.orchestra.run.smithi102.stdout:2: [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] mon.c 2024-03-18T16:19:52.437 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-18T16:19:52.439 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-18T16:19:52.486 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0179d728-7c99-4c29-b345-3950a2c4b164... 2024-03-18T16:19:52.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-18T16:19:52.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: chdir = "" 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: auth supported = cephx 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: filestore xattr use omap = true 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: mon clock drift allowed = 1.000 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: osd crush chooseleaf type = 0 2024-03-18T16:19:52.549 INFO:teuthology.orchestra.run.smithi154.stdout: auth debug = true 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on old message = true 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on bug = true 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: mon pg warn max object skew = 0 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: # disable pg_autoscaler by default for new pools 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-18T16:19:52.550 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default size = 2 2024-03-18T16:19:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow primary affinity = true 2024-03-18T16:19:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow pg remap = true 2024-03-18T16:19:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on legacy crush tunables = false 2024-03-18T16:19:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on crush straw calc version zero = false 2024-03-18T16:19:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on no sortbitwise = false 2024-03-18T16:19:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on osd down out interval zero = false 2024-03-18T16:19:52.551 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on too few osds = false 2024-03-18T16:19:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-18T16:19:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-18T16:19:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: mon_allow_pool_size_one = true 2024-03-18T16:19:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-18T16:19:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: osd default data pool replay window = 5 2024-03-18T16:19:52.552 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: mon allow pool delete = true 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: mon cluster log file level = debug 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: debug asserts on shutdown = true 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: mon health detail to clog = false 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: mon host = "172.21.15.102,172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790]" 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: mds beacon grace = 60 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: mds heartbeat grace = 60 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on skipped message = False 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: osd heartbeat grace = 60 2024-03-18T16:19:52.553 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = 0179d728-7c99-4c29-b345-3950a2c4b164 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout:[osd] 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: osd journal size = 100 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub load threshold = 5.0 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub max interval = 600 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock profile = high_recovery_ops 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: osd recover clone overlap = true 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: osd recovery max chunk = 1048576 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug shutdown = true 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug op order = true 2024-03-18T16:19:52.554 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify stray on activate = true 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug trim objects = true 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: osd open classes on start = true 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug pg log writeout = true 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: osd deep scrub update digest min age = 30 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: osd map max advance = 10 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: journal zero on create = true 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: filestore ondisk finisher threads = 3 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: filestore apply finisher threads = 3 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.555 INFO:teuthology.orchestra.run.smithi154.stdout: bdev debug aio = true 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug misdirected ops = true 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore block size = 96636764160 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore compression mode = aggressive 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore fsck on mount = True 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluefs = 20 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluestore = 20 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: debug osd = 20 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: debug rocksdb = 10 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd backfillfull_ratio = 0.85 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd full ratio = 0.9 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd nearfull ratio = 0.8 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: osd failsafe full ratio = 0.95 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: osd fast shutdown = False 2024-03-18T16:19:52.556 INFO:teuthology.orchestra.run.smithi154.stdout: osd objectstore = bluestore 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: osd op complaint time = 180 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr] 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: debug mgr = 20 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: mgr/telemetry/nag = false 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout:[mon] 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-03-18T16:19:52.557 INFO:teuthology.orchestra.run.smithi154.stdout: debug paxos = 20 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: mon data avail warn = 5 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: mon mgr mkfs grace = 240 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd reporter subtree level = osd 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd prime pg temp = true 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: auth mon ticket ttl = 660 # 11m 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: auth service ticket ttl = 240 # 4m 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: # don't complain about insecure global_id in the test suite 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-18T16:19:52.558 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: # 1m isn't quite enough 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: mon_down_mkfs_grace = 2m 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_filestore_osds = false 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: mon op complaint time = 120 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd crush smoke test = False 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout:[client] 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: rgw cache enabled = true 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable ops log = true 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable usage log = true 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: client mount timeout = 600 2024-03-18T16:19:52.559 INFO:teuthology.orchestra.run.smithi154.stdout: debug client = 20 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: rados mon op timeout = 900 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: rados osd op timeout = 900 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.a] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.b] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.c] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.a] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.c] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.e] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.b] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.d] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.f] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout:[mds] 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: debug mds = 20 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: debug mds balancer = 20 2024-03-18T16:19:52.560 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-03-18T16:19:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: mds debug frag = True 2024-03-18T16:19:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: mds debug scatterstat = True 2024-03-18T16:19:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: mds op complaint time = 180 2024-03-18T16:19:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: mds valgrind exit = True 2024-03-18T16:19:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: mds verify scatter = True 2024-03-18T16:19:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: osd op complaint time = 180 2024-03-18T16:19:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: rados mon op timeout = 900 2024-03-18T16:19:52.561 INFO:teuthology.orchestra.run.smithi154.stdout: rados osd op timeout = 900 2024-03-18T16:19:52.651 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-03-18T16:19:52.651 INFO:teuthology.orchestra.run.smithi102.stdout: chdir = "" 2024-03-18T16:19:52.651 INFO:teuthology.orchestra.run.smithi102.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-18T16:19:52.651 INFO:teuthology.orchestra.run.smithi102.stdout: auth supported = cephx 2024-03-18T16:19:52.651 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.651 INFO:teuthology.orchestra.run.smithi102.stdout: filestore xattr use omap = true 2024-03-18T16:19:52.651 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.651 INFO:teuthology.orchestra.run.smithi102.stdout: mon clock drift allowed = 1.000 2024-03-18T16:19:52.651 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: osd crush chooseleaf type = 0 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: auth debug = true 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on old message = true 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on bug = true 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: mon pg warn max object skew = 0 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: # disable pg_autoscaler by default for new pools 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default size = 2 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.652 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow primary affinity = true 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow pg remap = true 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on legacy crush tunables = false 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on crush straw calc version zero = false 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on no sortbitwise = false 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on osd down out interval zero = false 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on too few osds = false 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon_allow_pool_size_one = true 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: osd default data pool replay window = 5 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: mon allow pool delete = true 2024-03-18T16:19:52.653 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: mon cluster log file level = debug 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: debug asserts on shutdown = true 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: mon health detail to clog = false 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: mon host = "172.21.15.102,172.21.15.154,[v2:172.21.15.154:3301,v1:172.21.15.154:6790]" 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: mds beacon grace = 60 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: mds heartbeat grace = 60 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on skipped message = False 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: osd heartbeat grace = 60 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 0179d728-7c99-4c29-b345-3950a2c4b164 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout:[osd] 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: osd journal size = 100 2024-03-18T16:19:52.654 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub load threshold = 5.0 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub max interval = 600 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd mclock profile = high_recovery_ops 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd recover clone overlap = true 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd recovery max chunk = 1048576 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug shutdown = true 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug op order = true 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify stray on activate = true 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug trim objects = true 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd open classes on start = true 2024-03-18T16:19:52.655 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug pg log writeout = true 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: osd deep scrub update digest min age = 30 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: osd map max advance = 10 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: journal zero on create = true 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: filestore ondisk finisher threads = 3 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: filestore apply finisher threads = 3 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: bdev debug aio = true 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug misdirected ops = true 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore block size = 96636764160 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore compression mode = aggressive 2024-03-18T16:19:52.656 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore fsck on mount = True 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluefs = 20 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluestore = 20 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: debug osd = 20 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: debug rocksdb = 10 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd backfillfull_ratio = 0.85 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd full ratio = 0.9 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd nearfull ratio = 0.8 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: osd failsafe full ratio = 0.95 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: osd fast shutdown = False 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: osd objectstore = bluestore 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: osd op complaint time = 180 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr] 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-18T16:19:52.657 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgr = 20 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: mgr/telemetry/nag = false 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout:[mon] 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: debug paxos = 20 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: mon data avail warn = 5 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: mon mgr mkfs grace = 240 2024-03-18T16:19:52.658 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd reporter subtree level = osd 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd prime pg temp = true 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: auth mon ticket ttl = 660 # 11m 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: auth service ticket ttl = 240 # 4m 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: # don't complain about insecure global_id in the test suite 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: # 1m isn't quite enough 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: mon_down_mkfs_grace = 2m 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.659 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_filestore_osds = false 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: mon op complaint time = 120 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd crush smoke test = False 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout:[client] 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: rgw cache enabled = true 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable ops log = true 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable usage log = true 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: client mount timeout = 600 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: debug client = 20 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: rados mon op timeout = 900 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout: rados osd op timeout = 900 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.a] 2024-03-18T16:19:52.660 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.b] 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.c] 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.e] 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d] 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.f] 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout:[mds] 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout: debug mds = 20 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout: debug mds balancer = 20 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout: mds debug frag = True 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout: mds debug scatterstat = True 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout: mds op complaint time = 180 2024-03-18T16:19:52.661 INFO:teuthology.orchestra.run.smithi102.stdout: mds valgrind exit = True 2024-03-18T16:19:52.662 INFO:teuthology.orchestra.run.smithi102.stdout: mds verify scatter = True 2024-03-18T16:19:52.662 INFO:teuthology.orchestra.run.smithi102.stdout: osd op complaint time = 180 2024-03-18T16:19:52.662 INFO:teuthology.orchestra.run.smithi102.stdout: rados mon op timeout = 900 2024-03-18T16:19:52.662 INFO:teuthology.orchestra.run.smithi102.stdout: rados osd op timeout = 900 2024-03-18T16:19:52.693 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-18T16:19:52.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-18T16:19:52.838 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-18T16:19:52.838 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:52.838 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-18T16:19:52.886 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:52.886 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-18T16:19:52.981 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:19:52.981 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:52.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-18T16:19:52.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-18T16:19:53.125 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:19:53.172 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:19:53.225 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:19:53.225 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:19:53.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-18T16:19:53.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-18T16:19:53.270 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:19:53.271 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:19:53.328 INFO:tasks.ceph:Setting up mon nodes... 2024-03-18T16:19:53.398 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-18T16:19:53.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-18T16:19:53.581 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-18T16:19:53.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-18T16:19:53.665 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-18T16:19:53.668 INFO:tasks.ceph:Setting up mds nodes... 2024-03-18T16:19:53.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-18T16:19:53.853 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-18T16:19:53.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-18T16:19:53.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-18T16:19:54.129 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-18T16:19:54.138 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-18T16:19:54.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-18T16:19:54.480 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-18T16:19:54.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-18T16:19:54.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-18T16:19:54.703 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-18T16:19:54.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-18T16:19:54.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-18T16:19:54.832 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-18T16:19:54.838 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-18T16:19:54.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-18T16:19:54.959 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-18T16:19:54.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-18T16:19:54.987 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-18T16:19:54.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T16:19:55.115 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-18T16:19:55.169 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-18T16:19:55.169 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-18T16:19:55.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-18T16:19:55.258 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T16:19:55.292 INFO:tasks.ceph:role: osd.0 2024-03-18T16:19:55.292 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:19:55.292 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-18T16:19:55.393 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T16:19:55.393 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T16:19:55.393 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T16:19:55.393 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T16:19:55.394 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T16:19:55.394 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-18T16:19:55.394 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T16:19:55.394 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T16:19:55.394 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T16:19:55.394 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T16:19:55.917 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-18T16:19:55.925 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-18T16:19:55.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-18T16:19:56.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-18T16:19:56.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-18T16:19:56.282 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T16:19:56.283 INFO:tasks.ceph:role: osd.1 2024-03-18T16:19:56.283 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:19:56.283 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-18T16:19:56.381 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T16:19:56.381 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T16:19:56.382 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T16:19:56.382 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T16:19:56.382 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T16:19:56.382 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-18T16:19:56.382 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T16:19:56.382 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T16:19:56.382 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T16:19:56.382 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T16:19:56.870 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-18T16:19:56.877 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-18T16:19:56.877 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-18T16:19:57.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-18T16:19:57.111 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-18T16:19:57.195 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T16:19:57.259 INFO:tasks.ceph:role: osd.2 2024-03-18T16:19:57.259 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:19:57.259 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-18T16:19:57.362 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T16:19:57.456 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T16:19:57.456 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T16:19:57.456 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T16:19:57.456 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T16:19:57.456 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-18T16:19:57.456 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T16:19:57.457 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T16:19:57.457 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T16:19:57.457 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T16:19:57.882 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-18T16:19:57.889 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-18T16:19:57.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-18T16:19:58.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-18T16:19:58.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-18T16:19:58.360 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-18T16:19:58.429 INFO:tasks.ceph:role: osd.3 2024-03-18T16:19:58.429 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:19:58.429 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T16:19:58.532 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T16:19:59.029 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-18T16:19:59.036 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-18T16:19:59.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-18T16:19:59.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-18T16:19:59.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:19:59.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:19:59.428+0000 7f80a2a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-18T16:19:59.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:19:59.431+0000 7f80a2a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-18T16:19:59.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:19:59.431+0000 7f80a2a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-18T16:19:59.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:19:59.431+0000 7f80a2a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-18T16:19:59.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:19:59.431+0000 7f80a2a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-18T16:19:59.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:19:59.442+0000 7f80a2a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-18T16:20:02.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-18T16:20:02.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:20:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:02.863+0000 7f851320d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-18T16:20:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:02.864+0000 7f851320d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-18T16:20:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:02.864+0000 7f851320d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-18T16:20:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:02.864+0000 7f851320d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-18T16:20:02.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:02.864+0000 7f851320d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-18T16:20:02.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:02.866+0000 7f851320d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-18T16:20:05.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-18T16:20:05.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:20:06.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:05.999+0000 7f95e1af9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-18T16:20:06.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:05.999+0000 7f95e1af9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-18T16:20:06.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:05.999+0000 7f95e1af9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-18T16:20:06.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:05.999+0000 7f95e1af9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-18T16:20:06.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:05.999+0000 7f95e1af9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-18T16:20:06.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:06.002+0000 7f95e1af9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-18T16:20:09.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-18T16:20:09.517 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:20:09.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:09.650+0000 7f34fa617640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-18T16:20:09.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:09.651+0000 7f34fa617640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-18T16:20:09.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:09.651+0000 7f34fa617640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-18T16:20:09.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:09.651+0000 7f34fa617640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-18T16:20:09.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:09.651+0000 7f34fa617640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-18T16:20:09.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:09.653+0000 7f34fa617640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-18T16:20:12.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-18T16:20:12.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-18T16:20:12.930 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T16:20:12.930 INFO:tasks.ceph:role: osd.4 2024-03-18T16:20:12.930 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:20:12.930 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T16:20:13.008 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T16:20:13.521 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-03-18T16:20:13.524 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-03-18T16:20:13.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-18T16:20:13.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-18T16:20:13.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-18T16:20:13.741 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T16:20:13.741 INFO:tasks.ceph:role: osd.5 2024-03-18T16:20:13.741 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:20:13.741 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-18T16:20:13.815 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T16:20:13.815 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T16:20:13.815 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T16:20:13.816 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T16:20:13.816 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T16:20:13.816 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-03-18T16:20:13.816 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T16:20:13.816 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T16:20:13.816 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T16:20:13.816 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T16:20:14.278 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-03-18T16:20:14.280 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-03-18T16:20:14.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-18T16:20:14.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-18T16:20:14.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-18T16:20:14.500 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T16:20:14.500 INFO:tasks.ceph:role: osd.6 2024-03-18T16:20:14.500 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:20:14.500 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T16:20:14.574 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T16:20:15.080 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-03-18T16:20:15.082 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-03-18T16:20:15.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-18T16:20:15.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-18T16:20:15.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-18T16:20:15.387 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-18T16:20:15.387 INFO:tasks.ceph:role: osd.7 2024-03-18T16:20:15.387 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:20:15.387 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-18T16:20:15.460 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-18T16:20:15.460 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-18T16:20:15.460 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-18T16:20:15.461 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-18T16:20:15.461 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-18T16:20:15.461 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-03-18T16:20:15.461 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-18T16:20:15.461 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-18T16:20:15.461 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-18T16:20:15.461 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-18T16:20:15.959 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-03-18T16:20:15.960 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-03-18T16:20:15.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-18T16:20:16.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-18T16:20:16.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:20:16.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:16.215+0000 7f6451615640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-18T16:20:16.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:16.216+0000 7f6451615640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-18T16:20:16.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:16.216+0000 7f6451615640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-18T16:20:16.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:16.216+0000 7f6451615640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-18T16:20:16.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:16.216+0000 7f6451615640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-18T16:20:16.219 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:16.216+0000 7f6451615640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-18T16:20:19.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-18T16:20:19.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:20:19.193 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:19.190+0000 7fe755d8f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-18T16:20:19.193 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:19.190+0000 7fe755d8f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-18T16:20:19.193 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:19.190+0000 7fe755d8f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-18T16:20:19.193 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:19.191+0000 7fe755d8f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-18T16:20:19.193 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:19.191+0000 7fe755d8f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-18T16:20:19.193 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:19.191+0000 7fe755d8f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-18T16:20:19.833 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:20:19.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:20:22.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-18T16:20:22.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:20:22.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:22.156+0000 7f4513e1b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-18T16:20:22.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:22.156+0000 7f4513e1b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-18T16:20:22.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:22.156+0000 7f4513e1b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-18T16:20:22.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:22.157+0000 7f4513e1b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-18T16:20:22.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:22.157+0000 7f4513e1b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-18T16:20:22.218 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:22.157+0000 7f4513e1b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-18T16:20:25.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-18T16:20:25.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:20:25.130 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:25.128+0000 7fde5aa15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-18T16:20:25.130 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:25.128+0000 7fde5aa15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-18T16:20:25.130 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:25.128+0000 7fde5aa15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-18T16:20:25.130 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:25.128+0000 7fde5aa15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-18T16:20:25.130 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:25.128+0000 7fde5aa15640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-18T16:20:25.131 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:25.129+0000 7fde5aa15640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-18T16:20:27.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-18T16:20:28.016 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-18T16:20:28.080 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:28.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-18T16:20:28.178 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:28.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-18T16:20:28.275 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:28.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-18T16:20:28.369 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:28.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-18T16:20:28.463 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:28.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-18T16:20:28.547 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:28.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-18T16:20:28.830 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:28.830 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-18T16:20:28.921 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:28.921 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-18T16:20:29.014 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:29.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-18T16:20:29.046 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:29.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-18T16:20:29.120 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:29.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-18T16:20:29.189 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:29.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-18T16:20:29.259 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:29.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-18T16:20:29.328 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:29.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-18T16:20:29.421 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:29.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-18T16:20:29.527 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:29.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-18T16:20:29.594 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:29.594 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-18T16:20:29.650 INFO:tasks.ceph:Adding keys to all mons... 2024-03-18T16:20:29.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-18T16:20:29.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-18T16:20:29.681 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.y] 2024-03-18T16:20:29.744 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCpafhlNLeWIhAAeEaIzlXq20JNmc/1T6uEfw== 2024-03-18T16:20:29.744 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.a] 2024-03-18T16:20:29.744 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCpafhlmmDRMhAAgU9mle0rwW57J8O+HpzB/Q== 2024-03-18T16:20:29.744 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.c] 2024-03-18T16:20:29.744 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCqafhlbACvBxAApJMKo1/DKeht1Mb3KillIQ== 2024-03-18T16:20:29.744 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.e] 2024-03-18T16:20:29.744 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCqafhlSM+QHBAAbVTrrWMyQqneSMpy4HusFQ== 2024-03-18T16:20:29.744 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2024-03-18T16:20:29.744 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCvafhlGKSYGRAAfJC5ZISNm1NjE7vhLouZNw== 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCyafhl8SmFMxAABSqRZlj3ctVT5dMD2l7mPA== 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQC1afhl/QeZOxAAbu9nP9ZNpyKesxarxKKU3A== 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.3] 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQC5afhl1h7VJhAAIenrvWt7BWcxj51PdDYCrQ== 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.x] 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCpafhlK2qaJxAAus3xjBCyql5UhSuiSEwdUw== 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.b] 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCqafhlKO/ZKRAAN408Md12pkw2Ax3d2QdHWA== 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.d] 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCqafhlmhWTMRAAkK+seAWxfuKqn/zxdcLu0Q== 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout:[mds.f] 2024-03-18T16:20:29.745 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCqafhlQBYgORAA8naWzqTqu6GZyOlYmVAp6w== 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.4] 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDAafhl7QDuDBAA9NAw3HLXabOTEF4T7DmySA== 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.5] 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDDafhl7jRrCxAAn8gD1DEy3ycPkXqDwwchdw== 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.6] 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDGafhlfotmCRAAgXmzBIm5ziX31Lj+rgyqUQ== 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.7] 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDJafhlFFq0BxAAosDLsxw8aEDYraxrzrKVsg== 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout:[client.0] 2024-03-18T16:20:29.746 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQCrafhl+PTSBhAA1G+u7DYBmHojPPNn7pVOaw== 2024-03-18T16:20:29.748 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.y] 2024-03-18T16:20:29.748 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCpafhlNLeWIhAAeEaIzlXq20JNmc/1T6uEfw== 2024-03-18T16:20:29.748 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2024-03-18T16:20:29.748 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCpafhlmmDRMhAAgU9mle0rwW57J8O+HpzB/Q== 2024-03-18T16:20:29.748 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2024-03-18T16:20:29.748 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCqafhlbACvBxAApJMKo1/DKeht1Mb3KillIQ== 2024-03-18T16:20:29.748 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.e] 2024-03-18T16:20:29.748 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCqafhlSM+QHBAAbVTrrWMyQqneSMpy4HusFQ== 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.0] 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCvafhlGKSYGRAAfJC5ZISNm1NjE7vhLouZNw== 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.1] 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCyafhl8SmFMxAABSqRZlj3ctVT5dMD2l7mPA== 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.2] 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQC1afhl/QeZOxAAbu9nP9ZNpyKesxarxKKU3A== 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.3] 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQC5afhl1h7VJhAAIenrvWt7BWcxj51PdDYCrQ== 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.x] 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCpafhlK2qaJxAAus3xjBCyql5UhSuiSEwdUw== 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCqafhlKO/ZKRAAN408Md12pkw2Ax3d2QdHWA== 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d] 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCqafhlmhWTMRAAkK+seAWxfuKqn/zxdcLu0Q== 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.f] 2024-03-18T16:20:29.749 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCqafhlQBYgORAA8naWzqTqu6GZyOlYmVAp6w== 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.4] 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDAafhl7QDuDBAA9NAw3HLXabOTEF4T7DmySA== 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.5] 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDDafhl7jRrCxAAn8gD1DEy3ycPkXqDwwchdw== 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.6] 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDGafhlfotmCRAAgXmzBIm5ziX31Lj+rgyqUQ== 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.7] 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQDJafhlFFq0BxAAosDLsxw8aEDYraxrzrKVsg== 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-03-18T16:20:29.750 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCrafhl+PTSBhAA1G+u7DYBmHojPPNn7pVOaw== 2024-03-18T16:20:29.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T16:20:29.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T16:20:29.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:29.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:30.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:30.067 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:30.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:30.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:30.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:30.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:30.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:30.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:30.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:30.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:30.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:30.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:30.889 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T16:20:30.893 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-18T16:20:31.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:31.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:31.316 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:31.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:31.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:31.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-18T16:20:31.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:31.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:31.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:31.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:31.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:32.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:32.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:32.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-18T16:20:32.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T16:20:32.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-18T16:20:32.632 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-18T16:20:32.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-18T16:20:32.713 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-18T16:20:32.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-18T16:20:32.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-18T16:20:33.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-18T16:20:33.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-18T16:20:33.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-18T16:20:33.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-18T16:20:33.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-18T16:20:33.451 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:20:33.459 DEBUG:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-18T16:20:33.508 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-18T16:20:33.534 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-18T16:20:33.534 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-18T16:20:33.535 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-18T16:20:33.539 INFO:tasks.ceph.mon.a:Started 2024-03-18T16:20:33.539 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-18T16:20:33.540 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-18T16:20:33.540 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-18T16:20:33.548 INFO:tasks.ceph.mon.b:Started 2024-03-18T16:20:33.548 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-18T16:20:33.548 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-18T16:20:33.548 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-18T16:20:33.550 INFO:tasks.ceph.mon.c:Started 2024-03-18T16:20:33.550 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-18T16:20:33.550 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-18T16:20:33.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-18T16:20:33.555 INFO:tasks.ceph.mgr.y:Started 2024-03-18T16:20:33.555 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-18T16:20:33.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-18T16:20:33.557 INFO:tasks.ceph.mgr.x:Started 2024-03-18T16:20:33.557 INFO:tasks.ceph:Setting crush tunables to default 2024-03-18T16:20:33.558 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd crush tunables default 2024-03-18T16:20:33.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.733+0000 7fd7c3a00640 1 Processor -- start 2024-03-18T16:20:33.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.733+0000 7fd7c3a00640 1 -- start start 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.734+0000 7fd7c3a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd7bc0723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.734+0000 7fd7c3a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd7bc06be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.734+0000 7fd7c3a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc158390 con 0x7fd7bc0c4fb0 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.734+0000 7fd7c3a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd7bc141ae0 con 0x7fd7bc06ba70 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.734+0000 7fd7c3a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd7bc158670 con 0x7fd7bc141700 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.734+0000 7fd7c1a00640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fd7bc0c4fb0 legacy=0x7fd7bc15bb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.734+0000 7fd7bbe00640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd7bc06be90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:3300/0 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.735+0000 7fd7c1000640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd7bc0723b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3301/0 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.735+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd7bc06be90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T16:20:33.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.735+0000 7fd7c1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd7bc0723b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T16:20:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.936+0000 7fd7bbe00640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd7bc06be90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:3300/0 2024-03-18T16:20:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.936+0000 7fd7c1a00640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fd7bc0c4fb0 legacy=0x7fd7bc15bb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-18T16:20:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.936+0000 7fd7c1000640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd7bc0723b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3301/0 2024-03-18T16:20:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.936+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd7bc06be90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T16:20:33.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:33.936+0000 7fd7c1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd7bc0723b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T16:20:34.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:34.337+0000 7fd7c1a00640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fd7bc0c4fb0 legacy=0x7fd7bc15bb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-18T16:20:34.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:34.337+0000 7fd7c1000640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd7bc0723b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3301/0 2024-03-18T16:20:34.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:34.337+0000 7fd7bbe00640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd7bc06be90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:3300/0 2024-03-18T16:20:34.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:34.337+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd7bc06be90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T16:20:34.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:34.337+0000 7fd7c1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd7bc0723b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T16:20:35.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:35.138+0000 7fd7c1a00640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fd7bc0c4fb0 legacy=0x7fd7bc15bb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-18T16:20:35.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:35.138+0000 7fd7bbe00640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd7bc06be90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:3300/0 2024-03-18T16:20:35.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:35.138+0000 7fd7c1000640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd7bc0723b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3301/0 2024-03-18T16:20:35.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:35.138+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd7bc06be90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T16:20:35.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:35.138+0000 7fd7c1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd7bc0723b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T16:20:36.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7ba000640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd7bc0723b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:36.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7ba000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd7bc0723b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:36.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7ba000640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd7bc06be90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:36.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7ba000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd7bc06be90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:36.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7ba000640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fd7bc0c4fb0 legacy=0x7fd7bc15bb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:36.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7ba000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 0x7fd79c0098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:36.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7ba000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc158950 con 0x7fd79c0036c0 2024-03-18T16:20:36.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7ba000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd7bc141ae0 con 0x7fd79c007400 2024-03-18T16:20:36.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7bbe00640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 msgr2=0x7fd79c0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-03-18T16:20:36.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7c1000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c0036c0 legacy=0x7fd79c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-03-18T16:20:36.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.735+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 0x7fd79c0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T16:20:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.936+0000 7fd7bbe00640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 msgr2=0x7fd79c0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-03-18T16:20:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.936+0000 7fd7c1000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c0036c0 legacy=0x7fd79c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-03-18T16:20:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:36.936+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 0x7fd79c0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T16:20:37.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:37.337+0000 7fd7c1000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c0036c0 legacy=0x7fd79c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-03-18T16:20:37.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:37.337+0000 7fd7bbe00640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 msgr2=0x7fd79c0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-03-18T16:20:38.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:37.337+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 0x7fd79c0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T16:20:38.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:38.138+0000 7fd7c1000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c0036c0 legacy=0x7fd79c005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-03-18T16:20:38.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:38.138+0000 7fd7bbe00640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 msgr2=0x7fd79c0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-03-18T16:20:38.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:38.138+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 0x7fd79c0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T16:20:39.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.735+0000 7fd7ba000640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 msgr2=0x7fd79c0098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:39.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.735+0000 7fd7ba000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd79c007400 0x7fd79c0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:39.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.735+0000 7fd7ba000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c0036c0 legacy=0x7fd79c005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:39.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.735+0000 7fd7c1a00640 1 -- reap_dead start 2024-03-18T16:20:39.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.735+0000 7fd7ba000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd79c005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:39.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.735+0000 7fd7ba000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd79c006270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:39.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.735+0000 7fd7ba000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc158390 con 0x7fd79c009f80 2024-03-18T16:20:39.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.735+0000 7fd7ba000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd7bc141ae0 con 0x7fd7bc06ba70 2024-03-18T16:20:39.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.735+0000 7fd7ba000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd7bc158670 con 0x7fd7bc141700 2024-03-18T16:20:39.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.736+0000 7fd7c1000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c009f80 legacy=0x7fd79c00a3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-03-18T16:20:39.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.736+0000 7fd7c1a00640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3301/0 2024-03-18T16:20:39.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.736+0000 7fd7bbe00640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd79c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:3300/0 2024-03-18T16:20:39.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.736+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd79c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T16:20:39.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.736+0000 7fd7c1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T16:20:39.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.937+0000 7fd7c1000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c009f80 legacy=0x7fd79c00a3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-03-18T16:20:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.937+0000 7fd7c1a00640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3301/0 2024-03-18T16:20:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.937+0000 7fd7bbe00640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd79c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:3300/0 2024-03-18T16:20:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.937+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd79c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T16:20:40.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:39.937+0000 7fd7c1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T16:20:40.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:40.338+0000 7fd7c1a00640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3301/0 2024-03-18T16:20:40.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:40.338+0000 7fd7c1000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c009f80 legacy=0x7fd79c00a3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-03-18T16:20:40.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:40.338+0000 7fd7bbe00640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd79c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:3300/0 2024-03-18T16:20:40.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:40.338+0000 7fd7c1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T16:20:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:40.338+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd79c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T16:20:40.739 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:00:07.083 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:20:40.740 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:00:07.085 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:20:41.059 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:00:07.404 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:20:41.061 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:00:07.405 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:20:41.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:41.139+0000 7fd7c1000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c009f80 legacy=0x7fd79c00a3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-03-18T16:20:41.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:41.139+0000 7fd7c1a00640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3301/0 2024-03-18T16:20:41.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:41.139+0000 7fd7bbe00640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd79c005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.102:3300/0 2024-03-18T16:20:41.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:41.139+0000 7fd7c1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T16:20:41.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:41.139+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd79c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T16:20:41.480 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:00:07.451 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:20:41.510 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:00:07.769 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:20:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.735+0000 7fd7ba000640 1 -- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.735+0000 7fd7ba000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc141700 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:42.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.735+0000 7fd7ba000640 1 -- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 msgr2=0x7fd79c005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.735+0000 7fd7ba000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7bc06ba70 0x7fd79c005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.736+0000 7fd7ba000640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd79c009f80 legacy=0x7fd79c00a3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.736+0000 7fd7ba000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.736+0000 7fd7ba000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc158950 con 0x7fd79c00aad0 2024-03-18T16:20:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.736+0000 7fd7ba000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd7bc141ae0 con 0x7fd79c0098c0 2024-03-18T16:20:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.736+0000 7fd7c1000640 1 -- >> v1:172.21.15.102:6789/0 conn(0x7fd79c00aad0 legacy=0x7fd79c0067b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.102:6789/0 2024-03-18T16:20:42.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.760+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.769+0000 7fd7bbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41616/0 (socket says 172.21.15.102:41616) 2024-03-18T16:20:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.769+0000 7fd7bbe00640 1 -- 172.21.15.102:0/1296007740 learned_addr learned my addr 172.21.15.102:0/1296007740 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:20:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.899+0000 7fd7bbe00640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 msgr2=0x7fd79c013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T16:20:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.899+0000 7fd7bbe00640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 msgr2=0x7fd79c013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.899+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:42.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:42.900+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T16:20:43.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.227+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:43.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.242+0000 7fd7bbe00640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 msgr2=0x7fd79c013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-18T16:20:43.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.243+0000 7fd7bbe00640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 msgr2=0x7fd79c013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:43.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.243+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:43.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.243+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T16:20:43.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.531+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v1:172.21.15.102:6789/0 conn(0x7fd79c00aad0 legacy=0x7fd79c0067b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T16:20:43.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.531+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v1:172.21.15.102:6789/0 conn(0x7fd79c00aad0 legacy=0x7fd79c0067b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T16:20:43.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.531+0000 7fd7c1000640 1 --1- 172.21.15.102:0/1296007740 >> v1:172.21.15.102:6789/0 conn(0x7fd79c00aad0 0x7fd79c0067b0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-18T16:20:43.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.531+0000 7fd7c1000640 1 --1- 172.21.15.102:0/1296007740 >> v1:172.21.15.102:6789/0 conn(0x7fd79c00aad0 0x7fd79c0067b0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-18T16:20:43.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.703+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:43.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.705+0000 7fd7bbe00640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 msgr2=0x7fd79c013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T16:20:43.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.705+0000 7fd7bbe00640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 msgr2=0x7fd79c013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:43.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.705+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:43.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:43.705+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T16:20:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:44.507+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:44.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:44.509+0000 7fd7bbe00640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 msgr2=0x7fd79c013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T16:20:44.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:44.509+0000 7fd7bbe00640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 msgr2=0x7fd79c013130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:44.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:44.509+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:44.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:44.509+0000 7fd7bbe00640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T16:20:45.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.736+0000 7fd7ba000640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 msgr2=0x7fd79c013130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:45.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.736+0000 7fd7ba000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c0098c0 0x7fd79c013130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:45.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.736+0000 7fd7ba000640 1 -- 172.21.15.102:0/1296007740 >> v1:172.21.15.102:6789/0 conn(0x7fd79c00aad0 legacy=0x7fd79c0067b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-18T16:20:45.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.736+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 reap_dead start 2024-03-18T16:20:45.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.736+0000 7fd7ba000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:45.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.736+0000 7fd7ba000640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:45.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.736+0000 7fd7ba000640 1 -- 172.21.15.102:0/1296007740 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc158390 con 0x7fd79c0067b0 2024-03-18T16:20:45.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.736+0000 7fd7ba000640 1 -- 172.21.15.102:0/1296007740 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd7bc141ae0 con 0x7fd79c00a3c0 2024-03-18T16:20:45.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.736+0000 7fd7ba000640 1 -- 172.21.15.102:0/1296007740 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd7bc158670 con 0x7fd79c005ae0 2024-03-18T16:20:45.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.737+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:45.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.737+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:45.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.738+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd79c00e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-18T16:20:45.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.738+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd79c00e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:45.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.738+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:45.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.738+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T16:20:45.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.739+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T16:20:45.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.739+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:45.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.739+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:45.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.739+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T16:20:45.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.940+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.941+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd79c00e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T16:20:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.941+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd79c00e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.941+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.941+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T16:20:45.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.958+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.959+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-18T16:20:46.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.959+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:46.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.959+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:46.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:45.960+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T16:20:46.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.342+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.343+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd79c00e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T16:20:46.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.343+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd79c00e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:46.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.343+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:46.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.344+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T16:20:46.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.362+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.363+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T16:20:46.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.363+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:46.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.363+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:46.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:46.363+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T16:20:47.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.223+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.231+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd79c00e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-18T16:20:47.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.231+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd79c00e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:47.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.231+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:47.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.231+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T16:20:47.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.251+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.255+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-18T16:20:47.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.256+0000 7fd7c1000640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-18T16:20:47.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.256+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T16:20:47.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.256+0000 7fd7c1000640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T16:20:47.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.680+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1070847264 0 0) 0x7fd7bc158390 con 0x7fd79c0067b0 2024-03-18T16:20:47.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.680+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7a0003520 con 0x7fd79c0067b0 2024-03-18T16:20:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.981+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1818970112 0 0) 0x7fd7a0003520 con 0x7fd79c0067b0 2024-03-18T16:20:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.981+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd79c00e7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.981+0000 7fd7bb400640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd79c00e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 msgr2=0x7fd79c006270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 --2- 172.21.15.102:0/1296007740 >> v2:172.21.15.154:3300/0 conn(0x7fd79c00a3c0 0x7fd79c006270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c006b90 con 0x7fd79c0067b0 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7fd7a8009560 con 0x7fd79c0067b0 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 >> v1:172.21.15.154:6789/0 conn(0x7fd79c0067b0 legacy=0x7fd79c00b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd7a00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7a0003fe0 0x7fd7a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a0004d40 0x7fd7a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc141ae0 con 0x7fd7a0003fe0 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc158670 con 0x7fd7a0004d40 2024-03-18T16:20:47.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.982+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a000ff90 con 0x7fd79c005ae0 2024-03-18T16:20:47.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.988+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a0004d40 0x7fd7a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.990+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 msgr2=0x7fd7a00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.990+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd79c005ae0 0x7fd7a00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.990+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7a0003fe0 msgr2=0x7fd7a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.990+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7a0003fe0 0x7fd7a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.990+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a0003520 con 0x7fd7a0004d40 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:47.995+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a0004d40 0x7fd7a00051c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc155d50 tx=0x7fd7a0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad7deaf795a55131 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.000+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b403a070 con 0x7fd7a0004d40 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.000+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b402d920 con 0x7fd7a0004d40 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.000+0000 7fd7bb400640 1 -- 172.21.15.102:0/1296007740 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4007b00 con 0x7fd7a0004d40 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.002+0000 7fd7c3a00640 1 -- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a0004d40 msgr2=0x7fd7a00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.002+0000 7fd7c3a00640 1 --2- 172.21.15.102:0/1296007740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7a0004d40 0x7fd7a00051c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc155d50 tx=0x7fd7a0005830 comp rx=0 tx=0).stop 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.002+0000 7fd7c1a00640 1 -- 172.21.15.102:0/1296007740 reap_dead start 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.002+0000 7fd7c3a00640 1 -- 172.21.15.102:0/1296007740 shutdown_connections 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.002+0000 7fd7c3a00640 1 -- 172.21.15.102:0/1296007740 >> 172.21.15.102:0/1296007740 conn(0x7fd7bc0c83a0 msgr2=0x7fd7bc0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.002+0000 7fd7c3a00640 1 -- 172.21.15.102:0/1296007740 shutdown_connections 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.003+0000 7fd7c3a00640 1 -- 172.21.15.102:0/1296007740 wait complete. 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.010+0000 7fd7c3a00640 1 Processor -- start 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.011+0000 7fd7c3a00640 1 -- start start 2024-03-18T16:20:48.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.014+0000 7fd7c3a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc06fa70 0x7fd7bc159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.014+0000 7fd7c3a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7bc1576f0 0x7fd7bc1560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.014+0000 7fd7c3a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7bc06fe50 0x7fd7bc13d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.014+0000 7fd7c3a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc0723b0 con 0x7fd7bc1576f0 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.014+0000 7fd7c3a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc158f10 con 0x7fd7bc06fe50 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.014+0000 7fd7c3a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd7bc072530 con 0x7fd7bc06fa70 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.015+0000 7fd7c1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc06fa70 0x7fd7bc159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.015+0000 7fd7c1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc06fa70 0x7fd7bc159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57690/0 (socket says 172.21.15.102:57690) 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.015+0000 7fd7c1000640 1 -- 172.21.15.102:0/797752256 learned_addr learned my addr 172.21.15.102:0/797752256 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.015+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/797752256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7bc06fe50 0x7fd7bc13d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.021+0000 7fd7c1a00640 1 -- 172.21.15.102:0/797752256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc06fa70 msgr2=0x7fd7bc159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.021+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/797752256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc06fa70 0x7fd7bc159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.021+0000 7fd7c1a00640 1 -- 172.21.15.102:0/797752256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7bc1576f0 msgr2=0x7fd7bc1560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.021+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/797752256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7bc1576f0 0x7fd7bc1560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.021+0000 7fd7c1a00640 1 -- 172.21.15.102:0/797752256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7bc15adb0 con 0x7fd7bc06fe50 2024-03-18T16:20:48.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.024+0000 7fd7c1a00640 1 --2- 172.21.15.102:0/797752256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7bc06fe50 0x7fd7bc13d880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4034000 tx=0x7fd7b4006080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.027+0000 7fd7b8c00640 1 -- 172.21.15.102:0/797752256 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4048070 con 0x7fd7bc06fe50 2024-03-18T16:20:48.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.028+0000 7fd7b8c00640 1 -- 172.21.15.102:0/797752256 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b40359c0 con 0x7fd7bc06fe50 2024-03-18T16:20:48.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.028+0000 7fd7b8c00640 1 -- 172.21.15.102:0/797752256 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b403a040 con 0x7fd7bc06fe50 2024-03-18T16:20:48.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.028+0000 7fd7c3a00640 1 -- 172.21.15.102:0/797752256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7bc15b050 con 0x7fd7bc06fe50 2024-03-18T16:20:48.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.028+0000 7fd7c3a00640 1 -- 172.21.15.102:0/797752256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7bc15b310 con 0x7fd7bc06fe50 2024-03-18T16:20:48.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.029+0000 7fd7c3a00640 1 -- 172.21.15.102:0/797752256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7bc074fd0 con 0x7fd7bc06fe50 2024-03-18T16:20:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.049+0000 7fd7b8c00640 1 -- 172.21.15.102:0/797752256 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd7b404b070 con 0x7fd7bc06fe50 2024-03-18T16:20:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.049+0000 7fd7b8c00640 1 -- 172.21.15.102:0/797752256 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd7b4040c50 con 0x7fd7bc06fe50 2024-03-18T16:20:48.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.259+0000 7fd7b8c00640 1 -- 172.21.15.102:0/797752256 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd7b4051060 con 0x7fd7bc06fe50 2024-03-18T16:20:48.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.354+0000 7fd7c3a00640 1 -- 172.21.15.102:0/797752256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fd7bc0c63a0 con 0x7fd7bc06fe50 2024-03-18T16:20:48.451 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:48.433+0000 7f73bec3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T16:20:48.488 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:48.486+0000 7f73bec3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T16:20:48.719 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:48.584+0000 7fb8c843a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T16:20:48.719 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:48.653+0000 7fb8c843a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T16:20:48.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.889+0000 7fd7b8c00640 1 -- 172.21.15.102:0/797752256 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fd7b4040470 con 0x7fd7bc06fe50 2024-03-18T16:20:48.891 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-03-18T16:20:48.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.890+0000 7fd7c3a00640 1 -- 172.21.15.102:0/797752256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7bc06fe50 msgr2=0x7fd7bc13d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.890+0000 7fd7c3a00640 1 --2- 172.21.15.102:0/797752256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7bc06fe50 0x7fd7bc13d880 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4034000 tx=0x7fd7b4006080 comp rx=0 tx=0).stop 2024-03-18T16:20:48.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.891+0000 7fd7c3a00640 1 -- 172.21.15.102:0/797752256 shutdown_connections 2024-03-18T16:20:48.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.891+0000 7fd7c3a00640 1 --2- 172.21.15.102:0/797752256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7bc06fe50 0x7fd7bc13d880 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.891+0000 7fd7c3a00640 1 --2- 172.21.15.102:0/797752256 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7bc1576f0 0x7fd7bc1560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.891+0000 7fd7c3a00640 1 --2- 172.21.15.102:0/797752256 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7bc06fa70 0x7fd7bc159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.891+0000 7fd7c3a00640 1 -- 172.21.15.102:0/797752256 >> 172.21.15.102:0/797752256 conn(0x7fd7bc0c83a0 msgr2=0x7fd7bc0ca0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.891+0000 7fd7c3a00640 1 -- 172.21.15.102:0/797752256 shutdown_connections 2024-03-18T16:20:48.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:48.891+0000 7fd7c3a00640 1 -- 172.21.15.102:0/797752256 wait complete. 2024-03-18T16:20:48.913 INFO:tasks.ceph:check_enable_crimson: False 2024-03-18T16:20:48.966 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-18T16:20:49.025 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:49.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-18T16:20:49.026 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:48.999+0000 7f73bec3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T16:20:49.056 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.054+0000 7f73bec3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T16:20:49.108 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.100+0000 7f73bec3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T16:20:49.109 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:49.110 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-18T16:20:49.155 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.153+0000 7f73bec3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T16:20:49.198 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.196+0000 7f73bec3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T16:20:49.235 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:49.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-18T16:20:49.315 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.313+0000 7f73bec3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T16:20:49.367 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:20:49.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-18T16:20:49.386 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.383+0000 7f73bec3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T16:20:49.489 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:49.408+0000 7fb8c843a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T16:20:49.490 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.429+0000 7f73bec3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T16:20:49.490 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.480+0000 7f73bec3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T16:20:49.491 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:49.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-18T16:20:49.492 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:49.475+0000 7fb8c843a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T16:20:49.518 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:49.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-18T16:20:49.528 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:49.526+0000 7fb8c843a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T16:20:49.531 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.529+0000 7f73bec3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T16:20:49.558 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:49.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-18T16:20:49.602 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:49.588+0000 7fb8c843a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T16:20:49.628 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-18T16:20:49.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-18T16:20:49.653 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:49.637+0000 7fb8c843a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T16:20:49.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new d7aad584-b031-4470-8d3b-fe2fd9bd7d2a 0 2024-03-18T16:20:49.768 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:49.767+0000 7fb8c843a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T16:20:49.839 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.836+0000 7ff2e2f37640 1 Processor -- start 2024-03-18T16:20:49.845 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.836+0000 7ff2e2f37640 1 -- start start 2024-03-18T16:20:49.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.837+0000 7ff2e2f37640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff2dc0c71d0 0x7ff2dc0c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.837+0000 7ff2e2f37640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff2dc0c62d0 0x7ff2dc157720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.837+0000 7ff2e2f37640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2dc15aa40 con 0x7ff2dc1515f0 2024-03-18T16:20:49.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.837+0000 7ff2e2f37640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff2dc15ad20 con 0x7ff2dc0c71d0 2024-03-18T16:20:49.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.837+0000 7ff2e2f37640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff2dc15b000 con 0x7ff2dc0c62d0 2024-03-18T16:20:49.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.839+0000 7ff2e14d4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff2dc0c62d0 0x7ff2dc157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.839+0000 7ff2e14d4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff2dc0c62d0 0x7ff2dc157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:38638/0 (socket says 172.21.15.154:38638) 2024-03-18T16:20:49.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.839+0000 7ff2e14d4640 1 -- 172.21.15.154:0/1443840003 learned_addr learned my addr 172.21.15.154:0/1443840003 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:49.847 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.843+0000 7ff2dbfff640 1 --2- 172.21.15.154:0/1443840003 >> v2:172.21.15.102:3300/0 conn(0x7ff2dc0c71d0 0x7ff2dc0c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.854 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:49.852+0000 7fb8c843a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T16:20:49.860 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.858+0000 7ff2e14d4640 1 -- 172.21.15.154:0/1443840003 >> v2:172.21.15.102:3300/0 conn(0x7ff2dc0c71d0 msgr2=0x7ff2dc0c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:49.860 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.858+0000 7ff2e14d4640 1 --2- 172.21.15.154:0/1443840003 >> v2:172.21.15.102:3300/0 conn(0x7ff2dc0c71d0 0x7ff2dc0c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.861 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.858+0000 7ff2e14d4640 1 -- 172.21.15.154:0/1443840003 >> v1:172.21.15.102:6789/0 conn(0x7ff2dc1515f0 legacy=0x7ff2dc066f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:49.861 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.858+0000 7ff2e14d4640 1 -- 172.21.15.154:0/1443840003 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc15b2e0 con 0x7ff2dc0c62d0 2024-03-18T16:20:49.865 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.863+0000 7ff2e14d4640 1 --2- 172.21.15.154:0/1443840003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc0c62d0 0x7ff2dc157720 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc071fa0 tx=0x7ff2d402e180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=617ab7ff79a258f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:49.869 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.867+0000 7ff2db7fe640 1 -- 172.21.15.154:0/1443840003 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d400b0d0 con 0x7ff2dc0c62d0 2024-03-18T16:20:49.870 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.867+0000 7ff2db7fe640 1 -- 172.21.15.154:0/1443840003 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d400b270 con 0x7ff2dc0c62d0 2024-03-18T16:20:49.870 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.867+0000 7ff2db7fe640 1 -- 172.21.15.154:0/1443840003 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4009600 con 0x7ff2dc0c62d0 2024-03-18T16:20:49.871 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.868+0000 7ff2e2f37640 1 -- 172.21.15.154:0/1443840003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc0c62d0 msgr2=0x7ff2dc157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:49.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.868+0000 7ff2e2f37640 1 --2- 172.21.15.154:0/1443840003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc0c62d0 0x7ff2dc157720 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc071fa0 tx=0x7ff2d402e180 comp rx=0 tx=0).stop 2024-03-18T16:20:49.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.868+0000 7ff2e2f37640 1 -- 172.21.15.154:0/1443840003 shutdown_connections 2024-03-18T16:20:49.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.868+0000 7ff2e2f37640 1 --2- 172.21.15.154:0/1443840003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc0c62d0 0x7ff2dc157720 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.868+0000 7ff2e2f37640 1 --2- 172.21.15.154:0/1443840003 >> v2:172.21.15.102:3300/0 conn(0x7ff2dc0c71d0 0x7ff2dc0c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.868+0000 7ff2e2f37640 1 -- 172.21.15.154:0/1443840003 >> 172.21.15.154:0/1443840003 conn(0x7ff2dc0cc390 msgr2=0x7ff2dc0cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:49.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.868+0000 7ff2e2f37640 1 -- 172.21.15.154:0/1443840003 shutdown_connections 2024-03-18T16:20:49.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.868+0000 7ff2e2f37640 1 -- 172.21.15.154:0/1443840003 wait complete. 2024-03-18T16:20:49.927 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:49.908+0000 7fb8c843a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T16:20:49.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.869+0000 7ff2e2f37640 1 Processor -- start 2024-03-18T16:20:49.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.869+0000 7ff2e2f37640 1 -- start start 2024-03-18T16:20:49.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.870+0000 7ff2e2f37640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2dc0c62d0 0x7ff2dc1ec890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.870+0000 7ff2e2f37640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2dc0c71d0 0x7ff2dc152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.870+0000 7ff2e2f37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc152fc0 0x7ff2dc154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.870+0000 7ff2e2f37640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc15a6e0 con 0x7ff2dc0c71d0 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.870+0000 7ff2e2f37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc157200 con 0x7ff2dc152fc0 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.870+0000 7ff2e2f37640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff2dc157c60 con 0x7ff2dc0c62d0 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.871+0000 7ff2e0cd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2dc0c62d0 0x7ff2dc1ec890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.871+0000 7ff2e14d4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc152fc0 0x7ff2dc154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.871+0000 7ff2e0cd3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2dc0c62d0 0x7ff2dc1ec890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:42798/0 (socket says 172.21.15.154:42798) 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.871+0000 7ff2e0cd3640 1 -- 172.21.15.154:0/330029804 learned_addr learned my addr 172.21.15.154:0/330029804 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.875+0000 7ff2e14d4640 1 -- 172.21.15.154:0/330029804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2dc0c62d0 msgr2=0x7ff2dc1ec890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.875+0000 7ff2e14d4640 1 --2- 172.21.15.154:0/330029804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2dc0c62d0 0x7ff2dc1ec890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.875+0000 7ff2e14d4640 1 -- 172.21.15.154:0/330029804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2dc0c71d0 msgr2=0x7ff2dc152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:49.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.875+0000 7ff2e14d4640 1 --2- 172.21.15.154:0/330029804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2dc0c71d0 0x7ff2dc152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.875+0000 7ff2e14d4640 1 -- 172.21.15.154:0/330029804 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc155850 con 0x7ff2dc152fc0 2024-03-18T16:20:49.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.877+0000 7ff2e14d4640 1 --2- 172.21.15.154:0/330029804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc152fc0 0x7ff2dc154b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4034000 tx=0x7ff2d4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:49.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.880+0000 7ff2d97fa640 1 -- 172.21.15.154:0/330029804 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4045020 con 0x7ff2dc152fc0 2024-03-18T16:20:49.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.880+0000 7ff2d97fa640 1 -- 172.21.15.154:0/330029804 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d400b420 con 0x7ff2dc152fc0 2024-03-18T16:20:49.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.880+0000 7ff2d97fa640 1 -- 172.21.15.154:0/330029804 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d400b0d0 con 0x7ff2dc152fc0 2024-03-18T16:20:49.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.880+0000 7ff2e2f37640 1 -- 172.21.15.154:0/330029804 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2dc1519e0 con 0x7ff2dc152fc0 2024-03-18T16:20:49.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.881+0000 7ff2e2f37640 1 -- 172.21.15.154:0/330029804 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2dc151fa0 con 0x7ff2dc152fc0 2024-03-18T16:20:49.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.881+0000 7ff2e2f37640 1 -- 172.21.15.154:0/330029804 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2dc074fd0 con 0x7ff2dc152fc0 2024-03-18T16:20:49.943 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:49.941+0000 7f73bec3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T16:20:49.973 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:49.971+0000 7fb8c843a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T16:20:49.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:20:50.000 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.985+0000 7ff2d97fa640 1 -- 172.21.15.154:0/330029804 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff2d4048020 con 0x7ff2dc152fc0 2024-03-18T16:20:50.000 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.985+0000 7ff2d97fa640 1 -- 172.21.15.154:0/330029804 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7ff2d40407b0 con 0x7ff2dc152fc0 2024-03-18T16:20:50.000 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:49.985+0000 7ff2d97fa640 1 -- 172.21.15.154:0/330029804 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff2d404f030 con 0x7ff2dc152fc0 2024-03-18T16:20:50.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:20:50.030 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:50.028+0000 7fb8c843a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T16:20:50.076 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.060+0000 7ff2e2f37640 1 -- 172.21.15.154:0/330029804 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d7aad584-b031-4470-8d3b-fe2fd9bd7d2a", "id": 0} v 0) -- 0x7ff2dc0c6fb0 con 0x7ff2dc152fc0 2024-03-18T16:20:50.087 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:50.085+0000 7f73bec3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T16:20:50.206 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:50.204+0000 7f73bec3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T16:20:50.255 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:50.253+0000 7f73bec3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T16:20:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.292+0000 7ff2d97fa640 1 -- 172.21.15.154:0/330029804 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d7aad584-b031-4470-8d3b-fe2fd9bd7d2a", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7ff2d4036bf0 con 0x7ff2dc152fc0 2024-03-18T16:20:50.307 INFO:teuthology.orchestra.run.smithi154.stdout:0 2024-03-18T16:20:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.293+0000 7ff2e2f37640 1 -- 172.21.15.154:0/330029804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc152fc0 msgr2=0x7ff2dc154b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.293+0000 7ff2e2f37640 1 --2- 172.21.15.154:0/330029804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc152fc0 0x7ff2dc154b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4034000 tx=0x7ff2d4004a40 comp rx=0 tx=0).stop 2024-03-18T16:20:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.293+0000 7ff2e2f37640 1 -- 172.21.15.154:0/330029804 shutdown_connections 2024-03-18T16:20:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.293+0000 7ff2e2f37640 1 --2- 172.21.15.154:0/330029804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2dc152fc0 0x7ff2dc154b80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.293+0000 7ff2e2f37640 1 --2- 172.21.15.154:0/330029804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2dc0c71d0 0x7ff2dc152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.293+0000 7ff2e2f37640 1 --2- 172.21.15.154:0/330029804 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2dc0c62d0 0x7ff2dc1ec890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.293+0000 7ff2e2f37640 1 -- 172.21.15.154:0/330029804 >> 172.21.15.154:0/330029804 conn(0x7ff2dc0cc390 msgr2=0x7ff2dc13e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.293+0000 7ff2e2f37640 1 -- 172.21.15.154:0/330029804 shutdown_connections 2024-03-18T16:20:50.309 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.293+0000 7ff2e2f37640 1 -- 172.21.15.154:0/330029804 wait complete. 2024-03-18T16:20:50.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 84532350-698b-486e-9dae-4975d277af0b 1 2024-03-18T16:20:50.375 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:50.373+0000 7f73bec3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:20:50.450 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.448+0000 7fb8e90cc640 1 Processor -- start 2024-03-18T16:20:50.450 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.448+0000 7fb8e90cc640 1 -- start start 2024-03-18T16:20:50.451 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.448+0000 7fb8e90cc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb8e4071100 0x7fb8e406b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.451 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.448+0000 7fb8e90cc640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb8e40c3060 0x7fb8e4070550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.451 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.448+0000 7fb8e90cc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8e40c3530 con 0x7fb8e4154730 2024-03-18T16:20:50.451 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.448+0000 7fb8e90cc640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb8e40c3810 con 0x7fb8e40c3060 2024-03-18T16:20:50.451 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.448+0000 7fb8e90cc640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb8e40c3af0 con 0x7fb8e4071100 2024-03-18T16:20:50.451 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.449+0000 7fb8e259c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb8e4071100 0x7fb8e406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.452 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.449+0000 7fb8e259c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb8e4071100 0x7fb8e406b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:38654/0 (socket says 172.21.15.154:38654) 2024-03-18T16:20:50.452 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.449+0000 7fb8e259c640 1 -- 172.21.15.154:0/2408463857 learned_addr learned my addr 172.21.15.154:0/2408463857 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:50.453 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.451+0000 7fb8e359e640 1 --2- 172.21.15.154:0/2408463857 >> v2:172.21.15.102:3300/0 conn(0x7fb8e40c3060 0x7fb8e4070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.456 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.454+0000 7fb8e259c640 1 -- 172.21.15.154:0/2408463857 >> v2:172.21.15.102:3300/0 conn(0x7fb8e40c3060 msgr2=0x7fb8e4070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:50.456 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.454+0000 7fb8e259c640 1 --2- 172.21.15.154:0/2408463857 >> v2:172.21.15.102:3300/0 conn(0x7fb8e40c3060 0x7fb8e4070550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.456 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.454+0000 7fb8e259c640 1 -- 172.21.15.154:0/2408463857 >> v1:172.21.15.154:6789/0 conn(0x7fb8e4154730 legacy=0x7fb8e415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.456 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.454+0000 7fb8e259c640 1 -- 172.21.15.154:0/2408463857 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e415a650 con 0x7fb8e4071100 2024-03-18T16:20:50.460 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.458+0000 7fb8e259c640 1 --2- 172.21.15.154:0/2408463857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4071100 0x7fb8e406b950 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb8e406fc70 tx=0x7fb8d802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef55a54d825c3889 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:50.494 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:50.487+0000 7fb8c843a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.461+0000 7fb8e1d9b640 1 -- 172.21.15.154:0/2408463857 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d802f850 con 0x7fb8e4071100 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.461+0000 7fb8e1d9b640 1 -- 172.21.15.154:0/2408463857 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d802f9f0 con 0x7fb8e4071100 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.461+0000 7fb8e1d9b640 1 -- 172.21.15.154:0/2408463857 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d802fdc0 con 0x7fb8e4071100 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.462+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2408463857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4071100 msgr2=0x7fb8e406b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.462+0000 7fb8e90cc640 1 --2- 172.21.15.154:0/2408463857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4071100 0x7fb8e406b950 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb8e406fc70 tx=0x7fb8d802ba30 comp rx=0 tx=0).stop 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.462+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2408463857 shutdown_connections 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.462+0000 7fb8e90cc640 1 --2- 172.21.15.154:0/2408463857 >> v2:172.21.15.102:3300/0 conn(0x7fb8e40c3060 0x7fb8e4070550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.462+0000 7fb8e90cc640 1 --2- 172.21.15.154:0/2408463857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4071100 0x7fb8e406b950 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.462+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2408463857 >> 172.21.15.154:0/2408463857 conn(0x7fb8e4072930 msgr2=0x7fb8e4072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:50.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.462+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2408463857 shutdown_connections 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.462+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2408463857 wait complete. 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.464+0000 7fb8e90cc640 1 Processor -- start 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.464+0000 7fb8e90cc640 1 -- start start 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.464+0000 7fb8e90cc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8e4071100 0x7fb8e41f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.464+0000 7fb8e90cc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8e40c3060 0x7fb8e41f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.464+0000 7fb8e90cc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4154730 0x7fb8e41586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.464+0000 7fb8e90cc640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e4154b10 con 0x7fb8e40c3060 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.464+0000 7fb8e90cc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e406be90 con 0x7fb8e4154730 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.464+0000 7fb8e90cc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e415d9b0 con 0x7fb8e4071100 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.465+0000 7fb8e2d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8e4071100 0x7fb8e41f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.465+0000 7fb8e359e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4154730 0x7fb8e41586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.465+0000 7fb8e2d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8e4071100 0x7fb8e41f4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:42810/0 (socket says 172.21.15.154:42810) 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.465+0000 7fb8e2d9d640 1 -- 172.21.15.154:0/2330240786 learned_addr learned my addr 172.21.15.154:0/2330240786 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.466+0000 7fb8e259c640 1 --2- 172.21.15.154:0/2330240786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8e40c3060 0x7fb8e41f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.469+0000 7fb8e359e640 1 -- 172.21.15.154:0/2330240786 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8e4071100 msgr2=0x7fb8e41f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.469+0000 7fb8e359e640 1 --2- 172.21.15.154:0/2330240786 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8e4071100 0x7fb8e41f4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.469+0000 7fb8e359e640 1 -- 172.21.15.154:0/2330240786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8e40c3060 msgr2=0x7fb8e41f0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.469+0000 7fb8e359e640 1 --2- 172.21.15.154:0/2330240786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8e40c3060 0x7fb8e41f0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.469+0000 7fb8e359e640 1 -- 172.21.15.154:0/2330240786 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e41f38a0 con 0x7fb8e4154730 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.469+0000 7fb8e2d9d640 1 --2- 172.21.15.154:0/2330240786 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8e4071100 0x7fb8e41f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.471+0000 7fb8e359e640 1 --2- 172.21.15.154:0/2330240786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4154730 0x7fb8e41586f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00efc0 tx=0x7fb8dc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.474+0000 7fb8cb7fe640 1 -- 172.21.15.154:0/2330240786 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc018020 con 0x7fb8e4154730 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.475+0000 7fb8cb7fe640 1 -- 172.21.15.154:0/2330240786 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8dc014480 con 0x7fb8e4154730 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.475+0000 7fb8cb7fe640 1 -- 172.21.15.154:0/2330240786 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc014870 con 0x7fb8e4154730 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.475+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2330240786 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e4156910 con 0x7fb8e4154730 2024-03-18T16:20:50.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.475+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2330240786 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e41f3b40 con 0x7fb8e4154730 2024-03-18T16:20:50.500 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:50.475+0000 7f73bec3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T16:20:50.500 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.476+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2330240786 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e4074fd0 con 0x7fb8e4154730 2024-03-18T16:20:50.500 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.478+0000 7fb8cb7fe640 1 -- 172.21.15.154:0/2330240786 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb8dc005d10 con 0x7fb8e4154730 2024-03-18T16:20:50.500 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.478+0000 7fb8cb7fe640 1 -- 172.21.15.154:0/2330240786 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fb8dc0267a0 con 0x7fb8e4154730 2024-03-18T16:20:50.580 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.578+0000 7fb8cb7fe640 1 -- 172.21.15.154:0/2330240786 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb8dc022030 con 0x7fb8e4154730 2024-03-18T16:20:50.637 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:50.635+0000 7f73bec3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:20:50.657 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:50.646+0000 7fb8c843a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T16:20:50.658 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.652+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2330240786 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "84532350-698b-486e-9dae-4975d277af0b", "id": 1} v 0) -- 0x7fb8e40cb140 con 0x7fb8e4154730 2024-03-18T16:20:50.681 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:50.678+0000 7f73bec3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T16:20:50.796 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:50.794+0000 7fb8c843a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.803+0000 7fb8cb7fe640 1 -- 172.21.15.154:0/2330240786 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "84532350-698b-486e-9dae-4975d277af0b", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fb8dc010040 con 0x7fb8e4154730 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi154.stdout:1 2024-03-18T16:20:50.807 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.804+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2330240786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4154730 msgr2=0x7fb8e41586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.807 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.804+0000 7fb8e90cc640 1 --2- 172.21.15.154:0/2330240786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4154730 0x7fb8e41586f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00efc0 tx=0x7fb8dc00c490 comp rx=0 tx=0).stop 2024-03-18T16:20:50.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.804+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2330240786 shutdown_connections 2024-03-18T16:20:50.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.804+0000 7fb8e90cc640 1 --2- 172.21.15.154:0/2330240786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8e4154730 0x7fb8e41586f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.804+0000 7fb8e90cc640 1 --2- 172.21.15.154:0/2330240786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8e40c3060 0x7fb8e41f0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.804+0000 7fb8e90cc640 1 --2- 172.21.15.154:0/2330240786 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8e4071100 0x7fb8e41f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.804+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2330240786 >> 172.21.15.154:0/2330240786 conn(0x7fb8e4072930 msgr2=0x7fb8e4072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:50.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.805+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2330240786 shutdown_connections 2024-03-18T16:20:50.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.805+0000 7fb8e90cc640 1 -- 172.21.15.154:0/2330240786 wait complete. 2024-03-18T16:20:50.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new a0705f8f-0b79-4084-b22e-3a4df2150f22 2 2024-03-18T16:20:50.851 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:50.849+0000 7fb8c843a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T16:20:50.867 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:50.856+0000 7f73bec3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T16:20:50.983 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.981+0000 7f2bbc137640 1 Processor -- start 2024-03-18T16:20:50.983 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.981+0000 7f2bbc137640 1 -- start start 2024-03-18T16:20:50.984 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:50.982+0000 7fb8c843a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:20:50.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.982+0000 7f2bbc137640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40caac0 0x7f2bb406cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.982+0000 7f2bbc137640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb406d650 con 0x7f2bb40ca6e0 2024-03-18T16:20:50.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.982+0000 7f2bbc137640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb415b5b0 con 0x7f2bb406d1c0 2024-03-18T16:20:50.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.982+0000 7f2bbc137640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb415a770 con 0x7f2bb40caac0 2024-03-18T16:20:50.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.982+0000 7f2bb96d2640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40caac0 0x7f2bb406cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.982+0000 7f2bba6d4640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f2bb406d1c0 0x7f2bb4164f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:50918/0 (socket says 172.21.15.154:50918) 2024-03-18T16:20:50.985 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.982+0000 7f2bba6d4640 1 -- 172.21.15.154:0/1574483149 learned_addr learned my addr 172.21.15.154:0/1574483149 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-03-18T16:20:50.987 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.985+0000 7f2bb96d2640 1 -- 172.21.15.154:0/1574483149 >> v1:172.21.15.154:6789/0 conn(0x7f2bb406d1c0 legacy=0x7f2bb4164f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.987 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.985+0000 7f2bb96d2640 1 -- 172.21.15.154:0/1574483149 >> v1:172.21.15.102:6789/0 conn(0x7f2bb40ca6e0 legacy=0x7f2bb40cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:50.987 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.985+0000 7f2bb96d2640 1 -- 172.21.15.154:0/1574483149 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb415aa30 con 0x7f2bb40caac0 2024-03-18T16:20:50.989 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.987+0000 7f2bb96d2640 1 --2- 172.21.15.154:0/1574483149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40caac0 0x7f2bb406cc80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40097d0 tx=0x7f2ba402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a4d6b8a386b5f5b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:50.994 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.992+0000 7f2bb8ed1640 1 -- 172.21.15.154:0/1574483149 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4031770 con 0x7f2bb40caac0 2024-03-18T16:20:50.994 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.992+0000 7f2bb8ed1640 1 -- 172.21.15.154:0/1574483149 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ba4037070 con 0x7f2bb40caac0 2024-03-18T16:20:50.994 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.992+0000 7f2bb8ed1640 1 -- 172.21.15.154:0/1574483149 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4031b30 con 0x7f2bb40caac0 2024-03-18T16:20:50.995 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.993+0000 7f2bbc137640 1 -- 172.21.15.154:0/1574483149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40caac0 msgr2=0x7f2bb406cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.995 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.993+0000 7f2bbc137640 1 --2- 172.21.15.154:0/1574483149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40caac0 0x7f2bb406cc80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40097d0 tx=0x7f2ba402b810 comp rx=0 tx=0).stop 2024-03-18T16:20:50.996 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.993+0000 7f2bbc137640 1 -- 172.21.15.154:0/1574483149 shutdown_connections 2024-03-18T16:20:50.996 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.993+0000 7f2bbc137640 1 --2- 172.21.15.154:0/1574483149 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40caac0 0x7f2bb406cc80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.996 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.993+0000 7f2bbc137640 1 -- 172.21.15.154:0/1574483149 >> 172.21.15.154:0/1574483149 conn(0x7f2bb40b18b0 msgr2=0x7f2bb40b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:50.996 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.993+0000 7f2bbc137640 1 -- 172.21.15.154:0/1574483149 shutdown_connections 2024-03-18T16:20:50.996 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.993+0000 7f2bbc137640 1 -- 172.21.15.154:0/1574483149 wait complete. 2024-03-18T16:20:50.996 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.994+0000 7f2bbc137640 1 Processor -- start 2024-03-18T16:20:50.996 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.994+0000 7f2bbc137640 1 -- start start 2024-03-18T16:20:50.997 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.995+0000 7f2bbc137640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb406d1c0 0x7f2bb414caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.048 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.995+0000 7f2bbc137640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40ca6e0 0x7f2bb41486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.049 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.995+0000 7f2bbc137640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb40caac0 0x7f2bb4155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.049 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.995+0000 7f2bbc137640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb415b2d0 con 0x7f2bb40caac0 2024-03-18T16:20:51.049 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.995+0000 7f2bbc137640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb415aff0 con 0x7f2bb406d1c0 2024-03-18T16:20:51.049 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.995+0000 7f2bbc137640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb40caf20 con 0x7f2bb40ca6e0 2024-03-18T16:20:51.050 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.996+0000 7f2bb9ed3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb406d1c0 0x7f2bb414caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.996+0000 7f2bb9ed3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb406d1c0 0x7f2bb414caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:38670/0 (socket says 172.21.15.154:38670) 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.996+0000 7f2bb9ed3640 1 -- 172.21.15.154:0/1244649029 learned_addr learned my addr 172.21.15.154:0/1244649029 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:50.997+0000 7f2bba6d4640 1 --2- 172.21.15.154:0/1244649029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb40caac0 0x7f2bb4155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.000+0000 7f2bb9ed3640 1 -- 172.21.15.154:0/1244649029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40ca6e0 msgr2=0x7f2bb41486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.000+0000 7f2bb9ed3640 1 --2- 172.21.15.154:0/1244649029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40ca6e0 0x7f2bb41486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.000+0000 7f2bb9ed3640 1 -- 172.21.15.154:0/1244649029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb40caac0 msgr2=0x7f2bb4155080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.000+0000 7f2bb9ed3640 1 --2- 172.21.15.154:0/1244649029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb40caac0 0x7f2bb4155080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.000+0000 7f2bb9ed3640 1 -- 172.21.15.154:0/1244649029 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb4157ba0 con 0x7f2bb406d1c0 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.002+0000 7f2bb9ed3640 1 --2- 172.21.15.154:0/1244649029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb406d1c0 0x7f2bb414caa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2bb000c410 tx=0x7f2bb000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.007+0000 7f2baa7fc640 1 -- 172.21.15.154:0/1244649029 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0017070 con 0x7f2bb406d1c0 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.007+0000 7f2baa7fc640 1 -- 172.21.15.154:0/1244649029 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bb00035f0 con 0x7f2bb406d1c0 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.007+0000 7f2baa7fc640 1 -- 172.21.15.154:0/1244649029 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0003a20 con 0x7f2bb406d1c0 2024-03-18T16:20:51.051 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.007+0000 7f2bbc137640 1 -- 172.21.15.154:0/1244649029 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb4155f30 con 0x7f2bb406d1c0 2024-03-18T16:20:51.052 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.007+0000 7f2bbc137640 1 -- 172.21.15.154:0/1244649029 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb41564f0 con 0x7f2bb406d1c0 2024-03-18T16:20:51.052 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.008+0000 7f2bbc137640 1 -- 172.21.15.154:0/1244649029 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb4074fd0 con 0x7f2bb406d1c0 2024-03-18T16:20:51.095 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:51.094+0000 7fb8c843a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T16:20:51.129 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.110+0000 7f2baa7fc640 1 -- 172.21.15.154:0/1244649029 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2bb0003bc0 con 0x7f2bb406d1c0 2024-03-18T16:20:51.129 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.110+0000 7f2baa7fc640 1 -- 172.21.15.154:0/1244649029 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f2bb001a360 con 0x7f2bb406d1c0 2024-03-18T16:20:51.129 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.110+0000 7f2baa7fc640 1 -- 172.21.15.154:0/1244649029 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2bb0020080 con 0x7f2bb406d1c0 2024-03-18T16:20:51.185 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.182+0000 7f2bbc137640 1 -- 172.21.15.154:0/1244649029 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a0705f8f-0b79-4084-b22e-3a4df2150f22", "id": 2} v 0) -- 0x7f2bb40b5ce0 con 0x7f2bb406d1c0 2024-03-18T16:20:51.275 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:51.273+0000 7fb8c843a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:20:51.326 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:51.324+0000 7fb8c843a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T16:20:51.412 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.410+0000 7f2baa7fc640 1 -- 172.21.15.154:0/1244649029 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a0705f8f-0b79-4084-b22e-3a4df2150f22", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f2bb0006b50 con 0x7f2bb406d1c0 2024-03-18T16:20:51.445 INFO:teuthology.orchestra.run.smithi154.stdout:2 2024-03-18T16:20:51.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.411+0000 7f2bbc137640 1 -- 172.21.15.154:0/1244649029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb406d1c0 msgr2=0x7f2bb414caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.411+0000 7f2bbc137640 1 --2- 172.21.15.154:0/1244649029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb406d1c0 0x7f2bb414caa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2bb000c410 tx=0x7f2bb000c8d0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.411+0000 7f2bbc137640 1 -- 172.21.15.154:0/1244649029 shutdown_connections 2024-03-18T16:20:51.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.411+0000 7f2bbc137640 1 --2- 172.21.15.154:0/1244649029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb40caac0 0x7f2bb4155080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.411+0000 7f2bbc137640 1 --2- 172.21.15.154:0/1244649029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2bb40ca6e0 0x7f2bb41486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.411+0000 7f2bbc137640 1 --2- 172.21.15.154:0/1244649029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bb406d1c0 0x7f2bb414caa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.411+0000 7f2bbc137640 1 -- 172.21.15.154:0/1244649029 >> 172.21.15.154:0/1244649029 conn(0x7f2bb40b18b0 msgr2=0x7f2bb41544c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:51.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.412+0000 7f2bbc137640 1 -- 172.21.15.154:0/1244649029 shutdown_connections 2024-03-18T16:20:51.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.412+0000 7f2bbc137640 1 -- 172.21.15.154:0/1244649029 wait complete. 2024-03-18T16:20:51.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 014e4711-688e-4cc2-a8c7-85d8b6a9c5f0 3 2024-03-18T16:20:51.502 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:51.500+0000 7fb8c843a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T16:20:51.549 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.546+0000 7f246970a640 1 Processor -- start 2024-03-18T16:20:51.549 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.547+0000 7f246970a640 1 -- start start 2024-03-18T16:20:51.549 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.547+0000 7f246970a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2464154990 0x7f24640cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.550 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.547+0000 7f246970a640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24640c51a0 0x7f246415db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.550 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.547+0000 7f246970a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246415e0c0 0x7f24640d0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.550 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.547+0000 7f246970a640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f24640c4ff0 con 0x7f24640c51a0 2024-03-18T16:20:51.550 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.547+0000 7f246970a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f24640ccc20 con 0x7f2464154990 2024-03-18T16:20:51.550 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.547+0000 7f246970a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2464154d70 con 0x7f246415e0c0 2024-03-18T16:20:51.550 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.548+0000 7f2462ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2464154990 0x7f24640cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.551 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.548+0000 7f24637fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246415e0c0 0x7f24640d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.551 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.548+0000 7f24637fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246415e0c0 0x7f24640d0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:42826/0 (socket says 172.21.15.154:42826) 2024-03-18T16:20:51.551 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.548+0000 7f24637fe640 1 -- 172.21.15.154:0/784691379 learned_addr learned my addr 172.21.15.154:0/784691379 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:51.552 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.550+0000 7f24627fc640 1 --2- 172.21.15.154:0/784691379 >> v2:172.21.15.102:3300/0 conn(0x7f24640c51a0 0x7f246415db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.550+0000 7f24637fe640 1 -- 172.21.15.154:0/784691379 >> v2:172.21.15.154:3300/0 conn(0x7f2464154990 msgr2=0x7f24640cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:51.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.550+0000 7f24637fe640 1 --2- 172.21.15.154:0/784691379 >> v2:172.21.15.154:3300/0 conn(0x7f2464154990 0x7f24640cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.550+0000 7f24637fe640 1 -- 172.21.15.154:0/784691379 >> v2:172.21.15.102:3300/0 conn(0x7f24640c51a0 msgr2=0x7f246415db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:51.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.550+0000 7f24637fe640 1 --2- 172.21.15.154:0/784691379 >> v2:172.21.15.102:3300/0 conn(0x7f24640c51a0 0x7f246415db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.551+0000 7f24637fe640 1 -- 172.21.15.154:0/784691379 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246415e5c0 con 0x7f246415e0c0 2024-03-18T16:20:51.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.552+0000 7f24637fe640 1 --2- 172.21.15.154:0/784691379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246415e0c0 0x7f24640d0320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f246406d690 tx=0x7f245c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35682c273eb5b400 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:51.556 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.554+0000 7f2461ffb640 1 -- 172.21.15.154:0/784691379 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c02cb00 con 0x7f246415e0c0 2024-03-18T16:20:51.558 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.556+0000 7f2461ffb640 1 -- 172.21.15.154:0/784691379 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f245c02cca0 con 0x7f246415e0c0 2024-03-18T16:20:51.558 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.556+0000 7f2461ffb640 1 -- 172.21.15.154:0/784691379 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245c034610 con 0x7f246415e0c0 2024-03-18T16:20:51.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.557+0000 7f246970a640 1 -- 172.21.15.154:0/784691379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246415e0c0 msgr2=0x7f24640d0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.557+0000 7f246970a640 1 --2- 172.21.15.154:0/784691379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246415e0c0 0x7f24640d0320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f246406d690 tx=0x7f245c02bf60 comp rx=0 tx=0).stop 2024-03-18T16:20:51.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.557+0000 7f246970a640 1 -- 172.21.15.154:0/784691379 shutdown_connections 2024-03-18T16:20:51.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.557+0000 7f246970a640 1 --2- 172.21.15.154:0/784691379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f246415e0c0 0x7f24640d0320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.557+0000 7f246970a640 1 --2- 172.21.15.154:0/784691379 >> v2:172.21.15.102:3300/0 conn(0x7f24640c51a0 0x7f246415db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.557+0000 7f246970a640 1 --2- 172.21.15.154:0/784691379 >> v2:172.21.15.154:3300/0 conn(0x7f2464154990 0x7f24640cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.557+0000 7f246970a640 1 -- 172.21.15.154:0/784691379 >> 172.21.15.154:0/784691379 conn(0x7f246406cb00 msgr2=0x7f246406cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:51.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.557+0000 7f246970a640 1 -- 172.21.15.154:0/784691379 shutdown_connections 2024-03-18T16:20:51.559 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.557+0000 7f246970a640 1 -- 172.21.15.154:0/784691379 wait complete. 2024-03-18T16:20:51.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.558+0000 7f246970a640 1 Processor -- start 2024-03-18T16:20:51.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.558+0000 7f246970a640 1 -- start start 2024-03-18T16:20:51.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f246970a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24640c51a0 0x7f24640ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.562 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f246970a640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2464154990 0x7f24640c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f246970a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f246415e0c0 0x7f24641f08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f246970a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24640cc8b0 con 0x7f246415e0c0 2024-03-18T16:20:51.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f246970a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24640cc5d0 con 0x7f24640c51a0 2024-03-18T16:20:51.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f246970a640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24640cd5f0 con 0x7f2464154990 2024-03-18T16:20:51.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f24627fc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2464154990 0x7f24640c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f2462ffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24640c51a0 0x7f24640ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f24627fc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2464154990 0x7f24640c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:42842/0 (socket says 172.21.15.154:42842) 2024-03-18T16:20:51.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.559+0000 7f24627fc640 1 -- 172.21.15.154:0/1592894475 learned_addr learned my addr 172.21.15.154:0/1592894475 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:51.563 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.561+0000 7f24637fe640 1 --2- 172.21.15.154:0/1592894475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f246415e0c0 0x7f24641f08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.563+0000 7f24627fc640 1 -- 172.21.15.154:0/1592894475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24640c51a0 msgr2=0x7f24640ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.563+0000 7f24627fc640 1 --2- 172.21.15.154:0/1592894475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24640c51a0 0x7f24640ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.563+0000 7f24627fc640 1 -- 172.21.15.154:0/1592894475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f246415e0c0 msgr2=0x7f24641f08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.563+0000 7f24627fc640 1 --2- 172.21.15.154:0/1592894475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f246415e0c0 0x7f24641f08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.563+0000 7f24627fc640 1 -- 172.21.15.154:0/1592894475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f246415b0d0 con 0x7f2464154990 2024-03-18T16:20:51.565 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.563+0000 7f24637fe640 1 --2- 172.21.15.154:0/1592894475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f246415e0c0 0x7f24641f08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:51.567 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.565+0000 7f24627fc640 1 --2- 172.21.15.154:0/1592894475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2464154990 0x7f24640c70d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f245400c7f0 tx=0x7f245400ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:51.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.568+0000 7f244b7fe640 1 -- 172.21.15.154:0/1592894475 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454015070 con 0x7f2464154990 2024-03-18T16:20:51.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.568+0000 7f244b7fe640 1 -- 172.21.15.154:0/1592894475 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2454004220 con 0x7f2464154990 2024-03-18T16:20:51.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.568+0000 7f244b7fe640 1 -- 172.21.15.154:0/1592894475 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454007620 con 0x7f2464154990 2024-03-18T16:20:51.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.568+0000 7f246970a640 1 -- 172.21.15.154:0/1592894475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f246415b3b0 con 0x7f2464154990 2024-03-18T16:20:51.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.569+0000 7f246970a640 1 -- 172.21.15.154:0/1592894475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2464150340 con 0x7f2464154990 2024-03-18T16:20:51.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.569+0000 7f246970a640 1 -- 172.21.15.154:0/1592894475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2464074fd0 con 0x7f2464154990 2024-03-18T16:20:51.576 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.574+0000 7f244b7fe640 1 -- 172.21.15.154:0/1592894475 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f24540079d0 con 0x7f2464154990 2024-03-18T16:20:51.696 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:51.694+0000 7f73bec3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T16:20:51.745 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.743+0000 7f244b7fe640 1 -- 172.21.15.154:0/1592894475 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f2454007cc0 con 0x7f2464154990 2024-03-18T16:20:51.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.743+0000 7f244b7fe640 1 -- 172.21.15.154:0/1592894475 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2454003610 con 0x7f2464154990 2024-03-18T16:20:51.819 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.816+0000 7f246970a640 1 -- 172.21.15.154:0/1592894475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd new", "uuid": "014e4711-688e-4cc2-a8c7-85d8b6a9c5f0", "id": 3} v 0) -- 0x7f24640b5750 con 0x7f2464154990 2024-03-18T16:20:51.975 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.972+0000 7f244b7fe640 1 -- 172.21.15.154:0/1592894475 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "014e4711-688e-4cc2-a8c7-85d8b6a9c5f0", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f24540037f0 con 0x7f2464154990 2024-03-18T16:20:51.975 INFO:teuthology.orchestra.run.smithi154.stdout:3 2024-03-18T16:20:51.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.974+0000 7f246970a640 1 -- 172.21.15.154:0/1592894475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2464154990 msgr2=0x7f24640c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.974+0000 7f246970a640 1 --2- 172.21.15.154:0/1592894475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2464154990 0x7f24640c70d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f245400c7f0 tx=0x7f245400ccb0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.974+0000 7f246970a640 1 -- 172.21.15.154:0/1592894475 shutdown_connections 2024-03-18T16:20:51.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.974+0000 7f246970a640 1 --2- 172.21.15.154:0/1592894475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f246415e0c0 0x7f24641f08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.974+0000 7f246970a640 1 --2- 172.21.15.154:0/1592894475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2464154990 0x7f24640c70d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.974+0000 7f246970a640 1 --2- 172.21.15.154:0/1592894475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24640c51a0 0x7f24640ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.977 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.974+0000 7f246970a640 1 -- 172.21.15.154:0/1592894475 >> 172.21.15.154:0/1592894475 conn(0x7f246406cb00 msgr2=0x7f246415d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:51.978 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.974+0000 7f246970a640 1 -- 172.21.15.154:0/1592894475 shutdown_connections 2024-03-18T16:20:51.978 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:51.974+0000 7f246970a640 1 -- 172.21.15.154:0/1592894475 wait complete. 2024-03-18T16:20:51.984 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:51.982+0000 7f73bec3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T16:20:51.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 67312bab-afbc-4612-9ab8-f0e5f907cb09 4 2024-03-18T16:20:52.045 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:52.042+0000 7f73bec3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T16:20:52.090 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.088+0000 7fc842b76640 1 Processor -- start 2024-03-18T16:20:52.091 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.088+0000 7fc842b76640 1 -- start start 2024-03-18T16:20:52.092 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.089+0000 7fc842b76640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0c22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:52.092 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.089+0000 7fc842b76640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc83c070ea0 con 0x7fc83c151520 2024-03-18T16:20:52.092 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.090+0000 7fc842b76640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc83c071180 con 0x7fc83c157590 2024-03-18T16:20:52.092 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.090+0000 7fc842b76640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc83c0c82c0 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.092 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.090+0000 7fc841113640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0c22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:52.092 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.090+0000 7fc83bfff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc83c157590 0x7fc83c1579c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:50922/0 (socket says 172.21.15.154:50922) 2024-03-18T16:20:52.092 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.090+0000 7fc83bfff640 1 -- 172.21.15.154:0/1070043588 learned_addr learned my addr 172.21.15.154:0/1070043588 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-03-18T16:20:52.094 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.092+0000 7fc841113640 1 -- 172.21.15.154:0/1070043588 >> v1:172.21.15.154:6789/0 conn(0x7fc83c157590 legacy=0x7fc83c1579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.094 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.092+0000 7fc841113640 1 -- 172.21.15.154:0/1070043588 >> v1:172.21.15.102:6789/0 conn(0x7fc83c151520 legacy=0x7fc83c15b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:52.095 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.092+0000 7fc841113640 1 -- 172.21.15.154:0/1070043588 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c0c7d00 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.096 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.094+0000 7fc841113640 1 --2- 172.21.15.154:0/1070043588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0c22e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc8240097d0 tx=0x7fc82402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e52752515c9851a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:52.100 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.098+0000 7fc83b7fe640 1 -- 172.21.15.154:0/1070043588 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc824035020 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.100 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.098+0000 7fc83b7fe640 1 -- 172.21.15.154:0/1070043588 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc824031890 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.100 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.098+0000 7fc83b7fe640 1 -- 172.21.15.154:0/1070043588 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc824031c60 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.101 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.099+0000 7fc842b76640 1 -- 172.21.15.154:0/1070043588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 msgr2=0x7fc83c0c22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.101 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.099+0000 7fc842b76640 1 --2- 172.21.15.154:0/1070043588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0c22e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc8240097d0 tx=0x7fc82402b930 comp rx=0 tx=0).stop 2024-03-18T16:20:52.102 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.099+0000 7fc842b76640 1 -- 172.21.15.154:0/1070043588 shutdown_connections 2024-03-18T16:20:52.102 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.099+0000 7fc842b76640 1 --2- 172.21.15.154:0/1070043588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0c22e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.102 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.099+0000 7fc842b76640 1 -- 172.21.15.154:0/1070043588 >> 172.21.15.154:0/1070043588 conn(0x7fc83c0729d0 msgr2=0x7fc83c072db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:52.102 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.099+0000 7fc842b76640 1 -- 172.21.15.154:0/1070043588 shutdown_connections 2024-03-18T16:20:52.102 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.099+0000 7fc842b76640 1 -- 172.21.15.154:0/1070043588 wait complete. 2024-03-18T16:20:52.102 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.100+0000 7fc842b76640 1 Processor -- start 2024-03-18T16:20:52.103 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.100+0000 7fc842b76640 1 -- start start 2024-03-18T16:20:52.103 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.101+0000 7fc842b76640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:52.103 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.101+0000 7fc842b76640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83c151520 0x7fc83c148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:52.103 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.101+0000 7fc842b76640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83c157590 0x7fc83c153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:52.103 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.101+0000 7fc842b76640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c0c2820 con 0x7fc83c151520 2024-03-18T16:20:52.103 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.101+0000 7fc842b76640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c0b2ee0 con 0x7fc83c157590 2024-03-18T16:20:52.103 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.101+0000 7fc842b76640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc83c0c29a0 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.104 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.101+0000 7fc840912640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:52.104 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.102+0000 7fc841113640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83c157590 0x7fc83c153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:52.104 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.102+0000 7fc840912640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0cc020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:42860/0 (socket says 172.21.15.154:42860) 2024-03-18T16:20:52.104 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.102+0000 7fc840912640 1 -- 172.21.15.154:0/1159292113 learned_addr learned my addr 172.21.15.154:0/1159292113 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:52.105 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.103+0000 7fc83bfff640 1 --2- 172.21.15.154:0/1159292113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83c151520 0x7fc83c148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:52.108 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.105+0000 7fc840912640 1 -- 172.21.15.154:0/1159292113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83c157590 msgr2=0x7fc83c153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.168 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.105+0000 7fc840912640 1 --2- 172.21.15.154:0/1159292113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83c157590 0x7fc83c153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.168 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.105+0000 7fc840912640 1 -- 172.21.15.154:0/1159292113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83c151520 msgr2=0x7fc83c148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.168 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.105+0000 7fc840912640 1 --2- 172.21.15.154:0/1159292113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83c151520 0x7fc83c148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.105+0000 7fc840912640 1 -- 172.21.15.154:0/1159292113 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c0cadc0 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.169 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.105+0000 7fc83bfff640 1 --2- 172.21.15.154:0/1159292113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83c151520 0x7fc83c148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:52.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.107+0000 7fc840912640 1 --2- 172.21.15.154:0/1159292113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0cc020 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc83400c7d0 tx=0x7fc83400cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:52.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.111+0000 7fc8397fa640 1 -- 172.21.15.154:0/1159292113 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc834017070 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.111+0000 7fc8397fa640 1 -- 172.21.15.154:0/1159292113 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc834003da0 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.111+0000 7fc842b76640 1 -- 172.21.15.154:0/1159292113 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83c1545d0 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.111+0000 7fc8397fa640 1 -- 172.21.15.154:0/1159292113 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8340052b0 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.111+0000 7fc842b76640 1 -- 172.21.15.154:0/1159292113 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83c0cb030 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.171 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.112+0000 7fc842b76640 1 -- 172.21.15.154:0/1159292113 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc83c074fd0 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.173 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:52.168+0000 7f73bec3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T16:20:52.214 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.211+0000 7fc8397fa640 1 -- 172.21.15.154:0/1159292113 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc83401a070 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.214 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.211+0000 7fc8397fa640 1 -- 172.21.15.154:0/1159292113 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fc83400ad10 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.214 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.212+0000 7fc8397fa640 1 -- 172.21.15.154:0/1159292113 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc834021020 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.287 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.285+0000 7fc842b76640 1 -- 172.21.15.154:0/1159292113 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd new", "uuid": "67312bab-afbc-4612-9ab8-f0e5f907cb09", "id": 4} v 0) -- 0x7fc83c0c78d0 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.557 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:52.554+0000 7fb8c843a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T16:20:52.815 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.813+0000 7fc8397fa640 1 -- 172.21.15.154:0/1159292113 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mgrmap(e 2) ==== 45093+0+0 (secure 0 0 0) 0x7fc83400a5e0 con 0x7fc83c0c1eb0 2024-03-18T16:20:52.923 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:52.872+0000 7fb8c843a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T16:20:52.936 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:52.934+0000 7fb8c843a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T16:20:52.966 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.964+0000 7fc8397fa640 1 -- 172.21.15.154:0/1159292113 <== mon.2 v2:172.21.15.154:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "67312bab-afbc-4612-9ab8-f0e5f907cb09", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fc83401f5b0 con 0x7fc83c0c1eb0 2024-03-18T16:20:53.028 INFO:teuthology.orchestra.run.smithi154.stdout:4 2024-03-18T16:20:53.029 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.965+0000 7fc842b76640 1 -- 172.21.15.154:0/1159292113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 msgr2=0x7fc83c0cc020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:53.029 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.965+0000 7fc842b76640 1 --2- 172.21.15.154:0/1159292113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0cc020 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc83400c7d0 tx=0x7fc83400cc90 comp rx=0 tx=0).stop 2024-03-18T16:20:53.029 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.965+0000 7fc842b76640 1 -- 172.21.15.154:0/1159292113 shutdown_connections 2024-03-18T16:20:53.029 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.965+0000 7fc842b76640 1 --2- 172.21.15.154:0/1159292113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc83c157590 0x7fc83c153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.030 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.965+0000 7fc842b76640 1 --2- 172.21.15.154:0/1159292113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83c151520 0x7fc83c148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.030 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.965+0000 7fc842b76640 1 --2- 172.21.15.154:0/1159292113 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc83c0c1eb0 0x7fc83c0cc020 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.030 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.965+0000 7fc842b76640 1 -- 172.21.15.154:0/1159292113 >> 172.21.15.154:0/1159292113 conn(0x7fc83c0729d0 msgr2=0x7fc83c14fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:53.030 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.965+0000 7fc842b76640 1 -- 172.21.15.154:0/1159292113 shutdown_connections 2024-03-18T16:20:53.030 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:52.965+0000 7fc842b76640 1 -- 172.21.15.154:0/1159292113 wait complete. 2024-03-18T16:20:53.030 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 5e5e7bc5-c518-4f91-b6ce-3754524f5a56 5 2024-03-18T16:20:53.068 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-18T16:20:53.066+0000 7fb8c843a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T16:20:53.156 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.154+0000 7f9f61bbc640 1 Processor -- start 2024-03-18T16:20:53.315 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.154+0000 7f9f61bbc640 1 -- start start 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.155+0000 7f9f61bbc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152260 0x7f9f5c0cb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.155+0000 7f9f61bbc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9f5c152640 0x7f9f5c158260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.155+0000 7f9f61bbc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f5c1591b0 con 0x7f9f5c15c460 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.155+0000 7f9f61bbc640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9f5c158930 con 0x7f9f5c152640 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.155+0000 7f9f61bbc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9f5c0c92a0 con 0x7f9f5c152260 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.162+0000 7f9f5b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152260 0x7f9f5c0cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.163+0000 7f9f5b7fe640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152260 0x7f9f5c0cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:42910/0 (socket says 172.21.15.154:42910) 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.163+0000 7f9f5b7fe640 1 -- 172.21.15.154:0/1389211917 learned_addr learned my addr 172.21.15.154:0/1389211917 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.166+0000 7f9f5b7fe640 1 -- 172.21.15.154:0/1389211917 >> v2:172.21.15.154:3300/0 conn(0x7f9f5c152640 msgr2=0x7f9f5c158260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.166+0000 7f9f5b7fe640 1 --2- 172.21.15.154:0/1389211917 >> v2:172.21.15.154:3300/0 conn(0x7f9f5c152640 0x7f9f5c158260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.320 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.166+0000 7f9f5b7fe640 1 -- 172.21.15.154:0/1389211917 >> v1:172.21.15.154:6789/0 conn(0x7f9f5c15c460 legacy=0x7f9f5c0c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.166+0000 7f9f5b7fe640 1 -- 172.21.15.154:0/1389211917 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f5c158bf0 con 0x7f9f5c152260 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.168+0000 7f9f5b7fe640 1 --2- 172.21.15.154:0/1389211917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152260 0x7f9f5c0cb290 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009870 tx=0x7f9f4c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e441e48132fdcc8f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.173+0000 7f9f5a7fc640 1 -- 172.21.15.154:0/1389211917 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c035020 con 0x7f9f5c152260 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.173+0000 7f9f5a7fc640 1 -- 172.21.15.154:0/1389211917 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4c031640 con 0x7f9f5c152260 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.173+0000 7f9f5a7fc640 1 -- 172.21.15.154:0/1389211917 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c02d040 con 0x7f9f5c152260 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.174+0000 7f9f61bbc640 1 -- 172.21.15.154:0/1389211917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152260 msgr2=0x7f9f5c0cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.174+0000 7f9f61bbc640 1 --2- 172.21.15.154:0/1389211917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152260 0x7f9f5c0cb290 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009870 tx=0x7f9f4c02b8f0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.174+0000 7f9f61bbc640 1 -- 172.21.15.154:0/1389211917 shutdown_connections 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.174+0000 7f9f61bbc640 1 --2- 172.21.15.154:0/1389211917 >> v2:172.21.15.154:3300/0 conn(0x7f9f5c152640 0x7f9f5c158260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.174+0000 7f9f61bbc640 1 --2- 172.21.15.154:0/1389211917 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152260 0x7f9f5c0cb290 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.174+0000 7f9f61bbc640 1 -- 172.21.15.154:0/1389211917 >> 172.21.15.154:0/1389211917 conn(0x7f9f5c06b8e0 msgr2=0x7f9f5c06f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:53.321 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.174+0000 7f9f61bbc640 1 -- 172.21.15.154:0/1389211917 shutdown_connections 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.174+0000 7f9f61bbc640 1 -- 172.21.15.154:0/1389211917 wait complete. 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.175+0000 7f9f61bbc640 1 Processor -- start 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.175+0000 7f9f61bbc640 1 -- start start 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.176+0000 7f9f61bbc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f5c152260 0x7f9f5c1e4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.176+0000 7f9f61bbc640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152640 0x7f9f5c0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.176+0000 7f9f61bbc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f5c15c460 0x7f9f5c14d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.176+0000 7f9f61bbc640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5c15bc60 con 0x7f9f5c15c460 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.176+0000 7f9f61bbc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5c15b980 con 0x7f9f5c152260 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.176+0000 7f9f61bbc640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9f5c15bf40 con 0x7f9f5c152640 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.177+0000 7f9f5affd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152640 0x7f9f5c0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.177+0000 7f9f5affd640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152640 0x7f9f5c0c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.154:42926/0 (socket says 172.21.15.154:42926) 2024-03-18T16:20:53.322 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.177+0000 7f9f5affd640 1 -- 172.21.15.154:0/3930552678 learned_addr learned my addr 172.21.15.154:0/3930552678 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.180+0000 7f9f5affd640 1 -- 172.21.15.154:0/3930552678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f5c152260 msgr2=0x7f9f5c1e4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.180+0000 7f9f5affd640 1 --2- 172.21.15.154:0/3930552678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f5c152260 0x7f9f5c1e4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.180+0000 7f9f5affd640 1 -- 172.21.15.154:0/3930552678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f5c15c460 msgr2=0x7f9f5c14d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.180+0000 7f9f5affd640 1 --2- 172.21.15.154:0/3930552678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f5c15c460 0x7f9f5c14d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.180+0000 7f9f5affd640 1 -- 172.21.15.154:0/3930552678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f5c0b2090 con 0x7f9f5c152640 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.182+0000 7f9f5affd640 1 --2- 172.21.15.154:0/3930552678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152640 0x7f9f5c0c6fe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000c440 tx=0x7f9f5000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.187+0000 7f9f43fff640 1 -- 172.21.15.154:0/3930552678 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f50017070 con 0x7f9f5c152640 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.187+0000 7f9f43fff640 1 -- 172.21.15.154:0/3930552678 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f500035f0 con 0x7f9f5c152640 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.187+0000 7f9f43fff640 1 -- 172.21.15.154:0/3930552678 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f500039e0 con 0x7f9f5c152640 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.187+0000 7f9f61bbc640 1 -- 172.21.15.154:0/3930552678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f5c0c5e40 con 0x7f9f5c152640 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.187+0000 7f9f61bbc640 1 -- 172.21.15.154:0/3930552678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f5c0b2610 con 0x7f9f5c152640 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.188+0000 7f9f61bbc640 1 -- 172.21.15.154:0/3930552678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f5c074fd0 con 0x7f9f5c152640 2024-03-18T16:20:53.323 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.201+0000 7f9f43fff640 1 -- 172.21.15.154:0/3930552678 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 2) ==== 45093+0+0 (secure 0 0 0) 0x7f9f5001a070 con 0x7f9f5c152640 2024-03-18T16:20:53.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.202+0000 7f9f43fff640 1 -- 172.21.15.154:0/3930552678 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f9f50010df0 con 0x7f9f5c152640 2024-03-18T16:20:53.324 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.308+0000 7f9f43fff640 1 -- 172.21.15.154:0/3930552678 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9f50021020 con 0x7f9f5c152640 2024-03-18T16:20:53.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.383+0000 7f9f61bbc640 1 -- 172.21.15.154:0/3930552678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5e5e7bc5-c518-4f91-b6ce-3754524f5a56", "id": 5} v 0) -- 0x7f9f5c0c9910 con 0x7f9f5c152640 2024-03-18T16:20:53.806 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.802+0000 7f9f43fff640 1 -- 172.21.15.154:0/3930552678 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5e5e7bc5-c518-4f91-b6ce-3754524f5a56", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f9f50010300 con 0x7f9f5c152640 2024-03-18T16:20:53.896 INFO:teuthology.orchestra.run.smithi154.stdout:5 2024-03-18T16:20:53.898 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.803+0000 7f9f61bbc640 1 -- 172.21.15.154:0/3930552678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152640 msgr2=0x7f9f5c0c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:53.898 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.803+0000 7f9f61bbc640 1 --2- 172.21.15.154:0/3930552678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152640 0x7f9f5c0c6fe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9f5000c440 tx=0x7f9f5000c900 comp rx=0 tx=0).stop 2024-03-18T16:20:53.898 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.803+0000 7f9f61bbc640 1 -- 172.21.15.154:0/3930552678 shutdown_connections 2024-03-18T16:20:53.898 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.803+0000 7f9f61bbc640 1 --2- 172.21.15.154:0/3930552678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f5c15c460 0x7f9f5c14d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.898 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.803+0000 7f9f61bbc640 1 --2- 172.21.15.154:0/3930552678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f5c152640 0x7f9f5c0c6fe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.898 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.803+0000 7f9f61bbc640 1 --2- 172.21.15.154:0/3930552678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f5c152260 0x7f9f5c1e4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.898 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.803+0000 7f9f61bbc640 1 -- 172.21.15.154:0/3930552678 >> 172.21.15.154:0/3930552678 conn(0x7f9f5c06b8e0 msgr2=0x7f9f5c15ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:53.898 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.803+0000 7f9f61bbc640 1 -- 172.21.15.154:0/3930552678 shutdown_connections 2024-03-18T16:20:53.898 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.803+0000 7f9f61bbc640 1 -- 172.21.15.154:0/3930552678 wait complete. 2024-03-18T16:20:53.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new fe2b01af-9db4-4759-a97f-628cafc8aaf6 6 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.996+0000 7f4088b77640 1 Processor -- start 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.996+0000 7f4088b77640 1 -- start start 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.997+0000 7f4088b77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40841547b0 0x7f408415e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.997+0000 7f4088b77640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f408415a520 0x7f40840c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.997+0000 7f4088b77640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40840c4250 con 0x7f40840c3d40 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.997+0000 7f4088b77640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f40840ca5c0 con 0x7f408415a520 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.997+0000 7f4088b77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f408415e020 con 0x7f40841547b0 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.997+0000 7f408259c640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40841547b0 0x7f408415e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.997+0000 7f4082d9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f408415a520 0x7f40840c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.998+0000 7f4081d9b640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f40840c3d40 0x7f40840c9eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:50934/0 (socket says 172.21.15.154:50934) 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:53.998+0000 7f4081d9b640 1 -- 172.21.15.154:0/3278513191 learned_addr learned my addr 172.21.15.154:0/3278513191 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-03-18T16:20:54.002 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.000+0000 7f408259c640 1 -- 172.21.15.154:0/3278513191 >> v2:172.21.15.154:3300/0 conn(0x7f408415a520 msgr2=0x7f40840c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.000+0000 7f408259c640 1 --2- 172.21.15.154:0/3278513191 >> v2:172.21.15.154:3300/0 conn(0x7f408415a520 0x7f40840c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.000+0000 7f408259c640 1 -- 172.21.15.154:0/3278513191 >> v1:172.21.15.154:6789/0 conn(0x7f40840c3d40 legacy=0x7f40840c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.000+0000 7f408259c640 1 -- 172.21.15.154:0/3278513191 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40840c4530 con 0x7f40841547b0 2024-03-18T16:20:54.075 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.002+0000 7f408259c640 1 --2- 172.21.15.154:0/3278513191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40841547b0 0x7f408415e4b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4074009d90 tx=0x7f407402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abd6e318a255ec09 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:54.122 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f408159a640 1 -- 172.21.15.154:0/3278513191 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074039070 con 0x7f40841547b0 2024-03-18T16:20:54.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f408159a640 1 -- 172.21.15.154:0/3278513191 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f407402cce0 con 0x7f40841547b0 2024-03-18T16:20:54.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f408159a640 1 -- 172.21.15.154:0/3278513191 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074034660 con 0x7f40841547b0 2024-03-18T16:20:54.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f4088b77640 1 -- 172.21.15.154:0/3278513191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40841547b0 msgr2=0x7f408415e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:54.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f4088b77640 1 --2- 172.21.15.154:0/3278513191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40841547b0 0x7f408415e4b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4074009d90 tx=0x7f407402bf30 comp rx=0 tx=0).stop 2024-03-18T16:20:54.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f4088b77640 1 -- 172.21.15.154:0/3278513191 shutdown_connections 2024-03-18T16:20:54.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f4088b77640 1 --2- 172.21.15.154:0/3278513191 >> v2:172.21.15.154:3300/0 conn(0x7f408415a520 0x7f40840c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:54.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f4088b77640 1 --2- 172.21.15.154:0/3278513191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f40841547b0 0x7f408415e4b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:54.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f4088b77640 1 -- 172.21.15.154:0/3278513191 >> 172.21.15.154:0/3278513191 conn(0x7f4084071680 msgr2=0x7f4084071a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:54.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.006+0000 7f4088b77640 1 -- 172.21.15.154:0/3278513191 shutdown_connections 2024-03-18T16:20:54.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.007+0000 7f4088b77640 1 -- 172.21.15.154:0/3278513191 wait complete. 2024-03-18T16:20:54.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.008+0000 7f4088b77640 1 Processor -- start 2024-03-18T16:20:54.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.008+0000 7f4088b77640 1 -- start start 2024-03-18T16:20:54.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4088b77640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40840c3d40 0x7f40840cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:54.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4088b77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40841547b0 0x7f40840cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:54.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4088b77640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408415a520 0x7f4084156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4088b77640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f408415e9f0 con 0x7f40840c3d40 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4088b77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f408415dcb0 con 0x7f40841547b0 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4088b77640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4084154b90 con 0x7f408415a520 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4081d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40841547b0 0x7f40840cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4082d9d640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408415a520 0x7f4084156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4081d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40841547b0 0x7f40840cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:38758/0 (socket says 172.21.15.154:38758) 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.009+0000 7f4081d9b640 1 -- 172.21.15.154:0/3639046266 learned_addr learned my addr 172.21.15.154:0/3639046266 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.013+0000 7f4081d9b640 1 -- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408415a520 msgr2=0x7f4084156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.013+0000 7f4081d9b640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408415a520 0x7f4084156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.013+0000 7f4081d9b640 1 -- 172.21.15.154:0/3639046266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40840c3d40 msgr2=0x7f40840cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.013+0000 7f4081d9b640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40840c3d40 0x7f40840cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.013+0000 7f4081d9b640 1 -- 172.21.15.154:0/3639046266 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40841f3cb0 con 0x7f40841547b0 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.015+0000 7f4081d9b640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40841547b0 0x7f40840cc7a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f406c00c440 tx=0x7f406c00c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.018+0000 7f407affd640 1 -- 172.21.15.154:0/3639046266 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c017070 con 0x7f40841547b0 2024-03-18T16:20:54.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.018+0000 7f407affd640 1 -- 172.21.15.154:0/3639046266 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c005ce0 con 0x7f40841547b0 2024-03-18T16:20:54.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.018+0000 7f407affd640 1 -- 172.21.15.154:0/3639046266 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c0072f0 con 0x7f40841547b0 2024-03-18T16:20:54.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.019+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40841f3f90 con 0x7f40841547b0 2024-03-18T16:20:54.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.019+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40841f4550 con 0x7f40841547b0 2024-03-18T16:20:54.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.019+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4084074fd0 con 0x7f40841547b0 2024-03-18T16:20:54.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.030+0000 7f407affd640 1 -- 172.21.15.154:0/3639046266 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 2) ==== 45093+0+0 (secure 0 0 0) 0x7f406c01a070 con 0x7f40841547b0 2024-03-18T16:20:54.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.031+0000 7f407affd640 1 -- 172.21.15.154:0/3639046266 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f406c010950 con 0x7f40841547b0 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.130+0000 7f407affd640 1 -- 172.21.15.154:0/3639046266 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f406c021020 con 0x7f40841547b0 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:54.205+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fe2b01af-9db4-4759-a97f-628cafc8aaf6", "id": 6} v 0) -- 0x7f40840b4130 con 0x7f40841547b0 2024-03-18T16:20:55.261 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.259+0000 7f407affd640 1 -- 172.21.15.154:0/3639046266 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 3) ==== 89974+0+0 (secure 0 0 0) 0x7f406c01b8d0 con 0x7f40841547b0 2024-03-18T16:20:55.261 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.259+0000 7f407affd640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f406406bdd0 0x7f406406e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:55.261 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.259+0000 7f408259c640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f406406bdd0 0x7f406406e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:55.262 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.260+0000 7f408259c640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f406406bdd0 0x7f406406e1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4074036000 tx=0x7f407402e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:55.342 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:55.339+0000 7f735f911640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-18T16:20:55.401 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:20:55.399+0000 7f735f911640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-18T16:20:55.553 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.551+0000 7f407affd640 1 -- 172.21.15.154:0/3639046266 <== mon.1 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fe2b01af-9db4-4759-a97f-628cafc8aaf6", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f406c05b1d0 con 0x7f40841547b0 2024-03-18T16:20:55.553 INFO:teuthology.orchestra.run.smithi154.stdout:6 2024-03-18T16:20:55.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f406406bdd0 msgr2=0x7f406406e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:55.554 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f406406bdd0 0x7f406406e1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4074036000 tx=0x7f407402e000 comp rx=0 tx=0).stop 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40841547b0 msgr2=0x7f40840cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40841547b0 0x7f40840cc7a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f406c00c440 tx=0x7f406c00c900 comp rx=0 tx=0).stop 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 shutdown_connections 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f406406bdd0 0x7f406406e1c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f408415a520 0x7f4084156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40841547b0 0x7f40840cc7a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 --2- 172.21.15.154:0/3639046266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40840c3d40 0x7f40840cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 >> 172.21.15.154:0/3639046266 conn(0x7f4084071680 msgr2=0x7f4084159120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 shutdown_connections 2024-03-18T16:20:55.555 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.552+0000 7f4088b77640 1 -- 172.21.15.154:0/3639046266 wait complete. 2024-03-18T16:20:55.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new f9e0d662-1a25-4e47-a5c2-43ffb11478d8 7 2024-03-18T16:20:55.665 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.663+0000 7f33b04d3640 1 Processor -- start 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.663+0000 7f33b04d3640 1 -- start start 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.664+0000 7f33b04d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f33a81549a0 0x7f33a80c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.664+0000 7f33b04d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 0x7f33a815ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.664+0000 7f33b04d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33a80c80e0 con 0x7f33a806cf30 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.664+0000 7f33b04d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f33a80c8380 con 0x7f33a81549a0 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.664+0000 7f33b04d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f33a80c8660 con 0x7f33a80ca2c0 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.665+0000 7f33ae26f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f33a81549a0 0x7f33a80c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.665+0000 7f33ada6e640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f33a806cf30 0x7f33a80c96e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:50938/0 (socket says 172.21.15.154:50938) 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.665+0000 7f33ada6e640 1 -- 172.21.15.154:0/161187 learned_addr learned my addr 172.21.15.154:0/161187 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.667+0000 7f33aea70640 1 --2- 172.21.15.154:0/161187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 0x7f33a815ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.669+0000 7f33aea70640 1 -- 172.21.15.154:0/161187 >> v2:172.21.15.154:3300/0 conn(0x7f33a81549a0 msgr2=0x7f33a80c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.669+0000 7f33aea70640 1 --2- 172.21.15.154:0/161187 >> v2:172.21.15.154:3300/0 conn(0x7f33a81549a0 0x7f33a80c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.669+0000 7f33aea70640 1 -- 172.21.15.154:0/161187 >> v1:172.21.15.154:6789/0 conn(0x7f33a806cf30 legacy=0x7f33a80c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.669+0000 7f33aea70640 1 -- 172.21.15.154:0/161187 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a80c8940 con 0x7f33a80ca2c0 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.709+0000 7f33aea70640 1 --2- 172.21.15.154:0/161187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 0x7f33a815ddc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009d90 tx=0x7f33a002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=738d77d1cb36d77a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.713+0000 7f33ad26d640 1 -- 172.21.15.154:0/161187 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0039070 con 0x7f33a80ca2c0 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.713+0000 7f33ad26d640 1 -- 172.21.15.154:0/161187 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a002cc50 con 0x7f33a80ca2c0 2024-03-18T16:20:55.745 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.713+0000 7f33ad26d640 1 -- 172.21.15.154:0/161187 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0034440 con 0x7f33a80ca2c0 2024-03-18T16:20:55.745 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.713+0000 7f33b04d3640 1 -- 172.21.15.154:0/161187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 msgr2=0x7f33a815ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.714+0000 7f33b04d3640 1 --2- 172.21.15.154:0/161187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 0x7f33a815ddc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009d90 tx=0x7f33a002bea0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.714+0000 7f33b04d3640 1 -- 172.21.15.154:0/161187 shutdown_connections 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.714+0000 7f33b04d3640 1 --2- 172.21.15.154:0/161187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 0x7f33a815ddc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.714+0000 7f33b04d3640 1 --2- 172.21.15.154:0/161187 >> v2:172.21.15.154:3300/0 conn(0x7f33a81549a0 0x7f33a80c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.714+0000 7f33b04d3640 1 -- 172.21.15.154:0/161187 >> 172.21.15.154:0/161187 conn(0x7f33a806c310 msgr2=0x7f33a806c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.714+0000 7f33b04d3640 1 -- 172.21.15.154:0/161187 shutdown_connections 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.714+0000 7f33b04d3640 1 -- 172.21.15.154:0/161187 wait complete. 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.715+0000 7f33b04d3640 1 Processor -- start 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.715+0000 7f33b04d3640 1 -- start start 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33b04d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33a806cf30 0x7f33a81f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33b04d3640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 0x7f33a8153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33b04d3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33a81549a0 0x7f33a8146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33b04d3640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f33a80c9df0 con 0x7f33a81549a0 2024-03-18T16:20:55.746 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33b04d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f33a80c8f00 con 0x7f33a806cf30 2024-03-18T16:20:55.747 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33b04d3640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f33a80c9f70 con 0x7f33a80ca2c0 2024-03-18T16:20:55.747 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33ada6e640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 0x7f33a8153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:55.747 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33ae26f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33a806cf30 0x7f33a81f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:55.747 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33ae26f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33a806cf30 0x7f33a81f3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:38786/0 (socket says 172.21.15.154:38786) 2024-03-18T16:20:55.747 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.716+0000 7f33ae26f640 1 -- 172.21.15.154:0/475531736 learned_addr learned my addr 172.21.15.154:0/475531736 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-18T16:20:55.748 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.721+0000 7f33ae26f640 1 -- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 msgr2=0x7f33a8153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:55.748 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.721+0000 7f33ae26f640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 0x7f33a8153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.748 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.721+0000 7f33ae26f640 1 -- 172.21.15.154:0/475531736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33a81549a0 msgr2=0x7f33a8146550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.721+0000 7f33ae26f640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33a81549a0 0x7f33a8146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.721+0000 7f33ae26f640 1 -- 172.21.15.154:0/475531736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a815bc50 con 0x7f33a806cf30 2024-03-18T16:20:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.724+0000 7f33ae26f640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33a806cf30 0x7f33a81f3d80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f33a400e8d0 tx=0x7f33a400ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.727+0000 7f339effd640 1 -- 172.21.15.154:0/475531736 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4017070 con 0x7f33a806cf30 2024-03-18T16:20:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.727+0000 7f339effd640 1 -- 172.21.15.154:0/475531736 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a400c940 con 0x7f33a806cf30 2024-03-18T16:20:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.727+0000 7f339effd640 1 -- 172.21.15.154:0/475531736 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a400cd30 con 0x7f33a806cf30 2024-03-18T16:20:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.727+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a815d190 con 0x7f33a806cf30 2024-03-18T16:20:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.727+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a81f3580 con 0x7f33a806cf30 2024-03-18T16:20:55.749 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.729+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a8074fd0 con 0x7f33a806cf30 2024-03-18T16:20:55.975 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.973+0000 7f339effd640 1 -- 172.21.15.154:0/475531736 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 3) ==== 89974+0+0 (secure 0 0 0) 0x7f33a401a070 con 0x7f33a806cf30 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.974+0000 7f339effd640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f338c06ba70 0x7f338c06df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.974+0000 7f33ada6e640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f338c06ba70 0x7f338c06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.974+0000 7f33ada6e640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f338c06ba70 0x7f338c06df30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3398007c90 tx=0x7f3398007be0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.987+0000 7f339effd640 1 -- 172.21.15.154:0/475531736 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f33a4057650 con 0x7f33a806cf30 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:55.987+0000 7f339effd640 1 -- 172.21.15.154:0/475531736 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f33a40595b0 con 0x7f33a806cf30 2024-03-18T16:20:56.199 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.197+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f9e0d662-1a25-4e47-a5c2-43ffb11478d8", "id": 7} v 0) -- 0x7f33a814a380 con 0x7f33a806cf30 2024-03-18T16:20:56.611 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.609+0000 7f339effd640 1 -- 172.21.15.154:0/475531736 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7f33a4055c20 con 0x7f33a806cf30 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.750+0000 7f339effd640 1 -- 172.21.15.154:0/475531736 <== mon.1 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f9e0d662-1a25-4e47-a5c2-43ffb11478d8", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f33a4025e20 con 0x7f33a806cf30 2024-03-18T16:20:56.834 INFO:teuthology.orchestra.run.smithi154.stdout:7 2024-03-18T16:20:56.836 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f338c06ba70 msgr2=0x7f338c06df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:56.836 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f338c06ba70 0x7f338c06df30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3398007c90 tx=0x7f3398007be0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.837 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33a806cf30 msgr2=0x7f33a81f3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:56.837 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33a806cf30 0x7f33a81f3d80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f33a400e8d0 tx=0x7f33a400ed90 comp rx=0 tx=0).stop 2024-03-18T16:20:56.837 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 shutdown_connections 2024-03-18T16:20:56.837 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f338c06ba70 0x7f338c06df30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.837 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33a81549a0 0x7f33a8146550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.837 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33a80ca2c0 0x7f33a8153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.837 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 --2- 172.21.15.154:0/475531736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33a806cf30 0x7f33a81f3d80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.838 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 >> 172.21.15.154:0/475531736 conn(0x7f33a806c310 msgr2=0x7f33a8152990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:56.838 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 shutdown_connections 2024-03-18T16:20:56.838 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-18T16:20:56.753+0000 7f33b04d3640 1 -- 172.21.15.154:0/475531736 wait complete. 2024-03-18T16:20:56.838 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-18T16:20:56.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 0 2024-03-18T16:20:56.846 INFO:tasks.ceph.osd.0:Started 2024-03-18T16:20:56.847 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-18T16:20:56.847 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 1 2024-03-18T16:20:56.857 INFO:tasks.ceph.osd.1:Started 2024-03-18T16:20:56.865 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-18T16:20:56.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 2 2024-03-18T16:20:56.870 INFO:tasks.ceph.osd.2:Started 2024-03-18T16:20:56.871 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-18T16:20:56.871 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 3 2024-03-18T16:20:56.875 INFO:tasks.ceph.osd.3:Started 2024-03-18T16:20:56.875 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-18T16:20:56.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 4 2024-03-18T16:20:56.877 INFO:tasks.ceph.osd.4:Started 2024-03-18T16:20:56.877 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-18T16:20:56.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 5 2024-03-18T16:20:56.879 INFO:tasks.ceph.osd.5:Started 2024-03-18T16:20:56.879 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-18T16:20:56.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 6 2024-03-18T16:20:56.881 INFO:tasks.ceph.osd.6:Started 2024-03-18T16:20:56.881 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-18T16:20:56.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 7 2024-03-18T16:20:56.884 INFO:tasks.ceph.osd.7:Started 2024-03-18T16:20:56.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:20:57.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.127+0000 7f88f4c00640 1 Processor -- start 2024-03-18T16:20:57.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.127+0000 7f88f4c00640 1 -- start start 2024-03-18T16:20:57.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.128+0000 7f88f4c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88f00c9a20 0x7f88f00cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:57.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.128+0000 7f88f4c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f015aff0 0x7f88f00d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:57.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.128+0000 7f88f4c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f0154a90 con 0x7f88f006d450 2024-03-18T16:20:57.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.128+0000 7f88f4c00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f88f015e270 con 0x7f88f00c9a20 2024-03-18T16:20:57.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.128+0000 7f88f4c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f88f015e530 con 0x7f88f015aff0 2024-03-18T16:20:57.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.128+0000 7f88efe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f015aff0 0x7f88f00d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:57.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.128+0000 7f88ef400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88f00c9a20 0x7f88f00cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:57.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.129+0000 7f88efe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f015aff0 0x7f88f00d0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:49890/0 (socket says 172.21.15.102:49890) 2024-03-18T16:20:57.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.129+0000 7f88efe00640 1 -- 172.21.15.102:0/216049019 learned_addr learned my addr 172.21.15.102:0/216049019 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:20:57.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.129+0000 7f88e7e00640 1 --1- 172.21.15.102:0/216049019 >> v1:172.21.15.154:6789/0 conn(0x7f88f006d450 0x7f88f015a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:60774/0 (socket says 172.21.15.102:60774) 2024-03-18T16:20:57.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.131+0000 7f88efe00640 1 -- 172.21.15.102:0/216049019 >> v2:172.21.15.154:3300/0 conn(0x7f88f00c9a20 msgr2=0x7f88f00cfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:57.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.131+0000 7f88efe00640 1 --2- 172.21.15.102:0/216049019 >> v2:172.21.15.154:3300/0 conn(0x7f88f00c9a20 0x7f88f00cfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.131+0000 7f88efe00640 1 -- 172.21.15.102:0/216049019 >> v1:172.21.15.154:6789/0 conn(0x7f88f006d450 legacy=0x7f88f015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:57.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.131+0000 7f88efe00640 1 -- 172.21.15.102:0/216049019 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f00ca610 con 0x7f88f015aff0 2024-03-18T16:20:57.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.133+0000 7f88efe00640 1 --2- 172.21.15.102:0/216049019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f015aff0 0x7f88f00d0340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009e50 tx=0x7f88e802bfe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac02cf8ba9071e74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:57.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88eea00640 1 -- 172.21.15.102:0/216049019 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e802cb80 con 0x7f88f015aff0 2024-03-18T16:20:57.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88eea00640 1 -- 172.21.15.102:0/216049019 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e802cd20 con 0x7f88f015aff0 2024-03-18T16:20:57.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88eea00640 1 -- 172.21.15.102:0/216049019 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8034610 con 0x7f88f015aff0 2024-03-18T16:20:57.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88f4c00640 1 -- 172.21.15.102:0/216049019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f015aff0 msgr2=0x7f88f00d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:57.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88f4c00640 1 --2- 172.21.15.102:0/216049019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f015aff0 0x7f88f00d0340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f88e8009e50 tx=0x7f88e802bfe0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88f4c00640 1 -- 172.21.15.102:0/216049019 shutdown_connections 2024-03-18T16:20:57.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88f4c00640 1 --2- 172.21.15.102:0/216049019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f015aff0 0x7f88f00d0340 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88f4c00640 1 --2- 172.21.15.102:0/216049019 >> v2:172.21.15.154:3300/0 conn(0x7f88f00c9a20 0x7f88f00cfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88f4c00640 1 -- 172.21.15.102:0/216049019 >> 172.21.15.102:0/216049019 conn(0x7f88f0067160 msgr2=0x7f88f0067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:57.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.137+0000 7f88f4c00640 1 -- 172.21.15.102:0/216049019 shutdown_connections 2024-03-18T16:20:57.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.138+0000 7f88f4c00640 1 -- 172.21.15.102:0/216049019 wait complete. 2024-03-18T16:20:57.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.139+0000 7f88f4c00640 1 Processor -- start 2024-03-18T16:20:57.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.139+0000 7f88f4c00640 1 -- start start 2024-03-18T16:20:57.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.140+0000 7f88f4c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88f006d450 0x7f88f0153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.140+0000 7f88f4c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f00c9a20 0x7f88f01538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.140+0000 7f88f4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88f015aff0 0x7f88f01f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.140+0000 7f88f4c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f88f00cad50 con 0x7f88f015aff0 2024-03-18T16:20:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.140+0000 7f88f4c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f88f00cabd0 con 0x7f88f006d450 2024-03-18T16:20:57.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.140+0000 7f88f4c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f88f006e1d0 con 0x7f88f00c9a20 2024-03-18T16:20:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.141+0000 7f88efe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88f015aff0 0x7f88f01f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.141+0000 7f88ef400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88f006d450 0x7f88f0153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.141+0000 7f88e7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f00c9a20 0x7f88f01538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.141+0000 7f88ef400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88f006d450 0x7f88f0153390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:43316/0 (socket says 172.21.15.102:43316) 2024-03-18T16:20:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.141+0000 7f88ef400640 1 -- 172.21.15.102:0/858859347 learned_addr learned my addr 172.21.15.102:0/858859347 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:20:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.144+0000 7f88ef400640 1 -- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f00c9a20 msgr2=0x7f88f01538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.144+0000 7f88ef400640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f00c9a20 0x7f88f01538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.144+0000 7f88ef400640 1 -- 172.21.15.102:0/858859347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88f015aff0 msgr2=0x7f88f01f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:57.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.144+0000 7f88ef400640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88f015aff0 0x7f88f01f3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.144+0000 7f88ef400640 1 -- 172.21.15.102:0/858859347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f015d2c0 con 0x7f88f006d450 2024-03-18T16:20:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.147+0000 7f88ef400640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88f006d450 0x7f88f0153390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88e000e9b0 tx=0x7f88e000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.150+0000 7f88e7400640 1 -- 172.21.15.102:0/858859347 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e0017070 con 0x7f88f006d450 2024-03-18T16:20:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.150+0000 7f88e7400640 1 -- 172.21.15.102:0/858859347 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e000c9a0 con 0x7f88f006d450 2024-03-18T16:20:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.150+0000 7f88e7400640 1 -- 172.21.15.102:0/858859347 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e000cd90 con 0x7f88f006d450 2024-03-18T16:20:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.150+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f015d560 con 0x7f88f006d450 2024-03-18T16:20:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.151+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f0157a20 con 0x7f88f006d450 2024-03-18T16:20:57.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.152+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f006d270 con 0x7f88f006d450 2024-03-18T16:20:57.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.170+0000 7f88e7400640 1 -- 172.21.15.102:0/858859347 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7f88e001a070 con 0x7f88f006d450 2024-03-18T16:20:57.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.171+0000 7f88e7400640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88cc06bb90 0x7f88cc06e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:57.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.172+0000 7f88e7e00640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88cc06bb90 0x7f88cc06e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:57.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.172+0000 7f88e7e00640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88cc06bb90 0x7f88cc06e050 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f88f00675b0 tx=0x7f88d8000ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:57.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.172+0000 7f88e7400640 1 -- 172.21.15.102:0/858859347 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f88e00582b0 con 0x7f88f006d450 2024-03-18T16:20:57.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.414+0000 7f88e7400640 1 -- 172.21.15.102:0/858859347 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f88f006d270 con 0x7f88f006d450 2024-03-18T16:20:57.512 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-03-18T16:20:57.510+0000 7f6b7198c640 -1 Falling back to public interface 2024-03-18T16:20:57.537 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-03-18T16:20:57.522+0000 7f94c84f9640 -1 Falling back to public interface 2024-03-18T16:20:57.544 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-03-18T16:20:57.542+0000 7ff07008b640 -1 Falling back to public interface 2024-03-18T16:20:57.588 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-18T16:20:57.571+0000 7f76f3916640 -1 Falling back to public interface 2024-03-18T16:20:57.597 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-18T16:20:57.595+0000 7f3e7b924640 -1 Falling back to public interface 2024-03-18T16:20:57.706 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-18T16:20:57.595+0000 7f6f8e9bd640 -1 Falling back to public interface 2024-03-18T16:20:57.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.619+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88f0074fd0 con 0x7f88f006d450 2024-03-18T16:20:57.707 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-18T16:20:57.621+0000 7fb372e17640 -1 Falling back to public interface 2024-03-18T16:20:57.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.663+0000 7f88e7400640 1 -- 172.21.15.102:0/858859347 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f88e0059620 con 0x7f88f006d450 2024-03-18T16:20:57.708 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:20:57.708 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:20:56.257571+0000","last_up_change":"0.000000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88cc06bb90 msgr2=0x7f88cc06e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88cc06bb90 0x7f88cc06e050 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f88f00675b0 tx=0x7f88d8000ed0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88f006d450 msgr2=0x7f88f0153390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88f006d450 0x7f88f0153390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88e000e9b0 tx=0x7f88e000ee70 comp rx=0 tx=0).stop 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 shutdown_connections 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88cc06bb90 0x7f88cc06e050 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88f015aff0 0x7f88f01f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88f00c9a20 0x7f88f01538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 --2- 172.21.15.102:0/858859347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88f006d450 0x7f88f0153390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 >> 172.21.15.102:0/858859347 conn(0x7f88f0067160 msgr2=0x7f88f01599e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:57.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 shutdown_connections 2024-03-18T16:20:57.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:20:57.665+0000 7f88f4c00640 1 -- 172.21.15.102:0/858859347 wait complete. 2024-03-18T16:20:57.711 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-18T16:20:57.711 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-18T16:20:57.712 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-03-18T16:20:57.601+0000 7f03984bf640 -1 Falling back to public interface 2024-03-18T16:20:58.516 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-18T16:20:58.673 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.672+0000 7fc9edc00640 1 Processor -- start 2024-03-18T16:20:58.674 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.672+0000 7fc9edc00640 1 -- start start 2024-03-18T16:20:58.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.673+0000 7fc9edc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e81576c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:58.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.673+0000 7fc9edc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9e80c8a30 0x7fc9e8157c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:58.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.673+0000 7fc9edc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9e815a980 con 0x7fc9e80c7d70 2024-03-18T16:20:58.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.673+0000 7fc9edc00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc9e815ac60 con 0x7fc9e80c8a30 2024-03-18T16:20:58.675 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.673+0000 7fc9edc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e815af40 con 0x7fc9e80c7990 2024-03-18T16:20:58.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.674+0000 7fc9e7400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9e80c8a30 0x7fc9e8157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:58.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.674+0000 7fc9e6000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc9e80c7d70 0x7fc9e8151050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56466/0 (socket says 172.21.15.102:56466) 2024-03-18T16:20:58.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.674+0000 7fc9e6000640 1 -- 172.21.15.102:0/2047955363 learned_addr learned my addr 172.21.15.102:0/2047955363 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:20:58.676 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.674+0000 7fc9e6a00640 1 --2- 172.21.15.102:0/2047955363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e81576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:58.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.676+0000 7fc9e6a00640 1 -- 172.21.15.102:0/2047955363 >> v2:172.21.15.154:3300/0 conn(0x7fc9e80c8a30 msgr2=0x7fc9e8157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:58.678 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.676+0000 7fc9e6a00640 1 --2- 172.21.15.102:0/2047955363 >> v2:172.21.15.154:3300/0 conn(0x7fc9e80c8a30 0x7fc9e8157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:58.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.676+0000 7fc9e6a00640 1 -- 172.21.15.102:0/2047955363 >> v1:172.21.15.102:6789/0 conn(0x7fc9e80c7d70 legacy=0x7fc9e8151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:58.687 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.676+0000 7fc9e6a00640 1 -- 172.21.15.102:0/2047955363 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e815b220 con 0x7fc9e80c7990 2024-03-18T16:20:58.689 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.678+0000 7fc9e6a00640 1 --2- 172.21.15.102:0/2047955363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e81576c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009d90 tx=0x7fc9d802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e2e33d7bde39995 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.682+0000 7fc9e5600640 1 -- 172.21.15.102:0/2047955363 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8039070 con 0x7fc9e80c7990 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.682+0000 7fc9e5600640 1 -- 172.21.15.102:0/2047955363 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d802cce0 con 0x7fc9e80c7990 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.682+0000 7fc9e5600640 1 -- 172.21.15.102:0/2047955363 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8034660 con 0x7fc9e80c7990 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.683+0000 7fc9edc00640 1 -- 172.21.15.102:0/2047955363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 msgr2=0x7fc9e81576c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.683+0000 7fc9edc00640 1 --2- 172.21.15.102:0/2047955363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e81576c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009d90 tx=0x7fc9d802bf30 comp rx=0 tx=0).stop 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.683+0000 7fc9edc00640 1 -- 172.21.15.102:0/2047955363 shutdown_connections 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.683+0000 7fc9edc00640 1 --2- 172.21.15.102:0/2047955363 >> v2:172.21.15.154:3300/0 conn(0x7fc9e80c8a30 0x7fc9e8157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.683+0000 7fc9edc00640 1 --2- 172.21.15.102:0/2047955363 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e81576c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.683+0000 7fc9edc00640 1 -- 172.21.15.102:0/2047955363 >> 172.21.15.102:0/2047955363 conn(0x7fc9e806b1d0 msgr2=0x7fc9e806b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.683+0000 7fc9edc00640 1 -- 172.21.15.102:0/2047955363 shutdown_connections 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.683+0000 7fc9edc00640 1 -- 172.21.15.102:0/2047955363 wait complete. 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.684+0000 7fc9edc00640 1 Processor -- start 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.685+0000 7fc9edc00640 1 -- start start 2024-03-18T16:20:58.690 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.685+0000 7fc9edc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e8149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.685+0000 7fc9edc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80c7d70 0x7fc9e8159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.686+0000 7fc9edc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9e80c8a30 0x7fc9e8159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.686+0000 7fc9edc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e81518e0 con 0x7fc9e80c8a30 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.686+0000 7fc9edc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e8151760 con 0x7fc9e80c7d70 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.686+0000 7fc9edc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc9e8151a60 con 0x7fc9e80c7990 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.686+0000 7fc9e7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9e80c8a30 0x7fc9e8159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.686+0000 7fc9e6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e8149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.686+0000 7fc9e6000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80c7d70 0x7fc9e8159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.687+0000 7fc9e6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e8149930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:49910/0 (socket says 172.21.15.102:49910) 2024-03-18T16:20:58.691 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.687+0000 7fc9e6a00640 1 -- 172.21.15.102:0/1264873645 learned_addr learned my addr 172.21.15.102:0/1264873645 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:20:58.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.690+0000 7fc9e6a00640 1 -- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80c7d70 msgr2=0x7fc9e8159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:58.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.690+0000 7fc9e6a00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80c7d70 0x7fc9e8159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:58.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.690+0000 7fc9e6a00640 1 -- 172.21.15.102:0/1264873645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9e80c8a30 msgr2=0x7fc9e8159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:58.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.690+0000 7fc9e6a00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9e80c8a30 0x7fc9e8159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:58.692 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.690+0000 7fc9e6a00640 1 -- 172.21.15.102:0/1264873645 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e8149e70 con 0x7fc9e80c7990 2024-03-18T16:20:58.694 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.692+0000 7fc9e6a00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e8149930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009eb0 tx=0x7fc9d8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:58.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.696+0000 7fc9dea00640 1 -- 172.21.15.102:0/1264873645 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8044020 con 0x7fc9e80c7990 2024-03-18T16:20:58.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.696+0000 7fc9dea00640 1 -- 172.21.15.102:0/1264873645 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d803f470 con 0x7fc9e80c7990 2024-03-18T16:20:58.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.696+0000 7fc9dea00640 1 -- 172.21.15.102:0/1264873645 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8039040 con 0x7fc9e80c7990 2024-03-18T16:20:58.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.696+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e81e3260 con 0x7fc9e80c7990 2024-03-18T16:20:58.698 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.696+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e81521a0 con 0x7fc9e80c7990 2024-03-18T16:20:58.699 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.697+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e80b1800 con 0x7fc9e80c7990 2024-03-18T16:20:58.718 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.716+0000 7fc9dea00640 1 -- 172.21.15.102:0/1264873645 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7fc9d8047020 con 0x7fc9e80c7990 2024-03-18T16:20:58.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.717+0000 7fc9dea00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9c406bb90 0x7fc9c406e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:58.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.718+0000 7fc9dea00640 1 -- 172.21.15.102:0/1264873645 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fc9d8079ce0 con 0x7fc9e80c7990 2024-03-18T16:20:58.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.718+0000 7fc9e6000640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9c406bb90 0x7fc9c406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:58.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.718+0000 7fc9e6000640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9c406bb90 0x7fc9c406e050 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80ccbf0 tx=0x7fc9d0007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:58.956 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:58.954+0000 7fc9dea00640 1 -- 172.21.15.102:0/1264873645 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc9d80abe20 con 0x7fc9e80c7990 2024-03-18T16:20:59.160 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.158+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9e804f620 con 0x7fc9e80c7990 2024-03-18T16:20:59.207 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.205+0000 7fc9dea00640 1 -- 172.21.15.102:0/1264873645 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fc9e80b1800 con 0x7fc9e80c7990 2024-03-18T16:20:59.207 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-18T16:20:59.208 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:20:56.257571+0000","last_up_change":"0.000000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:20:59.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9c406bb90 msgr2=0x7fc9c406e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:59.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9c406bb90 0x7fc9c406e050 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80ccbf0 tx=0x7fc9d0007e80 comp rx=0 tx=0).stop 2024-03-18T16:20:59.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 msgr2=0x7fc9e8149930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:59.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e8149930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009eb0 tx=0x7fc9d8004a40 comp rx=0 tx=0).stop 2024-03-18T16:20:59.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 shutdown_connections 2024-03-18T16:20:59.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9c406bb90 0x7fc9c406e050 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:59.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9e80c8a30 0x7fc9e8159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:59.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9e80c7d70 0x7fc9e8159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:59.210 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 --2- 172.21.15.102:0/1264873645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9e80c7990 0x7fc9e8149930 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:59.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 >> 172.21.15.102:0/1264873645 conn(0x7fc9e806b1d0 msgr2=0x7fc9e8155100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:59.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 shutdown_connections 2024-03-18T16:20:59.211 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:20:59.208+0000 7fc9edc00640 1 -- 172.21.15.102:0/1264873645 wait complete. 2024-03-18T16:20:59.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-18T16:21:00.930 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-18T16:21:00.928+0000 7fb372e17640 -1 osd.2 0 log_to_monitors true 2024-03-18T16:21:00.956 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-03-18T16:21:00.953+0000 7f6b7198c640 -1 osd.5 0 log_to_monitors true 2024-03-18T16:21:01.014 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-03-18T16:21:00.995+0000 7f94c84f9640 -1 osd.6 0 log_to_monitors true 2024-03-18T16:21:01.033 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-03-18T16:21:01.031+0000 7ff07008b640 -1 osd.7 0 log_to_monitors true 2024-03-18T16:21:01.108 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-18T16:21:01.058+0000 7f76f3916640 -1 osd.1 0 log_to_monitors true 2024-03-18T16:21:01.109 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-03-18T16:21:01.064+0000 7f03984bf640 -1 osd.4 0 log_to_monitors true 2024-03-18T16:21:01.118 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-18T16:21:01.117+0000 7f3e7b924640 -1 osd.3 0 log_to_monitors true 2024-03-18T16:21:01.154 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-18T16:21:01.153+0000 7f6f8e9bd640 -1 osd.0 0 log_to_monitors true 2024-03-18T16:21:03.056 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-03-18T16:21:03.054+0000 7f6b6e136640 -1 osd.5 0 waiting for initial osdmap 2024-03-18T16:21:03.089 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-03-18T16:21:03.086+0000 7f6b65742640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:21:03.286 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-18T16:21:03.205+0000 7fb36d600640 -1 osd.2 0 waiting for initial osdmap 2024-03-18T16:21:03.286 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-03-18T16:21:03.095+0000 7f94c4ca3640 -1 osd.6 0 waiting for initial osdmap 2024-03-18T16:21:03.286 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-03-18T16:21:03.115+0000 7f94bc2af640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:21:03.313 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-18T16:21:03.311+0000 7fb363600640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:21:04.594 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-03-18T16:21:04.592+0000 7f0394c69640 -1 osd.4 0 waiting for initial osdmap 2024-03-18T16:21:04.652 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-03-18T16:21:04.650+0000 7f038c275640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:21:04.700 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-03-18T16:21:04.692+0000 7ff06c835640 -1 osd.7 0 waiting for initial osdmap 2024-03-18T16:21:04.713 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-03-18T16:21:04.711+0000 7ff063e41640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:21:05.162 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-18T16:21:05.161+0000 7f76ee800640 -1 osd.1 0 waiting for initial osdmap 2024-03-18T16:21:05.171 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-18T16:21:05.169+0000 7f76e4800640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:21:05.259 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-18T16:21:05.258+0000 7f3e77200640 -1 osd.3 0 waiting for initial osdmap 2024-03-18T16:21:05.276 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-18T16:21:05.267+0000 7f3e6c800640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:21:05.307 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-18T16:21:05.305+0000 7f6f89800640 -1 osd.0 0 waiting for initial osdmap 2024-03-18T16:21:05.365 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-18T16:21:05.315+0000 7f6f7f800640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:21:06.068 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:06.248 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.246+0000 7fcb1be00640 1 Processor -- start 2024-03-18T16:21:06.248 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.246+0000 7fcb1be00640 1 -- start start 2024-03-18T16:21:06.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.247+0000 7fcb1be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0c9410 0x7fcb1c0cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:06.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.247+0000 7fcb1be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcb1c15ad10 0x7fcb1c164ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:06.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.247+0000 7fcb1be00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb1c06c5f0 con 0x7fcb1c0d0710 2024-03-18T16:21:06.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.247+0000 7fcb1be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fcb1c154dc0 con 0x7fcb1c15ad10 2024-03-18T16:21:06.249 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.247+0000 7fcb1be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcb1c0cadd0 con 0x7fcb1c0c9410 2024-03-18T16:21:06.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.248+0000 7fcb1a000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fcb1c0d0710 0x7fcb1c15a600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:48452/0 (socket says 172.21.15.102:48452) 2024-03-18T16:21:06.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.248+0000 7fcb1a000640 1 -- 172.21.15.102:0/1759706394 learned_addr learned my addr 172.21.15.102:0/1759706394 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:06.250 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.248+0000 7fcb1aa00640 1 --2- 172.21.15.102:0/1759706394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0c9410 0x7fcb1c0cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:06.254 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.251+0000 7fcb1aa00640 1 -- 172.21.15.102:0/1759706394 >> v2:172.21.15.102:3300/0 conn(0x7fcb1c15ad10 msgr2=0x7fcb1c164ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:06.254 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.251+0000 7fcb1aa00640 1 --2- 172.21.15.102:0/1759706394 >> v2:172.21.15.102:3300/0 conn(0x7fcb1c15ad10 0x7fcb1c164ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:06.254 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.251+0000 7fcb1aa00640 1 -- 172.21.15.102:0/1759706394 >> v1:172.21.15.154:6789/0 conn(0x7fcb1c0d0710 legacy=0x7fcb1c15a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:06.254 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.251+0000 7fcb1aa00640 1 -- 172.21.15.102:0/1759706394 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c06c870 con 0x7fcb1c0c9410 2024-03-18T16:21:06.255 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.253+0000 7fcb1aa00640 1 --2- 172.21.15.102:0/1759706394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0c9410 0x7fcb1c0cbb00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0097a0 tx=0x7fcb0c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=912d1b058be0008c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:06.261 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.259+0000 7fcb19600640 1 -- 172.21.15.102:0/1759706394 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c035020 con 0x7fcb1c0c9410 2024-03-18T16:21:06.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.427+0000 7fcb19600640 1 -- 172.21.15.102:0/1759706394 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0c031830 con 0x7fcb1c0c9410 2024-03-18T16:21:06.429 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.427+0000 7fcb19600640 1 -- 172.21.15.102:0/1759706394 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c031c00 con 0x7fcb1c0c9410 2024-03-18T16:21:06.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.428+0000 7fcb1be00640 1 -- 172.21.15.102:0/1759706394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0c9410 msgr2=0x7fcb1c0cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:06.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.428+0000 7fcb1be00640 1 --2- 172.21.15.102:0/1759706394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0c9410 0x7fcb1c0cbb00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c0097a0 tx=0x7fcb0c02b8d0 comp rx=0 tx=0).stop 2024-03-18T16:21:06.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.429+0000 7fcb1be00640 1 -- 172.21.15.102:0/1759706394 shutdown_connections 2024-03-18T16:21:06.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.429+0000 7fcb1be00640 1 --2- 172.21.15.102:0/1759706394 >> v2:172.21.15.102:3300/0 conn(0x7fcb1c15ad10 0x7fcb1c164ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:06.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.429+0000 7fcb1be00640 1 --2- 172.21.15.102:0/1759706394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0c9410 0x7fcb1c0cbb00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:06.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.429+0000 7fcb1be00640 1 -- 172.21.15.102:0/1759706394 >> 172.21.15.102:0/1759706394 conn(0x7fcb1c0cfbf0 msgr2=0x7fcb1c0d0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:06.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.429+0000 7fcb1be00640 1 -- 172.21.15.102:0/1759706394 shutdown_connections 2024-03-18T16:21:06.431 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.429+0000 7fcb1be00640 1 -- 172.21.15.102:0/1759706394 wait complete. 2024-03-18T16:21:06.434 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.433+0000 7fcb1be00640 1 Processor -- start 2024-03-18T16:21:06.435 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.433+0000 7fcb1be00640 1 -- start start 2024-03-18T16:21:06.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.436+0000 7fcb1be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb1c0c9410 0x7fcb1c0abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:06.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.436+0000 7fcb1be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0d0710 0x7fcb1c155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:06.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.436+0000 7fcb1be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb1c15ad10 0x7fcb1c0cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:06.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.436+0000 7fcb1be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c0cb300 con 0x7fcb1c15ad10 2024-03-18T16:21:06.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.436+0000 7fcb1be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c0cb020 con 0x7fcb1c0c9410 2024-03-18T16:21:06.438 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.436+0000 7fcb1be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcb1c0cb5e0 con 0x7fcb1c0d0710 2024-03-18T16:21:06.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.436+0000 7fcb1a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0d0710 0x7fcb1c155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:06.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.437+0000 7fcb1a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0d0710 0x7fcb1c155280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50012/0 (socket says 172.21.15.102:50012) 2024-03-18T16:21:06.439 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.437+0000 7fcb1a000640 1 -- 172.21.15.102:0/587118066 learned_addr learned my addr 172.21.15.102:0/587118066 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:06.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.440+0000 7fcb1a000640 1 -- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb1c0c9410 msgr2=0x7fcb1c0abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:06.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.440+0000 7fcb1a000640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb1c0c9410 0x7fcb1c0abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:06.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.440+0000 7fcb1a000640 1 -- 172.21.15.102:0/587118066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb1c15ad10 msgr2=0x7fcb1c0cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:06.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.440+0000 7fcb1a000640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb1c15ad10 0x7fcb1c0cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:06.442 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.440+0000 7fcb1a000640 1 -- 172.21.15.102:0/587118066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c1f33d0 con 0x7fcb1c0d0710 2024-03-18T16:21:06.444 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.442+0000 7fcb1a000640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0d0710 0x7fcb1c155280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800c430 tx=0x7fcb0800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:06.448 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.446+0000 7fcb12a00640 1 -- 172.21.15.102:0/587118066 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb08017070 con 0x7fcb1c0d0710 2024-03-18T16:21:06.523 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.446+0000 7fcb12a00640 1 -- 172.21.15.102:0/587118066 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb08005ce0 con 0x7fcb1c0d0710 2024-03-18T16:21:06.523 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.446+0000 7fcb12a00640 1 -- 172.21.15.102:0/587118066 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb080072f0 con 0x7fcb1c0d0710 2024-03-18T16:21:06.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.447+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1c1f36b0 con 0x7fcb1c0d0710 2024-03-18T16:21:06.524 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.447+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb1c1f3c70 con 0x7fcb1c0d0710 2024-03-18T16:21:06.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.449+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb1c04f540 con 0x7fcb1c0d0710 2024-03-18T16:21:06.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.482+0000 7fcb12a00640 1 -- 172.21.15.102:0/587118066 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7fcb0801a070 con 0x7fcb1c0d0710 2024-03-18T16:21:06.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.483+0000 7fcb12a00640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcafc06bb40 0x7fcafc06e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:06.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.485+0000 7fcb12a00640 1 -- 172.21.15.102:0/587118066 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4846+0+0 (secure 0 0 0) 0x7fcb0805ea70 con 0x7fcb1c0d0710 2024-03-18T16:21:06.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.486+0000 7fcb1aa00640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcafc06bb40 0x7fcafc06e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:06.525 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.487+0000 7fcb1aa00640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcafc06bb40 0x7fcafc06e000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c036000 tx=0x7fcb0c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:06.733 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.731+0000 7fcb12a00640 1 -- 172.21.15.102:0/587118066 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcb08021020 con 0x7fcb1c0d0710 2024-03-18T16:21:06.957 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:06.955+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb1c0b5b70 con 0x7fcb1c0d0710 2024-03-18T16:21:07.049 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.047+0000 7fcb12a00640 1 -- 172.21.15.102:0/587118066 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12833 (secure 0 0 0) 0x7fcb1c06c5f0 con 0x7fcb1c0d0710 2024-03-18T16:21:07.050 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-18T16:21:07.050 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":15,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:05.847994+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:07.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcafc06bb40 msgr2=0x7fcafc06e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:07.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcafc06bb40 0x7fcafc06e000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c036000 tx=0x7fcb0c030000 comp rx=0 tx=0).stop 2024-03-18T16:21:07.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0d0710 msgr2=0x7fcb1c155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:07.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0d0710 0x7fcb1c155280 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcb0800c430 tx=0x7fcb0800c8f0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.052 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 shutdown_connections 2024-03-18T16:21:07.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcafc06bb40 0x7fcafc06e000 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb1c15ad10 0x7fcb1c0cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcb1c0d0710 0x7fcb1c155280 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 --2- 172.21.15.102:0/587118066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb1c0c9410 0x7fcb1c0abc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 >> 172.21.15.102:0/587118066 conn(0x7fcb1c0cfbf0 msgr2=0x7fcb1c0b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:07.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 shutdown_connections 2024-03-18T16:21:07.053 INFO:teuthology.misc.health.smithi102.stderr:2024-03-18T16:21:07.050+0000 7fcb1be00640 1 -- 172.21.15.102:0/587118066 wait complete. 2024-03-18T16:21:07.067 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-18T16:21:07.068 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-18T16:21:07.068 DEBUG:tasks.ceph_manager:running mds.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-18T16:21:07.068 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-18T16:21:07.069 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a 2024-03-18T16:21:07.076 INFO:tasks.ceph.mds.a:Started 2024-03-18T16:21:07.076 DEBUG:tasks.ceph_manager:running mds.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-18T16:21:07.076 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-18T16:21:07.076 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c 2024-03-18T16:21:07.083 INFO:tasks.ceph.mds.c:Started 2024-03-18T16:21:07.083 DEBUG:tasks.ceph_manager:running mds.e under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.e.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-18T16:21:07.083 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-18T16:21:07.083 DEBUG:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.e.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i e 2024-03-18T16:21:07.088 INFO:tasks.ceph.mds.e:Started 2024-03-18T16:21:07.088 DEBUG:tasks.ceph_manager:running mds.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-18T16:21:07.088 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-18T16:21:07.088 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i b 2024-03-18T16:21:07.090 INFO:tasks.ceph.mds.b:Started 2024-03-18T16:21:07.090 DEBUG:tasks.ceph_manager:running mds.d under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.d.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-18T16:21:07.091 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-18T16:21:07.091 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.d.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i d 2024-03-18T16:21:07.093 INFO:tasks.ceph.mds.d:Started 2024-03-18T16:21:07.093 DEBUG:tasks.ceph_manager:running mds.f under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.f.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-18T16:21:07.093 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-18T16:21:07.093 DEBUG:teuthology.orchestra.run.smithi154:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.f.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i f 2024-03-18T16:21:07.095 INFO:tasks.ceph.mds.f:Started 2024-03-18T16:21:07.095 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-18T16:21:07.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:07.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.279+0000 7fde4ce00640 1 Processor -- start 2024-03-18T16:21:07.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.279+0000 7fde4ce00640 1 -- start start 2024-03-18T16:21:07.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.280+0000 7fde4ce00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde480cdbd0 0x7fde480c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:07.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.280+0000 7fde4ce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde4815d8c0 0x7fde480c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:07.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.280+0000 7fde4ce00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde480ce0a0 con 0x7fde48154730 2024-03-18T16:21:07.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.280+0000 7fde4ce00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fde4815a830 con 0x7fde480cdbd0 2024-03-18T16:21:07.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.280+0000 7fde4ce00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde480ce380 con 0x7fde4815d8c0 2024-03-18T16:21:07.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.281+0000 7fde46800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde4815d8c0 0x7fde480c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:07.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.281+0000 7fde45400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde480cdbd0 0x7fde480c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:07.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.281+0000 7fde46800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde4815d8c0 0x7fde480c9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50020/0 (socket says 172.21.15.102:50020) 2024-03-18T16:21:07.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.282+0000 7fde46800640 1 -- 172.21.15.102:0/2458104458 learned_addr learned my addr 172.21.15.102:0/2458104458 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:07.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.284+0000 7fde46800640 1 -- 172.21.15.102:0/2458104458 >> v2:172.21.15.154:3300/0 conn(0x7fde480cdbd0 msgr2=0x7fde480c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:07.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.284+0000 7fde46800640 1 --2- 172.21.15.102:0/2458104458 >> v2:172.21.15.154:3300/0 conn(0x7fde480cdbd0 0x7fde480c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.284+0000 7fde46800640 1 -- 172.21.15.102:0/2458104458 >> v1:172.21.15.154:6789/0 conn(0x7fde48154730 legacy=0x7fde4815acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:07.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.284+0000 7fde46800640 1 -- 172.21.15.102:0/2458104458 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde480ce660 con 0x7fde4815d8c0 2024-03-18T16:21:07.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.286+0000 7fde46800640 1 --2- 172.21.15.102:0/2458104458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde4815d8c0 0x7fde480c9bf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fde40009e50 tx=0x7fde4002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30d98d5172344377 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:07.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.291+0000 7fde44a00640 1 -- 172.21.15.102:0/2458104458 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4002ca70 con 0x7fde4815d8c0 2024-03-18T16:21:07.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.291+0000 7fde44a00640 1 -- 172.21.15.102:0/2458104458 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde4002cc10 con 0x7fde4815d8c0 2024-03-18T16:21:07.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.291+0000 7fde44a00640 1 -- 172.21.15.102:0/2458104458 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde400343f0 con 0x7fde4815d8c0 2024-03-18T16:21:07.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.291+0000 7fde4ce00640 1 -- 172.21.15.102:0/2458104458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde4815d8c0 msgr2=0x7fde480c9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:07.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.291+0000 7fde4ce00640 1 --2- 172.21.15.102:0/2458104458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde4815d8c0 0x7fde480c9bf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fde40009e50 tx=0x7fde4002bed0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.292+0000 7fde4ce00640 1 -- 172.21.15.102:0/2458104458 shutdown_connections 2024-03-18T16:21:07.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.292+0000 7fde4ce00640 1 --2- 172.21.15.102:0/2458104458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde4815d8c0 0x7fde480c9bf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.292+0000 7fde4ce00640 1 --2- 172.21.15.102:0/2458104458 >> v2:172.21.15.154:3300/0 conn(0x7fde480cdbd0 0x7fde480c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.292+0000 7fde4ce00640 1 -- 172.21.15.102:0/2458104458 >> 172.21.15.102:0/2458104458 conn(0x7fde48066f80 msgr2=0x7fde48067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:07.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.292+0000 7fde4ce00640 1 -- 172.21.15.102:0/2458104458 shutdown_connections 2024-03-18T16:21:07.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.292+0000 7fde4ce00640 1 -- 172.21.15.102:0/2458104458 wait complete. 2024-03-18T16:21:07.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.294+0000 7fde4ce00640 1 Processor -- start 2024-03-18T16:21:07.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.295+0000 7fde4ce00640 1 -- start start 2024-03-18T16:21:07.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.296+0000 7fde4ce00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde480cdbd0 0x7fde481f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:07.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.296+0000 7fde4ce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde48154730 0x7fde48152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:07.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.296+0000 7fde4ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde4815d8c0 0x7fde48150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:07.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.296+0000 7fde4ce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde480c8a80 con 0x7fde4815d8c0 2024-03-18T16:21:07.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.296+0000 7fde4ce00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde480c8900 con 0x7fde480cdbd0 2024-03-18T16:21:07.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.296+0000 7fde4ce00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde480c8c00 con 0x7fde48154730 2024-03-18T16:21:07.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.297+0000 7fde45400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde48154730 0x7fde48152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:07.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.297+0000 7fde45e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde480cdbd0 0x7fde481f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:07.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.297+0000 7fde45e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde480cdbd0 0x7fde481f0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56510/0 (socket says 172.21.15.102:56510) 2024-03-18T16:21:07.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.297+0000 7fde45e00640 1 -- 172.21.15.102:0/4081198662 learned_addr learned my addr 172.21.15.102:0/4081198662 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:07.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.297+0000 7fde46800640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde4815d8c0 0x7fde48150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:07.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.302+0000 7fde45400640 1 -- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde480cdbd0 msgr2=0x7fde481f0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:07.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.302+0000 7fde45400640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde480cdbd0 0x7fde481f0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.302+0000 7fde45400640 1 -- 172.21.15.102:0/4081198662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde4815d8c0 msgr2=0x7fde48150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:07.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.302+0000 7fde45400640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde4815d8c0 0x7fde48150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.302+0000 7fde45400640 1 -- 172.21.15.102:0/4081198662 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde48156b60 con 0x7fde48154730 2024-03-18T16:21:07.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.302+0000 7fde45e00640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde480cdbd0 0x7fde481f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:07.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.306+0000 7fde45400640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde48154730 0x7fde48152330 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fde3400e820 tx=0x7fde3400ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:07.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.309+0000 7fde3e000640 1 -- 172.21.15.102:0/4081198662 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34017070 con 0x7fde48154730 2024-03-18T16:21:07.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.309+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde48156e40 con 0x7fde48154730 2024-03-18T16:21:07.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.309+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde48157400 con 0x7fde48154730 2024-03-18T16:21:07.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.310+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde480c8fc0 con 0x7fde48154730 2024-03-18T16:21:07.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.514+0000 7fde3e000640 1 -- 172.21.15.102:0/4081198662 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde34005990 con 0x7fde48154730 2024-03-18T16:21:07.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.586+0000 7fde3e000640 1 -- 172.21.15.102:0/4081198662 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde34021060 con 0x7fde48154730 2024-03-18T16:21:07.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.588+0000 7fde3e000640 1 -- 172.21.15.102:0/4081198662 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7fde3401a070 con 0x7fde48154730 2024-03-18T16:21:07.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.591+0000 7fde3e000640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde2806bb40 0x7fde2806e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:07.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.591+0000 7fde3e000640 1 -- 172.21.15.102:0/4081198662 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4846+0+0 (secure 0 0 0) 0x7fde3408ac60 con 0x7fde48154730 2024-03-18T16:21:07.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.591+0000 7fde45e00640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde2806bb40 0x7fde2806e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:07.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.592+0000 7fde45e00640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde2806bb40 0x7fde2806e000 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fde38001c30 tx=0x7fde38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:07.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.602+0000 7fde3e000640 1 -- 172.21.15.102:0/4081198662 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fde480c8fc0 con 0x7fde48154730 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.802+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde48074fd0 con 0x7fde48154730 2024-03-18T16:21:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.836+0000 7fde3e000640 1 -- 172.21.15.102:0/4081198662 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12833 (secure 0 0 0) 0x7fde34059540 con 0x7fde48154730 2024-03-18T16:21:07.880 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:07.880 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":15,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:05.847994+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.839+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde2806bb40 msgr2=0x7fde2806e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.839+0000 7fde4ce00640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde2806bb40 0x7fde2806e000 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fde38001c30 tx=0x7fde38009040 comp rx=0 tx=0).stop 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde48154730 msgr2=0x7fde48152330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde48154730 0x7fde48152330 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fde3400e820 tx=0x7fde3400ece0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 shutdown_connections 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde2806bb40 0x7fde2806e000 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde4815d8c0 0x7fde48150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde48154730 0x7fde48152330 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 --2- 172.21.15.102:0/4081198662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde480cdbd0 0x7fde481f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 >> 172.21.15.102:0/4081198662 conn(0x7fde48066f80 msgr2=0x7fde48067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 shutdown_connections 2024-03-18T16:21:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:07.840+0000 7fde4ce00640 1 -- 172.21.15.102:0/4081198662 wait complete. 2024-03-18T16:21:07.883 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:21:05.204661+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T16:21:07.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:21:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.097+0000 7fd53f200640 1 Processor -- start 2024-03-18T16:21:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.097+0000 7fd53f200640 1 -- start start 2024-03-18T16:21:08.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.099+0000 7fd53f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd538154990 0x7fd5380cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:08.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.099+0000 7fd53f200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd5380c51a0 0x7fd53815db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:08.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.099+0000 7fd53f200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd538154d70 con 0x7fd53815e0c0 2024-03-18T16:21:08.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.099+0000 7fd53f200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd5380c4ff0 con 0x7fd5380c51a0 2024-03-18T16:21:08.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.099+0000 7fd53f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5380ccc20 con 0x7fd538154990 2024-03-18T16:21:08.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.120+0000 7fd537400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd5380c51a0 0x7fd53815db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.121+0000 7fd53d200640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd53815e0c0 0x7fd5380d0320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:48464/0 (socket says 172.21.15.102:48464) 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.121+0000 7fd53d200640 1 -- 172.21.15.102:0/2493350629 learned_addr learned my addr 172.21.15.102:0/2493350629 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.127+0000 7fd537400640 1 -- 172.21.15.102:0/2493350629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd538154990 msgr2=0x7fd5380cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.127+0000 7fd537400640 1 --2- 172.21.15.102:0/2493350629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd538154990 0x7fd5380cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.127+0000 7fd537400640 1 -- 172.21.15.102:0/2493350629 >> v1:172.21.15.154:6789/0 conn(0x7fd53815e0c0 legacy=0x7fd5380d0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.127+0000 7fd537400640 1 -- 172.21.15.102:0/2493350629 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53815e5c0 con 0x7fd5380c51a0 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.132+0000 7fd537400640 1 --2- 172.21.15.102:0/2493350629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 0x7fd53815db80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd5240097a0 tx=0x7fd52402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3010995663444b1b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.136+0000 7fd536a00640 1 -- 172.21.15.102:0/2493350629 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524034070 con 0x7fd5380c51a0 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.146+0000 7fd536a00640 1 -- 172.21.15.102:0/2493350629 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd524030800 con 0x7fd5380c51a0 2024-03-18T16:21:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.146+0000 7fd536a00640 1 -- 172.21.15.102:0/2493350629 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524030bd0 con 0x7fd5380c51a0 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.147+0000 7fd53f200640 1 -- 172.21.15.102:0/2493350629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 msgr2=0x7fd53815db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.147+0000 7fd53f200640 1 --2- 172.21.15.102:0/2493350629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 0x7fd53815db80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd5240097a0 tx=0x7fd52402b940 comp rx=0 tx=0).stop 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.147+0000 7fd53f200640 1 -- 172.21.15.102:0/2493350629 shutdown_connections 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.147+0000 7fd53f200640 1 --2- 172.21.15.102:0/2493350629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 0x7fd53815db80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.147+0000 7fd53f200640 1 --2- 172.21.15.102:0/2493350629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd538154990 0x7fd5380cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.147+0000 7fd53f200640 1 -- 172.21.15.102:0/2493350629 >> 172.21.15.102:0/2493350629 conn(0x7fd53806cb00 msgr2=0x7fd53806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.147+0000 7fd53f200640 1 -- 172.21.15.102:0/2493350629 shutdown_connections 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.147+0000 7fd53f200640 1 -- 172.21.15.102:0/2493350629 wait complete. 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.149+0000 7fd53f200640 1 Processor -- start 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.149+0000 7fd53f200640 1 -- start start 2024-03-18T16:21:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.150+0000 7fd53f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 0x7fd5381f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.150+0000 7fd53f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd538154990 0x7fd53815d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.150+0000 7fd53f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53815e0c0 0x7fd538159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.150+0000 7fd53f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5380cc8b0 con 0x7fd538154990 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.150+0000 7fd53f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5380cc5d0 con 0x7fd5380c51a0 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.150+0000 7fd53f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5380cd5f0 con 0x7fd53815e0c0 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.151+0000 7fd537e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 0x7fd5381f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.151+0000 7fd537400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd538154990 0x7fd53815d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.151+0000 7fd537e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 0x7fd5381f0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56526/0 (socket says 172.21.15.102:56526) 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.151+0000 7fd537e00640 1 -- 172.21.15.102:0/959259441 learned_addr learned my addr 172.21.15.102:0/959259441 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.152+0000 7fd53d200640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53815e0c0 0x7fd538159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.155+0000 7fd537400640 1 -- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53815e0c0 msgr2=0x7fd538159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.155+0000 7fd537400640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53815e0c0 0x7fd538159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.155+0000 7fd537400640 1 -- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 msgr2=0x7fd5381f0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.155+0000 7fd537400640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 0x7fd5381f0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.155+0000 7fd537400640 1 -- 172.21.15.102:0/959259441 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd538158690 con 0x7fd538154990 2024-03-18T16:21:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.158+0000 7fd537400640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd538154990 0x7fd53815d1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd524033000 tx=0x7fd5240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.164+0000 7fd52fe00640 1 -- 172.21.15.102:0/959259441 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524034040 con 0x7fd538154990 2024-03-18T16:21:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.164+0000 7fd52fe00640 1 -- 172.21.15.102:0/959259441 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd524030600 con 0x7fd538154990 2024-03-18T16:21:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.164+0000 7fd52fe00640 1 -- 172.21.15.102:0/959259441 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5240423f0 con 0x7fd538154990 2024-03-18T16:21:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.165+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd538158970 con 0x7fd538154990 2024-03-18T16:21:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.165+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd538158f30 con 0x7fd538154990 2024-03-18T16:21:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.166+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd538074fd0 con 0x7fd538154990 2024-03-18T16:21:08.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.192+0000 7fd52fe00640 1 -- 172.21.15.102:0/959259441 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7fd524037070 con 0x7fd538154990 2024-03-18T16:21:08.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.193+0000 7fd52fe00640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd51806bbc0 0x7fd51806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:08.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.193+0000 7fd537e00640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd51806bbc0 0x7fd51806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:08.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.194+0000 7fd537e00640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd51806bbc0 0x7fd51806e080 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd528009a40 tx=0x7fd5280091c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:08.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.498+0000 7fd52fe00640 1 -- 172.21.15.102:0/959259441 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd5240307b0 con 0x7fd538154990 2024-03-18T16:21:08.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.498+0000 7fd52fe00640 1 -- 172.21.15.102:0/959259441 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd52407ba90 con 0x7fd538154990 2024-03-18T16:21:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.724+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd5380ced40 con 0x7fd538154990 2024-03-18T16:21:08.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.762+0000 7fd52fe00640 1 -- 172.21.15.102:0/959259441 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fd52407fb30 con 0x7fd538154990 2024-03-18T16:21:08.857 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-18T16:21:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd51806bbc0 msgr2=0x7fd51806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:08.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd51806bbc0 0x7fd51806e080 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd528009a40 tx=0x7fd5280091c0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd538154990 msgr2=0x7fd53815d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:08.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd538154990 0x7fd53815d1c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd524033000 tx=0x7fd5240046c0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 shutdown_connections 2024-03-18T16:21:08.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd51806bbc0 0x7fd51806e080 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd53815e0c0 0x7fd538159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd538154990 0x7fd53815d1c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 --2- 172.21.15.102:0/959259441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5380c51a0 0x7fd5381f0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:08.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.765+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 >> 172.21.15.102:0/959259441 conn(0x7fd53806cb00 msgr2=0x7fd5380cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:08.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.766+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 shutdown_connections 2024-03-18T16:21:08.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:08.766+0000 7fd53f200640 1 -- 172.21.15.102:0/959259441 wait complete. 2024-03-18T16:21:08.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:21:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.257+0000 7f81f1000640 1 Processor -- start 2024-03-18T16:21:09.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.258+0000 7f81f1000640 1 -- start start 2024-03-18T16:21:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.260+0000 7f81f1000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f81ec0c8740 0x7f81ec072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:09.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.260+0000 7f81f1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec158760 0x7f81ec15b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.260+0000 7f81f1000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81ec157ac0 con 0x7f81ec0c8b20 2024-03-18T16:21:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.260+0000 7f81f1000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f81ec15a860 con 0x7f81ec0c8740 2024-03-18T16:21:09.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.260+0000 7f81f1000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f81ec072840 con 0x7f81ec158760 2024-03-18T16:21:09.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.299+0000 7f81e9400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f81ec0c8b20 0x7f81ec1572f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:48480/0 (socket says 172.21.15.102:48480) 2024-03-18T16:21:09.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.299+0000 7f81e9400640 1 -- 172.21.15.102:0/2505720237 learned_addr learned my addr 172.21.15.102:0/2505720237 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:09.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.313+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2365537134 0 0) 0x7f81ec157ac0 con 0x7f81ec0c8b20 2024-03-18T16:21:09.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.313+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81d0003520 con 0x7f81ec0c8b20 2024-03-18T16:21:09.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2042860749 0 0) 0x7f81d0003520 con 0x7f81ec0c8b20 2024-03-18T16:21:09.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec158760 msgr2=0x7f81ec15b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:09.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 --2- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec158760 0x7f81ec15b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:09.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 >> v2:172.21.15.102:3300/0 conn(0x7f81ec0c8740 msgr2=0x7f81ec072300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:09.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 --2- 172.21.15.102:0/2505720237 >> v2:172.21.15.102:3300/0 conn(0x7f81ec0c8740 0x7f81ec072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:09.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec157da0 con 0x7f81ec0c8b20 2024-03-18T16:21:09.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f81d8001670 con 0x7f81ec0c8b20 2024-03-18T16:21:09.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 >> v1:172.21.15.154:6789/0 conn(0x7f81ec0c8b20 legacy=0x7f81ec1572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:09.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 --2- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81d00039b0 0x7f81d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:09.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 --2- 172.21.15.102:0/2505720237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d0004800 0x7f81d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:09.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 --2- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec158760 0x7f81d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:09.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec15a860 con 0x7f81d0004800 2024-03-18T16:21:09.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec072840 con 0x7f81d00039b0 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.321+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f81d0005550 con 0x7f81ec158760 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.324+0000 7f81e9e00640 1 --2- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81d00039b0 0x7f81d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.327+0000 7f81e9e00640 1 -- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec158760 msgr2=0x7f81d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.327+0000 7f81e9e00640 1 --2- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec158760 0x7f81d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.327+0000 7f81e9e00640 1 -- 172.21.15.102:0/2505720237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d0004800 msgr2=0x7f81d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.327+0000 7f81e9e00640 1 --2- 172.21.15.102:0/2505720237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d0004800 0x7f81d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.327+0000 7f81e9e00640 1 -- 172.21.15.102:0/2505720237 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec157da0 con 0x7f81d00039b0 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.330+0000 7f81e9e00640 1 --2- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81d00039b0 0x7f81d0003dd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f81dc009860 tx=0x7f81d0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1712ff836478f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.334+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc037070 con 0x7f81d00039b0 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.334+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81dc032620 con 0x7f81d00039b0 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.334+0000 7f81e8a00640 1 -- 172.21.15.102:0/2505720237 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81dc009460 con 0x7f81d00039b0 2024-03-18T16:21:09.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.335+0000 7f81f1000640 1 -- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81d00039b0 msgr2=0x7f81d0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.335+0000 7f81f1000640 1 --2- 172.21.15.102:0/2505720237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81d00039b0 0x7f81d0003dd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f81dc009860 tx=0x7f81d0005830 comp rx=0 tx=0).stop 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.335+0000 7f81ea800640 1 -- 172.21.15.102:0/2505720237 reap_dead start 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.336+0000 7f81f1000640 1 -- 172.21.15.102:0/2505720237 shutdown_connections 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.336+0000 7f81f1000640 1 -- 172.21.15.102:0/2505720237 >> 172.21.15.102:0/2505720237 conn(0x7f81ec0700d0 msgr2=0x7f81ec06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.336+0000 7f81f1000640 1 -- 172.21.15.102:0/2505720237 shutdown_connections 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.336+0000 7f81f1000640 1 -- 172.21.15.102:0/2505720237 wait complete. 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.339+0000 7f81f1000640 1 Processor -- start 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.339+0000 7f81f1000640 1 -- start start 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.340+0000 7f81f1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00039b0 0x7f81ec0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.340+0000 7f81f1000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec14d5e0 0x7f81ec14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.340+0000 7f81f1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec14df80 0x7f81ec14e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.340+0000 7f81f1000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec15ad90 con 0x7f81d00039b0 2024-03-18T16:21:09.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.340+0000 7f81f1000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81ec15aab0 con 0x7f81ec14d5e0 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.340+0000 7f81f1000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f81ec073170 con 0x7f81ec14df80 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.341+0000 7f81e9400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec14d5e0 0x7f81ec14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.341+0000 7f81e9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00039b0 0x7f81ec0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.342+0000 7f81e9400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec14d5e0 0x7f81ec14fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56546/0 (socket says 172.21.15.102:56546) 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.342+0000 7f81e9400640 1 -- 172.21.15.102:0/924546501 learned_addr learned my addr 172.21.15.102:0/924546501 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.345+0000 7f81ea800640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec14df80 0x7f81ec14e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.347+0000 7f81e9400640 1 -- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec14df80 msgr2=0x7f81ec14e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.347+0000 7f81e9400640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec14df80 0x7f81ec14e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.347+0000 7f81e9400640 1 -- 172.21.15.102:0/924546501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00039b0 msgr2=0x7f81ec0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.347+0000 7f81e9400640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00039b0 0x7f81ec0cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.347+0000 7f81e9400640 1 -- 172.21.15.102:0/924546501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81ec14c330 con 0x7f81ec14d5e0 2024-03-18T16:21:09.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.350+0000 7f81e9400640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec14d5e0 0x7f81ec14fde0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81d800c890 tx=0x7f81d800cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:09.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.355+0000 7f81e2000640 1 -- 172.21.15.102:0/924546501 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8018070 con 0x7f81ec14d5e0 2024-03-18T16:21:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.355+0000 7f81e2000640 1 -- 172.21.15.102:0/924546501 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d8007500 con 0x7f81ec14d5e0 2024-03-18T16:21:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.355+0000 7f81e2000640 1 -- 172.21.15.102:0/924546501 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8004a20 con 0x7f81ec14d5e0 2024-03-18T16:21:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.355+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81ec14ef50 con 0x7f81ec14d5e0 2024-03-18T16:21:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.355+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81ec14c5b0 con 0x7f81ec14d5e0 2024-03-18T16:21:09.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.356+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81ec074fd0 con 0x7f81ec14d5e0 2024-03-18T16:21:09.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.375+0000 7f81e2000640 1 -- 172.21.15.102:0/924546501 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7f81d801b070 con 0x7f81ec14d5e0 2024-03-18T16:21:09.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.376+0000 7f81e2000640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81cc06bbc0 0x7f81cc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:09.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.376+0000 7f81e9e00640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81cc06bbc0 0x7f81cc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:09.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.377+0000 7f81e9e00640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81cc06bbc0 0x7f81cc06e080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81dc034000 tx=0x7f81dc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:09.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.619+0000 7f81e2000640 1 -- 172.21.15.102:0/924546501 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f81d805ed10 con 0x7f81ec14d5e0 2024-03-18T16:21:09.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:09.633+0000 7f81e2000640 1 -- 172.21.15.102:0/924546501 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f81d8022020 con 0x7f81ec14d5e0 2024-03-18T16:21:10.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.120+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81ec0c4b80 con 0x7f81ec14d5e0 2024-03-18T16:21:10.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.173+0000 7f81e2000640 1 -- 172.21.15.102:0/924546501 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f81ec14c330 con 0x7f81ec14d5e0 2024-03-18T16:21:10.238 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:10.239 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T16:21:10.240 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 1 2024-03-18T16:21:10.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81cc06bbc0 msgr2=0x7f81cc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:10.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81cc06bbc0 0x7f81cc06e080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f81dc034000 tx=0x7f81dc008030 comp rx=0 tx=0).stop 2024-03-18T16:21:10.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec14d5e0 msgr2=0x7f81ec14fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:10.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec14d5e0 0x7f81ec14fde0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81d800c890 tx=0x7f81d800cd50 comp rx=0 tx=0).stop 2024-03-18T16:21:10.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 shutdown_connections 2024-03-18T16:21:10.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81cc06bbc0 0x7f81cc06e080 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:10.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f81ec14df80 0x7f81ec14e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:10.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ec14d5e0 0x7f81ec14fde0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f81d800c890 tx=0x7f81d800cd50 comp rx=0 tx=0).stop 2024-03-18T16:21:10.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 --2- 172.21.15.102:0/924546501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81d00039b0 0x7f81ec0cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:10.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 >> 172.21.15.102:0/924546501 conn(0x7f81ec0700d0 msgr2=0x7f81ec0c0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:10.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 shutdown_connections 2024-03-18T16:21:10.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:10.177+0000 7f81f1000640 1 -- 172.21.15.102:0/924546501 wait complete. 2024-03-18T16:21:12.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:21:12.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.414+0000 7f3783e00640 1 Processor -- start 2024-03-18T16:21:12.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.414+0000 7f3783e00640 1 -- start start 2024-03-18T16:21:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.415+0000 7f3783e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cfe70 0x7f378415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.415+0000 7f3783e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f37840d0250 0x7f37840c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.415+0000 7f3783e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37840cc2f0 0x7f37840cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.415+0000 7f3783e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f37840ccca0 con 0x7f37840cc2f0 2024-03-18T16:21:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.415+0000 7f3783e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f37840cabd0 con 0x7f37840d0250 2024-03-18T16:21:12.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.415+0000 7f3783e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f37840caea0 con 0x7f37840cfe70 2024-03-18T16:21:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.416+0000 7f3783400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37840cc2f0 0x7f37840cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.416+0000 7f3782a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cfe70 0x7f378415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.417+0000 7f3783400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37840cc2f0 0x7f37840cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35946/0 (socket says 172.21.15.102:35946) 2024-03-18T16:21:12.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.417+0000 7f3783400640 1 -- 172.21.15.102:0/3944916265 learned_addr learned my addr 172.21.15.102:0/3944916265 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:12.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.420+0000 7f3783400640 1 -- 172.21.15.102:0/3944916265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cfe70 msgr2=0x7f378415ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:12.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.420+0000 7f3783400640 1 --2- 172.21.15.102:0/3944916265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cfe70 0x7f378415ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:12.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.420+0000 7f3783400640 1 -- 172.21.15.102:0/3944916265 >> v2:172.21.15.154:3300/0 conn(0x7f37840d0250 msgr2=0x7f37840c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:12.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.420+0000 7f3783400640 1 --2- 172.21.15.102:0/3944916265 >> v2:172.21.15.154:3300/0 conn(0x7f37840d0250 0x7f37840c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:12.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.420+0000 7f3783400640 1 -- 172.21.15.102:0/3944916265 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37840cb180 con 0x7f37840cc2f0 2024-03-18T16:21:12.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.421+0000 7f3782a00640 1 --2- 172.21.15.102:0/3944916265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cfe70 0x7f378415ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:12.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.425+0000 7f3783400640 1 --2- 172.21.15.102:0/3944916265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37840cc2f0 0x7f37840cc760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f37840cd830 tx=0x7f377c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3cb3dabf952a3ed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:12.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.432+0000 7f3781600640 1 -- 172.21.15.102:0/3944916265 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c02cb00 con 0x7f37840cc2f0 2024-03-18T16:21:12.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.432+0000 7f3781600640 1 -- 172.21.15.102:0/3944916265 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f377c02cca0 con 0x7f37840cc2f0 2024-03-18T16:21:12.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.432+0000 7f3781600640 1 -- 172.21.15.102:0/3944916265 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c034610 con 0x7f37840cc2f0 2024-03-18T16:21:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.433+0000 7f3783e00640 1 -- 172.21.15.102:0/3944916265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37840cc2f0 msgr2=0x7f37840cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.433+0000 7f3783e00640 1 --2- 172.21.15.102:0/3944916265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37840cc2f0 0x7f37840cc760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f37840cd830 tx=0x7f377c02bf60 comp rx=0 tx=0).stop 2024-03-18T16:21:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.433+0000 7f3783e00640 1 -- 172.21.15.102:0/3944916265 shutdown_connections 2024-03-18T16:21:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.433+0000 7f3783e00640 1 --2- 172.21.15.102:0/3944916265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37840cc2f0 0x7f37840cc760 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.433+0000 7f3783e00640 1 --2- 172.21.15.102:0/3944916265 >> v2:172.21.15.154:3300/0 conn(0x7f37840d0250 0x7f37840c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.433+0000 7f3783e00640 1 --2- 172.21.15.102:0/3944916265 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cfe70 0x7f378415ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.433+0000 7f3783e00640 1 -- 172.21.15.102:0/3944916265 >> 172.21.15.102:0/3944916265 conn(0x7f378408c3e0 msgr2=0x7f378406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:12.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.433+0000 7f3783e00640 1 -- 172.21.15.102:0/3944916265 shutdown_connections 2024-03-18T16:21:12.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.433+0000 7f3783e00640 1 -- 172.21.15.102:0/3944916265 wait complete. 2024-03-18T16:21:12.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.436+0000 7f3783e00640 1 Processor -- start 2024-03-18T16:21:12.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.436+0000 7f3783e00640 1 -- start start 2024-03-18T16:21:12.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.438+0000 7f3783e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cc2f0 0x7f37841e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:12.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.438+0000 7f3783e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37840cfe70 0x7f3784156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:12.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.438+0000 7f3783e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37840d0250 0x7f378414fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:12.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.438+0000 7f3783e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f37840cba20 con 0x7f37840d0250 2024-03-18T16:21:12.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.438+0000 7f3783e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f37840cb740 con 0x7f37840cfe70 2024-03-18T16:21:12.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.438+0000 7f3783e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f37840cbd00 con 0x7f37840cc2f0 2024-03-18T16:21:12.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.442+0000 7f3782000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37840cfe70 0x7f3784156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:12.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.445+0000 7f3782000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37840cfe70 0x7f3784156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42148/0 (socket says 172.21.15.102:42148) 2024-03-18T16:21:12.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.445+0000 7f3782000640 1 -- 172.21.15.102:0/2536040998 learned_addr learned my addr 172.21.15.102:0/2536040998 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:12.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.445+0000 7f3782a00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cc2f0 0x7f37841e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:12.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.446+0000 7f3783400640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37840d0250 0x7f378414fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:12.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.449+0000 7f3782000640 1 -- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cc2f0 msgr2=0x7f37841e03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:12.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.449+0000 7f3782000640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cc2f0 0x7f37841e03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:12.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.449+0000 7f3782000640 1 -- 172.21.15.102:0/2536040998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37840d0250 msgr2=0x7f378414fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:12.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.449+0000 7f3782000640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37840d0250 0x7f378414fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:12.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.449+0000 7f3782000640 1 -- 172.21.15.102:0/2536040998 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37841e08f0 con 0x7f37840cfe70 2024-03-18T16:21:12.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.452+0000 7f3782000640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37840cfe70 0x7f3784156940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f377000b410 tx=0x7f377000b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.456+0000 7f377aa00640 1 -- 172.21.15.102:0/2536040998 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770015020 con 0x7f37840cfe70 2024-03-18T16:21:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.456+0000 7f377aa00640 1 -- 172.21.15.102:0/2536040998 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3770011850 con 0x7f37840cfe70 2024-03-18T16:21:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.456+0000 7f377aa00640 1 -- 172.21.15.102:0/2536040998 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770011c40 con 0x7f37840cfe70 2024-03-18T16:21:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.456+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37841f3b60 con 0x7f37840cfe70 2024-03-18T16:21:12.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.456+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37841e0b60 con 0x7f37840cfe70 2024-03-18T16:21:12.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.458+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3784074fd0 con 0x7f37840cfe70 2024-03-18T16:21:12.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.485+0000 7f377aa00640 1 -- 172.21.15.102:0/2536040998 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3770018020 con 0x7f37840cfe70 2024-03-18T16:21:12.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.487+0000 7f377aa00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f376406bc20 0x7f376406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:12.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.487+0000 7f3782a00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f376406bc20 0x7f376406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:12.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.488+0000 7f3782a00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f376406bc20 0x7f376406e0e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3774009560 tx=0x7f3774009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:12.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.782+0000 7f377aa00640 1 -- 172.21.15.102:0/2536040998 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f377005f6e0 con 0x7f37840cfe70 2024-03-18T16:21:12.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:12.794+0000 7f377aa00640 1 -- 172.21.15.102:0/2536040998 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f377001f020 con 0x7f37840cfe70 2024-03-18T16:21:13.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.058+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37841e0e40 con 0x7f37840cfe70 2024-03-18T16:21:13.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.064+0000 7f377aa00640 1 -- 172.21.15.102:0/2536040998 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f3770059a10 con 0x7f37840cfe70 2024-03-18T16:21:13.067 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:13.067 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T16:21:13.067 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 1 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f376406bc20 msgr2=0x7f376406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f376406bc20 0x7f376406e0e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3774009560 tx=0x7f3774009190 comp rx=0 tx=0).stop 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37840cfe70 msgr2=0x7f3784156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37840cfe70 0x7f3784156940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f377000b410 tx=0x7f377000b8d0 comp rx=0 tx=0).stop 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 shutdown_connections 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f376406bc20 0x7f376406e0e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37840d0250 0x7f378414fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37840cfe70 0x7f3784156940 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 --2- 172.21.15.102:0/2536040998 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f37840cc2f0 0x7f37841e03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 >> 172.21.15.102:0/2536040998 conn(0x7f378408c3e0 msgr2=0x7f37840cea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 shutdown_connections 2024-03-18T16:21:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:13.066+0000 7f3783e00640 1 -- 172.21.15.102:0/2536040998 wait complete. 2024-03-18T16:21:13.645 INFO:tasks.ceph.mds.f.smithi154.stdout:starting mds.f at 2024-03-18T16:21:13.762 INFO:tasks.ceph.mds.b.smithi154.stdout:starting mds.b at 2024-03-18T16:21:13.779 INFO:tasks.ceph.mds.d.smithi154.stdout:starting mds.d at 2024-03-18T16:21:14.135 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2024-03-18T16:21:14.137 INFO:tasks.ceph.mds.e.smithi102.stdout:starting mds.e at 2024-03-18T16:21:14.416 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-18T16:21:15.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:21:15.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.300+0000 7f6fd0400640 1 Processor -- start 2024-03-18T16:21:15.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.301+0000 7f6fd0400640 1 -- start start 2024-03-18T16:21:15.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.302+0000 7f6fd0400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6fc80c5050 0x7f6fc815a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:15.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.302+0000 7f6fd0400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fc815daa0 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.302+0000 7f6fd0400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fc815aad0 con 0x7f6fc8154780 2024-03-18T16:21:15.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.302+0000 7f6fd0400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6fc815e970 con 0x7f6fc80c5050 2024-03-18T16:21:15.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.302+0000 7f6fcda00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f6fc8154780 0x7f6fc815e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:53230/0 (socket says 172.21.15.102:53230) 2024-03-18T16:21:15.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.302+0000 7f6fcda00640 1 -- 172.21.15.102:0/1631140562 learned_addr learned my addr 172.21.15.102:0/1631140562 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:15.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.390+0000 7f6fcd000640 1 --2- 172.21.15.102:0/1631140562 >> v2:172.21.15.102:3300/0 conn(0x7f6fc80c5050 0x7f6fc815a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:15.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.397+0000 7f6fcd000640 1 -- 172.21.15.102:0/1631140562 >> v1:172.21.15.154:6789/0 conn(0x7f6fc8154780 legacy=0x7f6fc815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:15.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.397+0000 7f6fcd000640 1 -- 172.21.15.102:0/1631140562 >> v1:172.21.15.102:6789/0 conn(0x7f6fc80cc1f0 legacy=0x7f6fc80caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:15.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.397+0000 7f6fcd000640 1 -- 172.21.15.102:0/1631140562 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc815adb0 con 0x7f6fc80c5050 2024-03-18T16:21:15.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.409+0000 7f6fcd000640 1 --2- 172.21.15.102:0/1631140562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80c5050 0x7f6fc815a490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40097a0 tx=0x7f6fb402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45371812b33fba71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:15.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.423+0000 7f6fc7e00640 1 -- 172.21.15.102:0/1631140562 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4034070 con 0x7f6fc80c5050 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.423+0000 7f6fc7e00640 1 -- 172.21.15.102:0/1631140562 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb4030610 con 0x7f6fc80c5050 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.423+0000 7f6fc7e00640 1 -- 172.21.15.102:0/1631140562 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb40309e0 con 0x7f6fc80c5050 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.424+0000 7f6fd0400640 1 -- 172.21.15.102:0/1631140562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80c5050 msgr2=0x7f6fc815a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.424+0000 7f6fd0400640 1 --2- 172.21.15.102:0/1631140562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80c5050 0x7f6fc815a490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40097a0 tx=0x7f6fb402b960 comp rx=0 tx=0).stop 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.424+0000 7f6fd0400640 1 -- 172.21.15.102:0/1631140562 shutdown_connections 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.424+0000 7f6fd0400640 1 --2- 172.21.15.102:0/1631140562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80c5050 0x7f6fc815a490 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.424+0000 7f6fd0400640 1 -- 172.21.15.102:0/1631140562 >> 172.21.15.102:0/1631140562 conn(0x7f6fc8067160 msgr2=0x7f6fc8067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.425+0000 7f6fd0400640 1 -- 172.21.15.102:0/1631140562 shutdown_connections 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.425+0000 7f6fd0400640 1 -- 172.21.15.102:0/1631140562 wait complete. 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.429+0000 7f6fd0400640 1 Processor -- start 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.429+0000 7f6fd0400640 1 -- start start 2024-03-18T16:21:15.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.431+0000 7f6fd0400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc80c5050 0x7f6fc815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.431+0000 7f6fd0400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80cc1f0 0x7f6fc814ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.431+0000 7f6fd0400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc8154780 0x7f6fc814f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.431+0000 7f6fd0400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc815b370 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.431+0000 7f6fd0400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc815dd40 con 0x7f6fc80c5050 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.431+0000 7f6fd0400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc8154b60 con 0x7f6fc8154780 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.432+0000 7f6fcd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80cc1f0 0x7f6fc814ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.432+0000 7f6fcd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80cc1f0 0x7f6fc814ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36008/0 (socket says 172.21.15.102:36008) 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.432+0000 7f6fcd000640 1 -- 172.21.15.102:0/2268096146 learned_addr learned my addr 172.21.15.102:0/2268096146 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.436+0000 7f6fcda00640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc80c5050 0x7f6fc815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.437+0000 7f6fcd000640 1 -- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc8154780 msgr2=0x7f6fc814f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.437+0000 7f6fcd000640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc8154780 0x7f6fc814f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.437+0000 7f6fcd000640 1 -- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc80c5050 msgr2=0x7f6fc815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.437+0000 7f6fcd000640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc80c5050 0x7f6fc815cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.437+0000 7f6fcd000640 1 -- 172.21.15.102:0/2268096146 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc8150550 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.441+0000 7f6fcd000640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80cc1f0 0x7f6fc814ed90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009890 tx=0x7f6fb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:15.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.447+0000 7f6fc5600640 1 -- 172.21.15.102:0/2268096146 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4034040 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.447+0000 7f6fc5600640 1 -- 172.21.15.102:0/2268096146 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb403acc0 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.447+0000 7f6fc5600640 1 -- 172.21.15.102:0/2268096146 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4044630 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.448+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc81585c0 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.448+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc80c6a30 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.450+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa4005290 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.788+0000 7f6fc5600640 1 -- 172.21.15.102:0/2268096146 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6fb404b060 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.789+0000 7f6fc5600640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fa806bc20 0x7f6fa806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.790+0000 7f6fcda00640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fa806bc20 0x7f6fa806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.790+0000 7f6fcda00640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fa806bc20 0x7f6fa806e0e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc005880 tx=0x7f6fbc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.805+0000 7f6fc5600640 1 -- 172.21.15.102:0/2268096146 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f6fb4081500 con 0x7f6fc80cc1f0 2024-03-18T16:21:15.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:15.806+0000 7f6fc5600640 1 -- 172.21.15.102:0/2268096146 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6fb403f070 con 0x7f6fc80cc1f0 2024-03-18T16:21:16.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.071+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fa4005580 con 0x7f6fc80cc1f0 2024-03-18T16:21:16.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.106+0000 7f6fc5600640 1 -- 172.21.15.102:0/2268096146 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5524 (secure 0 0 0) 0x7f6fb407b830 con 0x7f6fc80cc1f0 2024-03-18T16:21:16.121 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:16.121 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4250,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6834/3423054813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3423054813},{"type":"v1","addr":"172.21.15.102:6834","nonce":3423054813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4253,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4256,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 3 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.109+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fa806bc20 msgr2=0x7f6fa806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.109+0000 7f6fd0400640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fa806bc20 0x7f6fa806e0e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc005880 tx=0x7f6fbc009040 comp rx=0 tx=0).stop 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.109+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80cc1f0 msgr2=0x7f6fc814ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.109+0000 7f6fd0400640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80cc1f0 0x7f6fc814ed90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009890 tx=0x7f6fb40046c0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.110+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 shutdown_connections 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.110+0000 7f6fd0400640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fa806bc20 0x7f6fa806e0e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.110+0000 7f6fd0400640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fc8154780 0x7f6fc814f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.110+0000 7f6fd0400640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fc80cc1f0 0x7f6fc814ed90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.110+0000 7f6fd0400640 1 --2- 172.21.15.102:0/2268096146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fc80c5050 0x7f6fc815cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.110+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 >> 172.21.15.102:0/2268096146 conn(0x7f6fc8067160 msgr2=0x7f6fc80caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:16.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.110+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 shutdown_connections 2024-03-18T16:21:16.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.110+0000 7f6fd0400640 1 -- 172.21.15.102:0/2268096146 wait complete. 2024-03-18T16:21:16.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:16.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.329+0000 7fc6fb000640 1 Processor -- start 2024-03-18T16:21:16.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.330+0000 7fc6fb000640 1 -- start start 2024-03-18T16:21:16.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.331+0000 7fc6fb000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f4072700 0x7fc6f4158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:16.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.331+0000 7fc6fb000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc6f415b8e0 0x7fc6f40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:16.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.331+0000 7fc6fb000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6f415c200 con 0x7fc6f41524c0 2024-03-18T16:21:16.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.331+0000 7fc6fb000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc6f4072b20 con 0x7fc6f415b8e0 2024-03-18T16:21:16.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.331+0000 7fc6fb000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f415c4e0 con 0x7fc6f4072700 2024-03-18T16:21:16.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.332+0000 7fc6f3e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc6f41524c0 0x7fc6f406e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:53246/0 (socket says 172.21.15.102:53246) 2024-03-18T16:21:16.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.332+0000 7fc6f3e00640 1 -- 172.21.15.102:0/443693087 learned_addr learned my addr 172.21.15.102:0/443693087 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:16.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.332+0000 7fc6eb400640 1 --2- 172.21.15.102:0/443693087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f4072700 0x7fc6f4158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:16.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.335+0000 7fc6f3400640 1 -- 172.21.15.102:0/443693087 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569861510 0 0) 0x7fc6f415c200 con 0x7fc6f41524c0 2024-03-18T16:21:16.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.335+0000 7fc6f3400640 1 -- 172.21.15.102:0/443693087 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d8003520 con 0x7fc6f41524c0 2024-03-18T16:21:16.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.335+0000 7fc6eb400640 1 -- 172.21.15.102:0/443693087 >> v2:172.21.15.102:3300/0 conn(0x7fc6f415b8e0 msgr2=0x7fc6f40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:16.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.335+0000 7fc6eb400640 1 --2- 172.21.15.102:0/443693087 >> v2:172.21.15.102:3300/0 conn(0x7fc6f415b8e0 0x7fc6f40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.335+0000 7fc6eb400640 1 -- 172.21.15.102:0/443693087 >> v1:172.21.15.154:6789/0 conn(0x7fc6f41524c0 legacy=0x7fc6f406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:16.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.335+0000 7fc6eb400640 1 -- 172.21.15.102:0/443693087 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f415c7c0 con 0x7fc6f4072700 2024-03-18T16:21:16.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.337+0000 7fc6eb400640 1 --2- 172.21.15.102:0/443693087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f4072700 0x7fc6f4158ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc0097a0 tx=0x7fc6dc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=644149c0e39e8239 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:16.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.340+0000 7fc6f3400640 1 -- 172.21.15.102:0/443693087 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc035020 con 0x7fc6f4072700 2024-03-18T16:21:16.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.340+0000 7fc6f3400640 1 -- 172.21.15.102:0/443693087 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc031830 con 0x7fc6f4072700 2024-03-18T16:21:16.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.340+0000 7fc6f3400640 1 -- 172.21.15.102:0/443693087 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc031c00 con 0x7fc6f4072700 2024-03-18T16:21:16.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.341+0000 7fc6fb000640 1 -- 172.21.15.102:0/443693087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f4072700 msgr2=0x7fc6f4158ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:16.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.341+0000 7fc6fb000640 1 --2- 172.21.15.102:0/443693087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f4072700 0x7fc6f4158ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc0097a0 tx=0x7fc6dc02b8d0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.341+0000 7fc6fb000640 1 -- 172.21.15.102:0/443693087 shutdown_connections 2024-03-18T16:21:16.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.341+0000 7fc6fb000640 1 --2- 172.21.15.102:0/443693087 >> v2:172.21.15.102:3300/0 conn(0x7fc6f415b8e0 0x7fc6f40c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.341+0000 7fc6fb000640 1 --2- 172.21.15.102:0/443693087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f4072700 0x7fc6f4158ee0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.341+0000 7fc6fb000640 1 -- 172.21.15.102:0/443693087 >> 172.21.15.102:0/443693087 conn(0x7fc6f40700d0 msgr2=0x7fc6f406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:16.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.341+0000 7fc6fb000640 1 -- 172.21.15.102:0/443693087 shutdown_connections 2024-03-18T16:21:16.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.341+0000 7fc6fb000640 1 -- 172.21.15.102:0/443693087 wait complete. 2024-03-18T16:21:16.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.343+0000 7fc6fb000640 1 Processor -- start 2024-03-18T16:21:16.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.343+0000 7fc6fb000640 1 -- start start 2024-03-18T16:21:16.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6fb000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6f4072700 0x7fc6f414bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:16.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6fb000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f41524c0 0x7fc6f414cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:16.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6fb000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f415b8e0 0x7fc6f41f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:16.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6fb000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f41589c0 con 0x7fc6f415b8e0 2024-03-18T16:21:16.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6fb000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f41586e0 con 0x7fc6f4072700 2024-03-18T16:21:16.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6fb000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f41528a0 con 0x7fc6f41524c0 2024-03-18T16:21:16.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6f3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6f4072700 0x7fc6f414bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:16.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6eb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f41524c0 0x7fc6f414cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:16.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6f3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6f4072700 0x7fc6f414bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42202/0 (socket says 172.21.15.102:42202) 2024-03-18T16:21:16.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.344+0000 7fc6f3e00640 1 -- 172.21.15.102:0/3781989144 learned_addr learned my addr 172.21.15.102:0/3781989144 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:16.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.348+0000 7fc6f3e00640 1 -- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f41524c0 msgr2=0x7fc6f414cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:16.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.348+0000 7fc6f3e00640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f41524c0 0x7fc6f414cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.348+0000 7fc6f3e00640 1 -- 172.21.15.102:0/3781989144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f415b8e0 msgr2=0x7fc6f41f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:16.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.348+0000 7fc6f3e00640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f415b8e0 0x7fc6f41f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:16.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.348+0000 7fc6f3e00640 1 -- 172.21.15.102:0/3781989144 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f414b5a0 con 0x7fc6f4072700 2024-03-18T16:21:16.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.351+0000 7fc6f3e00640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6f4072700 0x7fc6f414bb50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400c500 tx=0x7fc6e400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:16.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.355+0000 7fc6f0c00640 1 -- 172.21.15.102:0/3781989144 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4017070 con 0x7fc6f4072700 2024-03-18T16:21:16.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.355+0000 7fc6f0c00640 1 -- 172.21.15.102:0/3781989144 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e4005ce0 con 0x7fc6f4072700 2024-03-18T16:21:16.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.355+0000 7fc6f0c00640 1 -- 172.21.15.102:0/3781989144 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4007330 con 0x7fc6f4072700 2024-03-18T16:21:16.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.355+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f41f4040 con 0x7fc6f4072700 2024-03-18T16:21:16.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.355+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f40c7ba0 con 0x7fc6f4072700 2024-03-18T16:21:16.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.356+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6f4074fd0 con 0x7fc6f4072700 2024-03-18T16:21:16.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.605+0000 7fc6f0c00640 1 -- 172.21.15.102:0/3781989144 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc6e401a070 con 0x7fc6f4072700 2024-03-18T16:21:16.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.607+0000 7fc6f0c00640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc6d006bc50 0x7fc6d006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:16.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.608+0000 7fc6f0c00640 1 -- 172.21.15.102:0/3781989144 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fc6e4089f00 con 0x7fc6f4072700 2024-03-18T16:21:16.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.608+0000 7fc6eb400640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc6d006bc50 0x7fc6d006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:16.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.609+0000 7fc6eb400640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc6d006bc50 0x7fc6d006e110 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc036000 tx=0x7fc6dc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:16.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.617+0000 7fc6f0c00640 1 -- 172.21.15.102:0/3781989144 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc6e4021020 con 0x7fc6f4072700 2024-03-18T16:21:16.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:16.830+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6f4156b70 con 0x7fc6f4072700 2024-03-18T16:21:17.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.087+0000 7fc6f0c00640 1 -- 172.21.15.102:0/3781989144 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12944 (secure 0 0 0) 0x7fc6e4058820 con 0x7fc6f4072700 2024-03-18T16:21:17.090 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:17.090 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":17,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:09.030388+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc6d006bc50 msgr2=0x7fc6d006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc6d006bc50 0x7fc6d006e110 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc036000 tx=0x7fc6dc03c040 comp rx=0 tx=0).stop 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6f4072700 msgr2=0x7fc6f414bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6f4072700 0x7fc6f414bb50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400c500 tx=0x7fc6e400c9c0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 shutdown_connections 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc6d006bc50 0x7fc6d006e110 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f415b8e0 0x7fc6f41f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc6f41524c0 0x7fc6f414cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 --2- 172.21.15.102:0/3781989144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6f4072700 0x7fc6f414bb50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 >> 172.21.15.102:0/3781989144 conn(0x7fc6f40700d0 msgr2=0x7fc6f4071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:17.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 shutdown_connections 2024-03-18T16:21:17.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.091+0000 7fc6fb000640 1 -- 172.21.15.102:0/3781989144 wait complete. 2024-03-18T16:21:17.115 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:21:05.204661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T16:21:17.171 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:21:17.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.379+0000 7f61b3000640 1 Processor -- start 2024-03-18T16:21:17.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.379+0000 7f61b3000640 1 -- start start 2024-03-18T16:21:17.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.381+0000 7f61b3000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac0723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:17.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.381+0000 7f61b3000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f61ac06ba70 0x7f61ac06be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:17.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.381+0000 7f61b3000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61ac158390 con 0x7f61ac0c4fb0 2024-03-18T16:21:17.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.381+0000 7f61b3000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f61ac141ae0 con 0x7f61ac06ba70 2024-03-18T16:21:17.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.381+0000 7f61b3000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f61ac158670 con 0x7f61ac141700 2024-03-18T16:21:17.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.382+0000 7f61abe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac0723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:17.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.382+0000 7f61abe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac0723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58958/0 (socket says 172.21.15.102:58958) 2024-03-18T16:21:17.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.382+0000 7f61abe00640 1 -- 172.21.15.102:0/766806870 learned_addr learned my addr 172.21.15.102:0/766806870 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:17.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.382+0000 7f61a3400640 1 --2- 172.21.15.102:0/766806870 >> v2:172.21.15.102:3300/0 conn(0x7f61ac06ba70 0x7f61ac06be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:17.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.384+0000 7f61abe00640 1 -- 172.21.15.102:0/766806870 >> v2:172.21.15.102:3300/0 conn(0x7f61ac06ba70 msgr2=0x7f61ac06be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:17.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.384+0000 7f61abe00640 1 --2- 172.21.15.102:0/766806870 >> v2:172.21.15.102:3300/0 conn(0x7f61ac06ba70 0x7f61ac06be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.384+0000 7f61abe00640 1 -- 172.21.15.102:0/766806870 >> v1:172.21.15.102:6789/0 conn(0x7f61ac0c4fb0 legacy=0x7f61ac15bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:17.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.384+0000 7f61abe00640 1 -- 172.21.15.102:0/766806870 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61ac158950 con 0x7f61ac141700 2024-03-18T16:21:17.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.386+0000 7f61abe00640 1 --2- 172.21.15.102:0/766806870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac0723b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f619c009870 tx=0x7f619c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20ca10f16114dd98 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:17.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.390+0000 7f61ab400640 1 -- 172.21.15.102:0/766806870 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c02f830 con 0x7f61ac141700 2024-03-18T16:21:17.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.390+0000 7f61ab400640 1 -- 172.21.15.102:0/766806870 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f619c02f9d0 con 0x7f61ac141700 2024-03-18T16:21:17.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.390+0000 7f61ab400640 1 -- 172.21.15.102:0/766806870 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c02fda0 con 0x7f61ac141700 2024-03-18T16:21:17.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.391+0000 7f61b3000640 1 -- 172.21.15.102:0/766806870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 msgr2=0x7f61ac0723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:17.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.391+0000 7f61b3000640 1 --2- 172.21.15.102:0/766806870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac0723b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f619c009870 tx=0x7f619c02ba10 comp rx=0 tx=0).stop 2024-03-18T16:21:17.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.391+0000 7f61b3000640 1 -- 172.21.15.102:0/766806870 shutdown_connections 2024-03-18T16:21:17.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.391+0000 7f61b3000640 1 --2- 172.21.15.102:0/766806870 >> v2:172.21.15.102:3300/0 conn(0x7f61ac06ba70 0x7f61ac06be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.391+0000 7f61b3000640 1 --2- 172.21.15.102:0/766806870 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac0723b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.391+0000 7f61b3000640 1 -- 172.21.15.102:0/766806870 >> 172.21.15.102:0/766806870 conn(0x7f61ac0c83a0 msgr2=0x7f61ac0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:17.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.391+0000 7f61b3000640 1 -- 172.21.15.102:0/766806870 shutdown_connections 2024-03-18T16:21:17.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.391+0000 7f61b3000640 1 -- 172.21.15.102:0/766806870 wait complete. 2024-03-18T16:21:17.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.393+0000 7f61b3000640 1 Processor -- start 2024-03-18T16:21:17.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.393+0000 7f61b3000640 1 -- start start 2024-03-18T16:21:17.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.394+0000 7f61b3000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac06ba70 0x7f61ac1f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:17.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.394+0000 7f61b3000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61ac0c4fb0 0x7f61ac13e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:17.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.394+0000 7f61b3000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac1404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:17.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.394+0000 7f61b3000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f61ac15c260 con 0x7f61ac06ba70 2024-03-18T16:21:17.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.394+0000 7f61b3000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f61ac158f10 con 0x7f61ac0c4fb0 2024-03-18T16:21:17.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.394+0000 7f61b3000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f61ac15c3e0 con 0x7f61ac141700 2024-03-18T16:21:17.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.395+0000 7f61abe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac06ba70 0x7f61ac1f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:17.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.395+0000 7f61a3400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61ac0c4fb0 0x7f61ac13e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:17.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.395+0000 7f61b1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac1404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:17.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.395+0000 7f61abe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac06ba70 0x7f61ac1f10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:36036/0 (socket says 172.21.15.102:36036) 2024-03-18T16:21:17.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.395+0000 7f61abe00640 1 -- 172.21.15.102:0/1098785214 learned_addr learned my addr 172.21.15.102:0/1098785214 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:17.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.399+0000 7f61b1000640 1 -- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61ac0c4fb0 msgr2=0x7f61ac13e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:17.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.399+0000 7f61b1000640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61ac0c4fb0 0x7f61ac13e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.399+0000 7f61b1000640 1 -- 172.21.15.102:0/1098785214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac06ba70 msgr2=0x7f61ac1f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:17.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.399+0000 7f61b1000640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac06ba70 0x7f61ac1f10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:17.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.399+0000 7f61b1000640 1 -- 172.21.15.102:0/1098785214 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61ac1e1520 con 0x7f61ac141700 2024-03-18T16:21:17.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.399+0000 7f61a3400640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61ac0c4fb0 0x7f61ac13e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:17.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.401+0000 7f61b1000640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac1404f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61a400c690 tx=0x7f61a400cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:17.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.405+0000 7f61a8c00640 1 -- 172.21.15.102:0/1098785214 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4003b70 con 0x7f61ac141700 2024-03-18T16:21:17.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.405+0000 7f61a8c00640 1 -- 172.21.15.102:0/1098785214 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a4003d10 con 0x7f61ac141700 2024-03-18T16:21:17.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.405+0000 7f61a8c00640 1 -- 172.21.15.102:0/1098785214 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4005310 con 0x7f61ac141700 2024-03-18T16:21:17.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.405+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61ac1f3f70 con 0x7f61ac141700 2024-03-18T16:21:17.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.405+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61ac1f4530 con 0x7f61ac141700 2024-03-18T16:21:17.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.407+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61ac074fd0 con 0x7f61ac141700 2024-03-18T16:21:17.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.426+0000 7f61a8c00640 1 -- 172.21.15.102:0/1098785214 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f61a40054b0 con 0x7f61ac141700 2024-03-18T16:21:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.426+0000 7f61a8c00640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f618806bc20 0x7f618806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.427+0000 7f61a8c00640 1 -- 172.21.15.102:0/1098785214 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f61a408ad40 con 0x7f61ac141700 2024-03-18T16:21:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.427+0000 7f61abe00640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f618806bc20 0x7f618806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.428+0000 7f61abe00640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f618806bc20 0x7f618806e0e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f619c009870 tx=0x7f619c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:17.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.667+0000 7f61a8c00640 1 -- 172.21.15.102:0/1098785214 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f61a40596a0 con 0x7f61ac141700 2024-03-18T16:21:17.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:17.877+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f61ac0c8990 con 0x7f61ac141700 2024-03-18T16:21:18.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.008+0000 7f61a8c00640 1 -- 172.21.15.102:0/1098785214 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f61a405d740 con 0x7f61ac141700 2024-03-18T16:21:18.012 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-18T16:21:18.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f618806bc20 msgr2=0x7f618806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:18.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f618806bc20 0x7f618806e0e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f619c009870 tx=0x7f619c008030 comp rx=0 tx=0).stop 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 msgr2=0x7f61ac1404f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac1404f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61a400c690 tx=0x7f61a400cb50 comp rx=0 tx=0).stop 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 shutdown_connections 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f618806bc20 0x7f618806e0e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f61ac141700 0x7f61ac1404f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61ac0c4fb0 0x7f61ac13e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 --2- 172.21.15.102:0/1098785214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61ac06ba70 0x7f61ac1f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 >> 172.21.15.102:0/1098785214 conn(0x7f61ac0c83a0 msgr2=0x7f61ac0723b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 shutdown_connections 2024-03-18T16:21:18.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.013+0000 7f61b3000640 1 -- 172.21.15.102:0/1098785214 wait complete. 2024-03-18T16:21:18.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T16:21:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.240+0000 7f2b05600640 1 Processor -- start 2024-03-18T16:21:18.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.240+0000 7f2b05600640 1 -- start start 2024-03-18T16:21:18.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.241+0000 7f2b05600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2b000c5830 0x7f2b00141580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:18.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.241+0000 7f2b05600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b00072050 con 0x7f2b000c5450 2024-03-18T16:21:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.241+0000 7f2b05600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b000722f0 con 0x7f2b001582f0 2024-03-18T16:21:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.241+0000 7f2b05600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2b0015c8b0 con 0x7f2b000c5830 2024-03-18T16:21:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.241+0000 7f2aff400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f2b001582f0 0x7f2b00151ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:53258/0 (socket says 172.21.15.102:53258) 2024-03-18T16:21:18.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.241+0000 7f2aff400640 1 -- 172.21.15.102:0/2513948683 learned_addr learned my addr 172.21.15.102:0/2513948683 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:18.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.242+0000 7f2afe000640 1 --2- 172.21.15.102:0/2513948683 >> v2:172.21.15.102:3300/0 conn(0x7f2b000c5830 0x7f2b00141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:18.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.244+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050225579 0 0) 0x7f2b000722f0 con 0x7f2b001582f0 2024-03-18T16:21:18.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.244+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ae4003520 con 0x7f2b001582f0 2024-03-18T16:21:18.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3074200270 0 0) 0x7f2ae4003520 con 0x7f2b001582f0 2024-03-18T16:21:18.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 >> v2:172.21.15.102:3300/0 conn(0x7f2b000c5830 msgr2=0x7f2b00141580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:18.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 --2- 172.21.15.102:0/2513948683 >> v2:172.21.15.102:3300/0 conn(0x7f2b000c5830 0x7f2b00141580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 >> v1:172.21.15.102:6789/0 conn(0x7f2b000c5450 legacy=0x7f2b00071940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:18.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b0015baf0 con 0x7f2b001582f0 2024-03-18T16:21:18.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f2af8001670 con 0x7f2b001582f0 2024-03-18T16:21:18.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 >> v1:172.21.15.154:6789/0 conn(0x7f2b001582f0 legacy=0x7f2b00151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:18.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae4003a00 0x7f2ae4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:18.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae4004800 0x7f2ae4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:18.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ae40051c0 0x7f2ae400fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:18.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0015c8b0 con 0x7f2ae4004800 2024-03-18T16:21:18.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.247+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae40104d0 con 0x7f2ae4003a00 2024-03-18T16:21:18.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.248+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae40106d0 con 0x7f2ae40051c0 2024-03-18T16:21:18.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.248+0000 7f2aff400640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ae40051c0 0x7f2ae400fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:18.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.250+0000 7f2afea00640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae4003a00 0x7f2ae4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:18.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.251+0000 7f2aff400640 1 -- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae4003a00 msgr2=0x7f2ae4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:18.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.251+0000 7f2aff400640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae4003a00 0x7f2ae4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.251+0000 7f2aff400640 1 -- 172.21.15.102:0/2513948683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae4004800 msgr2=0x7f2ae4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:18.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.251+0000 7f2aff400640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ae4004800 0x7f2ae4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.251+0000 7f2aff400640 1 -- 172.21.15.102:0/2513948683 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae4003520 con 0x7f2ae40051c0 2024-03-18T16:21:18.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.252+0000 7f2aff400640 1 -- 172.21.15.102:0/2513948683 reap_dead start 2024-03-18T16:21:18.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.253+0000 7f2aff400640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ae40051c0 0x7f2ae400fdd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2af800c110 tx=0x7f2ae4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a675504aa6020cf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:18.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.257+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8007c40 con 0x7f2ae40051c0 2024-03-18T16:21:18.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.257+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2af8030070 con 0x7f2ae40051c0 2024-03-18T16:21:18.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.257+0000 7f2afd600640 1 -- 172.21.15.102:0/2513948683 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8005d50 con 0x7f2ae40051c0 2024-03-18T16:21:18.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.258+0000 7f2b05600640 1 -- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ae40051c0 msgr2=0x7f2ae400fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:18.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.258+0000 7f2b05600640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ae40051c0 0x7f2ae400fdd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2af800c110 tx=0x7f2ae4005830 comp rx=0 tx=0).stop 2024-03-18T16:21:18.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.258+0000 7f2b05600640 1 -- 172.21.15.102:0/2513948683 shutdown_connections 2024-03-18T16:21:18.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.258+0000 7f2b05600640 1 --2- 172.21.15.102:0/2513948683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ae40051c0 0x7f2ae400fdd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.259+0000 7f2b05600640 1 -- 172.21.15.102:0/2513948683 >> 172.21.15.102:0/2513948683 conn(0x7f2b000700d0 msgr2=0x7f2b0006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.259+0000 7f2b05600640 1 -- 172.21.15.102:0/2513948683 shutdown_connections 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.259+0000 7f2b05600640 1 -- 172.21.15.102:0/2513948683 wait complete. 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.261+0000 7f2b05600640 1 Processor -- start 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.261+0000 7f2b05600640 1 -- start start 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2b05600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae40051c0 0x7f2b001ec4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2b05600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b000c6d70 0x7f2b000c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2b05600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b000c3140 0x7f2b000c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2b05600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2b00158780 con 0x7f2b000c3140 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2b05600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0015c0b0 con 0x7f2ae40051c0 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2b05600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b00158900 con 0x7f2b000c6d70 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2afe000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b000c6d70 0x7f2b000c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2afea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae40051c0 0x7f2b001ec4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2aff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b000c3140 0x7f2b000c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:18.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2afea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae40051c0 0x7f2b001ec4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42228/0 (socket says 172.21.15.102:42228) 2024-03-18T16:21:18.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.264+0000 7f2afea00640 1 -- 172.21.15.102:0/46213721 learned_addr learned my addr 172.21.15.102:0/46213721 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:18.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.267+0000 7f2afea00640 1 -- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b000c6d70 msgr2=0x7f2b000c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:18.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.267+0000 7f2afea00640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b000c6d70 0x7f2b000c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.267+0000 7f2afea00640 1 -- 172.21.15.102:0/46213721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b000c3140 msgr2=0x7f2b000c3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:18.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.267+0000 7f2afea00640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b000c3140 0x7f2b000c3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.267+0000 7f2afea00640 1 -- 172.21.15.102:0/46213721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b001ec9f0 con 0x7f2ae40051c0 2024-03-18T16:21:18.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.269+0000 7f2afea00640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae40051c0 0x7f2b001ec4b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800f5f0 tx=0x7f2ae800f9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:18.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.274+0000 7f2af6a00640 1 -- 172.21.15.102:0/46213721 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae80052e0 con 0x7f2ae40051c0 2024-03-18T16:21:18.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.274+0000 7f2af6a00640 1 -- 172.21.15.102:0/46213721 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae8018070 con 0x7f2ae40051c0 2024-03-18T16:21:18.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.274+0000 7f2af6a00640 1 -- 172.21.15.102:0/46213721 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8003810 con 0x7f2ae40051c0 2024-03-18T16:21:18.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.274+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b000c76f0 con 0x7f2ae40051c0 2024-03-18T16:21:18.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.274+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b000c4d00 con 0x7f2ae40051c0 2024-03-18T16:21:18.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.274+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b000caa50 con 0x7f2ae40051c0 2024-03-18T16:21:18.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.295+0000 7f2af6a00640 1 -- 172.21.15.102:0/46213721 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2ae80039b0 con 0x7f2ae40051c0 2024-03-18T16:21:18.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.296+0000 7f2af6a00640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2adc06bc50 0x7f2adc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:18.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.296+0000 7f2afe000640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2adc06bc50 0x7f2adc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:18.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.297+0000 7f2afe000640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2adc06bc50 0x7f2adc06e110 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b000c23a0 tx=0x7f2af000d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:18.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.525+0000 7f2af6a00640 1 -- 172.21.15.102:0/46213721 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f2ae805f9e0 con 0x7f2ae40051c0 2024-03-18T16:21:18.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.549+0000 7f2af6a00640 1 -- 172.21.15.102:0/46213721 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2b000caa50 con 0x7f2ae40051c0 2024-03-18T16:21:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.771+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2b000c5ec0 con 0x7f2ae40051c0 2024-03-18T16:21:18.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.778+0000 7f2af6a00640 1 -- 172.21.15.102:0/46213721 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f2b000c5ec0 con 0x7f2ae40051c0 2024-03-18T16:21:18.780 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:18.780 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-03-18T16:21:18.780 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-18T16:21:18.780 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-03-18T16:21:18.781 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-03-18T16:21:18.781 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-18T16:21:18.781 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-03-18T16:21:18.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2adc06bc50 msgr2=0x7f2adc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:18.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2adc06bc50 0x7f2adc06e110 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2b000c23a0 tx=0x7f2af000d040 comp rx=0 tx=0).stop 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae40051c0 msgr2=0x7f2b001ec4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae40051c0 0x7f2b001ec4b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800f5f0 tx=0x7f2ae800f9f0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 shutdown_connections 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2adc06bc50 0x7f2adc06e110 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b000c3140 0x7f2b000c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b000c6d70 0x7f2b000c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 --2- 172.21.15.102:0/46213721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ae40051c0 0x7f2b001ec4b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 >> 172.21.15.102:0/46213721 conn(0x7f2b000700d0 msgr2=0x7f2b000cc000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 shutdown_connections 2024-03-18T16:21:18.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.780+0000 7f2b05600640 1 -- 172.21.15.102:0/46213721 wait complete. 2024-03-18T16:21:18.801 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T16:21:18.802 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T16:21:19.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.998+0000 7f1e0ea00640 1 Processor -- start 2024-03-18T16:21:19.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:18.999+0000 7f1e0ea00640 1 -- start start 2024-03-18T16:21:19.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.000+0000 7f1e0ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1e08141700 0x7f1e080723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:19.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.000+0000 7f1e0ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0806ba70 0x7f1e0806be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:19.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.000+0000 7f1e0ea00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e08158390 con 0x7f1e080c4fb0 2024-03-18T16:21:19.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.000+0000 7f1e0ea00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1e08141ae0 con 0x7f1e08141700 2024-03-18T16:21:19.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.000+0000 7f1e0ea00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1e08158670 con 0x7f1e0806ba70 2024-03-18T16:21:19.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.001+0000 7f1e0ca00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f1e080c4fb0 0x7f1e0815bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:53266/0 (socket says 172.21.15.102:53266) 2024-03-18T16:21:19.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.001+0000 7f1e0ca00640 1 -- 172.21.15.102:0/2880397685 learned_addr learned my addr 172.21.15.102:0/2880397685 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:19.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.001+0000 7f1e07400640 1 --2- 172.21.15.102:0/2880397685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0806ba70 0x7f1e0806be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.005+0000 7f1e07400640 1 -- 172.21.15.102:0/2880397685 >> v2:172.21.15.102:3300/0 conn(0x7f1e08141700 msgr2=0x7f1e080723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.005+0000 7f1e07400640 1 --2- 172.21.15.102:0/2880397685 >> v2:172.21.15.102:3300/0 conn(0x7f1e08141700 0x7f1e080723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.005+0000 7f1e07400640 1 -- 172.21.15.102:0/2880397685 >> v1:172.21.15.154:6789/0 conn(0x7f1e080c4fb0 legacy=0x7f1e0815bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.005+0000 7f1e07400640 1 -- 172.21.15.102:0/2880397685 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e08158950 con 0x7f1e0806ba70 2024-03-18T16:21:19.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.006+0000 7f1e07400640 1 --2- 172.21.15.102:0/2880397685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0806ba70 0x7f1e0806be90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1df40097a0 tx=0x7f1df402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=379a9a61afb182aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:19.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e06a00640 1 -- 172.21.15.102:0/2880397685 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4034070 con 0x7f1e0806ba70 2024-03-18T16:21:19.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e06a00640 1 -- 172.21.15.102:0/2880397685 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df4030800 con 0x7f1e0806ba70 2024-03-18T16:21:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e06a00640 1 -- 172.21.15.102:0/2880397685 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4030bd0 con 0x7f1e0806ba70 2024-03-18T16:21:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2880397685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0806ba70 msgr2=0x7f1e0806be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e0ea00640 1 --2- 172.21.15.102:0/2880397685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0806ba70 0x7f1e0806be90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1df40097a0 tx=0x7f1df402b940 comp rx=0 tx=0).stop 2024-03-18T16:21:19.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2880397685 shutdown_connections 2024-03-18T16:21:19.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e0ea00640 1 --2- 172.21.15.102:0/2880397685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e0806ba70 0x7f1e0806be90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:19.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e0ea00640 1 --2- 172.21.15.102:0/2880397685 >> v2:172.21.15.102:3300/0 conn(0x7f1e08141700 0x7f1e080723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:19.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2880397685 >> 172.21.15.102:0/2880397685 conn(0x7f1e080c83a0 msgr2=0x7f1e080c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:19.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2880397685 shutdown_connections 2024-03-18T16:21:19.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.011+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2880397685 wait complete. 2024-03-18T16:21:19.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.012+0000 7f1e0ea00640 1 Processor -- start 2024-03-18T16:21:19.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.013+0000 7f1e0ea00640 1 -- start start 2024-03-18T16:21:19.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.013+0000 7f1e0ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e0806ba70 0x7f1e081e28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:19.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.014+0000 7f1e0ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e080c4fb0 0x7f1e081f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:19.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.014+0000 7f1e0ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e08141700 0x7f1e0813d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:19.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.014+0000 7f1e0ea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0815c260 con 0x7f1e0806ba70 2024-03-18T16:21:19.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.014+0000 7f1e0ea00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e08158f10 con 0x7f1e080c4fb0 2024-03-18T16:21:19.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.014+0000 7f1e0ea00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1e0815c3e0 con 0x7f1e08141700 2024-03-18T16:21:19.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.014+0000 7f1e0ca00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e08141700 0x7f1e0813d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:19.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.014+0000 7f1e07400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e080c4fb0 0x7f1e081f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:19.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.015+0000 7f1e07400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e080c4fb0 0x7f1e081f0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42234/0 (socket says 172.21.15.102:42234) 2024-03-18T16:21:19.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.015+0000 7f1e07400640 1 -- 172.21.15.102:0/2771587273 learned_addr learned my addr 172.21.15.102:0/2771587273 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:19.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.018+0000 7f1e07400640 1 -- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e08141700 msgr2=0x7f1e0813d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:19.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.018+0000 7f1e07400640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e08141700 0x7f1e0813d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:19.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.018+0000 7f1e07400640 1 -- 172.21.15.102:0/2771587273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e0806ba70 msgr2=0x7f1e081e28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:19.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.018+0000 7f1e07400640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e0806ba70 0x7f1e081e28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:19.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.018+0000 7f1e07400640 1 -- 172.21.15.102:0/2771587273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e081e31c0 con 0x7f1e080c4fb0 2024-03-18T16:21:19.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.020+0000 7f1e07400640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e080c4fb0 0x7f1e081f0840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1df40098c0 tx=0x7f1df40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:19.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.025+0000 7f1dffe00640 1 -- 172.21.15.102:0/2771587273 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4034040 con 0x7f1e080c4fb0 2024-03-18T16:21:19.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.025+0000 7f1dffe00640 1 -- 172.21.15.102:0/2771587273 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df4039d10 con 0x7f1e080c4fb0 2024-03-18T16:21:19.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.025+0000 7f1dffe00640 1 -- 172.21.15.102:0/2771587273 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4042610 con 0x7f1e080c4fb0 2024-03-18T16:21:19.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.025+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e081f00a0 con 0x7f1e080c4fb0 2024-03-18T16:21:19.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.025+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e08159620 con 0x7f1e080c4fb0 2024-03-18T16:21:19.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.025+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e08074fd0 con 0x7f1e080c4fb0 2024-03-18T16:21:19.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.044+0000 7f1dffe00640 1 -- 172.21.15.102:0/2771587273 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1df4037070 con 0x7f1e080c4fb0 2024-03-18T16:21:19.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.045+0000 7f1dffe00640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1de806bc50 0x7f1de806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:19.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.046+0000 7f1dffe00640 1 -- 172.21.15.102:0/2771587273 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f1df4081e80 con 0x7f1e080c4fb0 2024-03-18T16:21:19.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.046+0000 7f1e07e00640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1de806bc50 0x7f1de806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:19.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.047+0000 7f1e07e00640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1de806bc50 0x7f1de806e110 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009670 tx=0x7f1df8006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:19.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.292+0000 7f1dffe00640 1 -- 172.21.15.102:0/2771587273 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1e081f00a0 con 0x7f1e080c4fb0 2024-03-18T16:21:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:19.497+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f1e0813eab0 con 0x7f1e080c4fb0 2024-03-18T16:21:20.117 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:21:20.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:21:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.202+0000 7f1dffe00640 1 -- 172.21.15.102:0/2771587273 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f1e081e31c0 con 0x7f1e080c4fb0 2024-03-18T16:21:20.204 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1de806bc50 msgr2=0x7f1de806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1de806bc50 0x7f1de806e110 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009670 tx=0x7f1df8006bf0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e080c4fb0 msgr2=0x7f1e081f0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e080c4fb0 0x7f1e081f0840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1df40098c0 tx=0x7f1df40046c0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 shutdown_connections 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1de806bc50 0x7f1de806e110 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1e08141700 0x7f1e0813d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e080c4fb0 0x7f1e081f0840 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 --2- 172.21.15.102:0/2771587273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e0806ba70 0x7f1e081e28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 >> 172.21.15.102:0/2771587273 conn(0x7f1e080c83a0 msgr2=0x7f1e0815bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 shutdown_connections 2024-03-18T16:21:20.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.206+0000 7f1e0ea00640 1 -- 172.21.15.102:0/2771587273 wait complete. 2024-03-18T16:21:20.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-18T16:21:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.421+0000 7f0a94600640 1 Processor -- start 2024-03-18T16:21:20.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.421+0000 7f0a94600640 1 -- start start 2024-03-18T16:21:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.423+0000 7f0a94600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 0x7f0a8c140010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.423+0000 7f0a94600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0a8c071cc0 0x7f0a8c072120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.423+0000 7f0a94600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0a8c06f250 0x7f0a8c06f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.423+0000 7f0a94600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0a8c0c5160 con 0x7f0a8c071cc0 2024-03-18T16:21:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.423+0000 7f0a94600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0a8c1409b0 con 0x7f0a8c06f250 2024-03-18T16:21:20.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.423+0000 7f0a94600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c0c2da0 con 0x7f0a8c13fc30 2024-03-18T16:21:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.424+0000 7f0a92600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0a8c06f250 0x7f0a8c06f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.424+0000 7f0a91c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 0x7f0a8c140010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.424+0000 7f0a91200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0a8c071cc0 0x7f0a8c072120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.424+0000 7f0a92600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0a8c06f250 0x7f0a8c06f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42250/0 (socket says 172.21.15.102:42250) 2024-03-18T16:21:20.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.424+0000 7f0a92600640 1 -- 172.21.15.102:0/2476632908 learned_addr learned my addr 172.21.15.102:0/2476632908 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:20.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.427+0000 7f0a91200640 1 -- 172.21.15.102:0/2476632908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 msgr2=0x7f0a8c140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:20.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.427+0000 7f0a91200640 1 --2- 172.21.15.102:0/2476632908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 0x7f0a8c140010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.427+0000 7f0a91200640 1 -- 172.21.15.102:0/2476632908 >> v2:172.21.15.154:3300/0 conn(0x7f0a8c06f250 msgr2=0x7f0a8c06f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:20.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.427+0000 7f0a91200640 1 --2- 172.21.15.102:0/2476632908 >> v2:172.21.15.154:3300/0 conn(0x7f0a8c06f250 0x7f0a8c06f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.427+0000 7f0a91200640 1 -- 172.21.15.102:0/2476632908 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c0c53d0 con 0x7f0a8c071cc0 2024-03-18T16:21:20.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.430+0000 7f0a91200640 1 --2- 172.21.15.102:0/2476632908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c071cc0 0x7f0a8c072120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0716d0 tx=0x7f0a7802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a3ed3e43cd0b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:20.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.432+0000 7f0a8be00640 1 -- 172.21.15.102:0/2476632908 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78038070 con 0x7f0a8c071cc0 2024-03-18T16:21:20.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.434+0000 7f0a8be00640 1 -- 172.21.15.102:0/2476632908 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a78031700 con 0x7f0a8c071cc0 2024-03-18T16:21:20.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.434+0000 7f0a8be00640 1 -- 172.21.15.102:0/2476632908 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78031ad0 con 0x7f0a8c071cc0 2024-03-18T16:21:20.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.435+0000 7f0a94600640 1 -- 172.21.15.102:0/2476632908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c071cc0 msgr2=0x7f0a8c072120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:20.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.435+0000 7f0a94600640 1 --2- 172.21.15.102:0/2476632908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c071cc0 0x7f0a8c072120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0716d0 tx=0x7f0a7802b7a0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.435+0000 7f0a94600640 1 -- 172.21.15.102:0/2476632908 shutdown_connections 2024-03-18T16:21:20.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.435+0000 7f0a94600640 1 --2- 172.21.15.102:0/2476632908 >> v2:172.21.15.154:3300/0 conn(0x7f0a8c06f250 0x7f0a8c06f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.435+0000 7f0a94600640 1 --2- 172.21.15.102:0/2476632908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c071cc0 0x7f0a8c072120 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.435+0000 7f0a94600640 1 --2- 172.21.15.102:0/2476632908 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 0x7f0a8c140010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.435+0000 7f0a94600640 1 -- 172.21.15.102:0/2476632908 >> 172.21.15.102:0/2476632908 conn(0x7f0a8c0c83a0 msgr2=0x7f0a8c0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:20.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.435+0000 7f0a94600640 1 -- 172.21.15.102:0/2476632908 shutdown_connections 2024-03-18T16:21:20.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.435+0000 7f0a94600640 1 -- 172.21.15.102:0/2476632908 wait complete. 2024-03-18T16:21:20.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.437+0000 7f0a94600640 1 Processor -- start 2024-03-18T16:21:20.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.437+0000 7f0a94600640 1 -- start start 2024-03-18T16:21:20.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.438+0000 7f0a94600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c06f250 0x7f0a8c1f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:20.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.438+0000 7f0a94600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a8c071cc0 0x7f0a8c1f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:20.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.438+0000 7f0a94600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 0x7f0a8c1557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:20.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.438+0000 7f0a94600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c0c5c70 con 0x7f0a8c06f250 2024-03-18T16:21:20.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.438+0000 7f0a94600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c0c5990 con 0x7f0a8c071cc0 2024-03-18T16:21:20.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.438+0000 7f0a94600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0a8c0c3280 con 0x7f0a8c13fc30 2024-03-18T16:21:20.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.439+0000 7f0a91200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a8c071cc0 0x7f0a8c1f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:20.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.439+0000 7f0a92600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 0x7f0a8c1557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:20.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.439+0000 7f0a91c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c06f250 0x7f0a8c1f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:20.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.439+0000 7f0a91200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a8c071cc0 0x7f0a8c1f3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42260/0 (socket says 172.21.15.102:42260) 2024-03-18T16:21:20.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.439+0000 7f0a91200640 1 -- 172.21.15.102:0/207239781 learned_addr learned my addr 172.21.15.102:0/207239781 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:20.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.443+0000 7f0a91c00640 1 -- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 msgr2=0x7f0a8c1557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:20.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.443+0000 7f0a91c00640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 0x7f0a8c1557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.443+0000 7f0a91c00640 1 -- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a8c071cc0 msgr2=0x7f0a8c1f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:20.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.443+0000 7f0a91c00640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a8c071cc0 0x7f0a8c1f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:20.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.443+0000 7f0a91c00640 1 -- 172.21.15.102:0/207239781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c13b870 con 0x7f0a8c06f250 2024-03-18T16:21:20.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.446+0000 7f0a91c00640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c06f250 0x7f0a8c1f0e90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000ebf0 tx=0x7f0a8000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.450+0000 7f0a89600640 1 -- 172.21.15.102:0/207239781 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80018020 con 0x7f0a8c06f250 2024-03-18T16:21:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.450+0000 7f0a89600640 1 -- 172.21.15.102:0/207239781 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8000cda0 con 0x7f0a8c06f250 2024-03-18T16:21:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.450+0000 7f0a89600640 1 -- 172.21.15.102:0/207239781 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a800146c0 con 0x7f0a8c06f250 2024-03-18T16:21:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.450+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c13bb30 con 0x7f0a8c06f250 2024-03-18T16:21:20.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.451+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c1f03b0 con 0x7f0a8c06f250 2024-03-18T16:21:20.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.452+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a8c074fd0 con 0x7f0a8c06f250 2024-03-18T16:21:20.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.477+0000 7f0a89600640 1 -- 172.21.15.102:0/207239781 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0a80005d10 con 0x7f0a8c06f250 2024-03-18T16:21:20.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.478+0000 7f0a89600640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a6c06bc50 0x7f0a6c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:20.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.479+0000 7f0a91200640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a6c06bc50 0x7f0a6c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:20.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.480+0000 7f0a91200640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a6c06bc50 0x7f0a6c06e110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0735c0 tx=0x7f0a78008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:20.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.800+0000 7f0a89600640 1 -- 172.21.15.102:0/207239781 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f0a80058010 con 0x7f0a8c06f250 2024-03-18T16:21:20.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:20.800+0000 7f0a89600640 1 -- 172.21.15.102:0/207239781 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0a80022030 con 0x7f0a8c06f250 2024-03-18T16:21:21.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.014+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f0a8c151600 con 0x7f0a8c06f250 2024-03-18T16:21:21.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.306+0000 7f0a89600640 1 -- 172.21.15.102:0/207239781 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7f0a80010040 con 0x7f0a8c06f250 2024-03-18T16:21:21.308 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-03-18T16:21:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a6c06bc50 msgr2=0x7f0a6c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a6c06bc50 0x7f0a6c06e110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0735c0 tx=0x7f0a78008030 comp rx=0 tx=0).stop 2024-03-18T16:21:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c06f250 msgr2=0x7f0a8c1f0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c06f250 0x7f0a8c1f0e90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000ebf0 tx=0x7f0a8000c490 comp rx=0 tx=0).stop 2024-03-18T16:21:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 shutdown_connections 2024-03-18T16:21:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a6c06bc50 0x7f0a6c06e110 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a8c13fc30 0x7f0a8c1557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a8c071cc0 0x7f0a8c1f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:21.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 --2- 172.21.15.102:0/207239781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a8c06f250 0x7f0a8c1f0e90 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:21.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 >> 172.21.15.102:0/207239781 conn(0x7f0a8c0c83a0 msgr2=0x7f0a8c0c2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:21.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 shutdown_connections 2024-03-18T16:21:21.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.310+0000 7f0a94600640 1 -- 172.21.15.102:0/207239781 wait complete. 2024-03-18T16:21:21.333 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T16:21:21.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.551+0000 7f31a6400640 1 Processor -- start 2024-03-18T16:21:21.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.552+0000 7f31a6400640 1 -- start start 2024-03-18T16:21:21.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.553+0000 7f31a6400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a0141700 0x7f31a00723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:21.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.553+0000 7f31a6400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31a0158390 con 0x7f31a006be90 2024-03-18T16:21:21.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.553+0000 7f31a6400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31a0158670 con 0x7f31a006ba70 2024-03-18T16:21:21.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.553+0000 7f31a6400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f31a0141ae0 con 0x7f31a0141700 2024-03-18T16:21:21.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.553+0000 7f319ea00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f31a006ba70 0x7f31a00c4fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:54858/0 (socket says 172.21.15.102:54858) 2024-03-18T16:21:21.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.554+0000 7f319ea00640 1 -- 172.21.15.102:0/728048836 learned_addr learned my addr 172.21.15.102:0/728048836 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:21.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.554+0000 7f319f400640 1 --2- 172.21.15.102:0/728048836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a0141700 0x7f31a00723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:21.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.556+0000 7f319f400640 1 -- 172.21.15.102:0/728048836 >> v1:172.21.15.154:6789/0 conn(0x7f31a006ba70 legacy=0x7f31a00c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:21.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.557+0000 7f319f400640 1 -- 172.21.15.102:0/728048836 >> v1:172.21.15.102:6789/0 conn(0x7f31a006be90 legacy=0x7f31a015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:21.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.557+0000 7f319f400640 1 -- 172.21.15.102:0/728048836 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a0158950 con 0x7f31a0141700 2024-03-18T16:21:21.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.559+0000 7f319f400640 1 --2- 172.21.15.102:0/728048836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a0141700 0x7f31a00723b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f31900097d0 tx=0x7f319002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc470c0367b263f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:21.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.562+0000 7f319e000640 1 -- 172.21.15.102:0/728048836 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190035020 con 0x7f31a0141700 2024-03-18T16:21:21.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.562+0000 7f319e000640 1 -- 172.21.15.102:0/728048836 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3190031890 con 0x7f31a0141700 2024-03-18T16:21:21.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.562+0000 7f319e000640 1 -- 172.21.15.102:0/728048836 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190031c60 con 0x7f31a0141700 2024-03-18T16:21:21.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.563+0000 7f31a6400640 1 -- 172.21.15.102:0/728048836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a0141700 msgr2=0x7f31a00723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:21.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.563+0000 7f31a6400640 1 --2- 172.21.15.102:0/728048836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a0141700 0x7f31a00723b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f31900097d0 tx=0x7f319002b930 comp rx=0 tx=0).stop 2024-03-18T16:21:21.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.563+0000 7f31a6400640 1 -- 172.21.15.102:0/728048836 shutdown_connections 2024-03-18T16:21:21.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.563+0000 7f31a6400640 1 --2- 172.21.15.102:0/728048836 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a0141700 0x7f31a00723b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:21.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.563+0000 7f31a6400640 1 -- 172.21.15.102:0/728048836 >> 172.21.15.102:0/728048836 conn(0x7f31a00c83a0 msgr2=0x7f31a00c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:21.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.563+0000 7f31a6400640 1 -- 172.21.15.102:0/728048836 shutdown_connections 2024-03-18T16:21:21.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.563+0000 7f31a6400640 1 -- 172.21.15.102:0/728048836 wait complete. 2024-03-18T16:21:21.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.565+0000 7f31a6400640 1 Processor -- start 2024-03-18T16:21:21.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.565+0000 7f31a6400640 1 -- start start 2024-03-18T16:21:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.566+0000 7f31a6400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a006ba70 0x7f31a015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.566+0000 7f31a6400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a006be90 0x7f31a0155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.566+0000 7f31a6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31a0141700 0x7f31a0155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.566+0000 7f31a6400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0158f10 con 0x7f31a0141700 2024-03-18T16:21:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.566+0000 7f31a6400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31a006c300 con 0x7f31a006be90 2024-03-18T16:21:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.566+0000 7f31a6400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f31a00c56c0 con 0x7f31a006ba70 2024-03-18T16:21:21.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.567+0000 7f319f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a006ba70 0x7f31a015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:21.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.567+0000 7f319ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a006be90 0x7f31a0155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:21.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.567+0000 7f319ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a006be90 0x7f31a0155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40648/0 (socket says 172.21.15.102:40648) 2024-03-18T16:21:21.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.567+0000 7f319ea00640 1 -- 172.21.15.102:0/281087072 learned_addr learned my addr 172.21.15.102:0/281087072 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:21.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.570+0000 7f319f400640 1 -- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a006be90 msgr2=0x7f31a0155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:21.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.570+0000 7f319f400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a006be90 0x7f31a0155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:21.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.570+0000 7f319f400640 1 -- 172.21.15.102:0/281087072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31a0141700 msgr2=0x7f31a0155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:21.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.570+0000 7f319f400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31a0141700 0x7f31a0155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:21.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.570+0000 7f319f400640 1 -- 172.21.15.102:0/281087072 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a01f05f0 con 0x7f31a006ba70 2024-03-18T16:21:21.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.573+0000 7f319f400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a006ba70 0x7f31a015ac70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f31a0073630 tx=0x7f31900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:21.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.577+0000 7f3197400640 1 -- 172.21.15.102:0/281087072 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190044400 con 0x7f31a006ba70 2024-03-18T16:21:21.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.577+0000 7f3197400640 1 -- 172.21.15.102:0/281087072 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31900445e0 con 0x7f31a006ba70 2024-03-18T16:21:21.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.577+0000 7f3197400640 1 -- 172.21.15.102:0/281087072 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31900449d0 con 0x7f31a006ba70 2024-03-18T16:21:21.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.577+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a01ecbe0 con 0x7f31a006ba70 2024-03-18T16:21:21.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.577+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a00c2170 con 0x7f31a006ba70 2024-03-18T16:21:21.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.578+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a00c8ab0 con 0x7f31a006ba70 2024-03-18T16:21:21.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.598+0000 7f3197400640 1 -- 172.21.15.102:0/281087072 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3190038020 con 0x7f31a006ba70 2024-03-18T16:21:21.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.599+0000 7f3197400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f318006bc20 0x7f318006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:21.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.600+0000 7f319ea00640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f318006bc20 0x7f318006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:21.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.600+0000 7f3197400640 1 -- 172.21.15.102:0/281087072 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f3190031840 con 0x7f31a006ba70 2024-03-18T16:21:21.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.601+0000 7f319ea00640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f318006bc20 0x7f318006e0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f31a01e9d90 tx=0x7f318c007f10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:21.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:21.846+0000 7f3197400640 1 -- 172.21.15.102:0/281087072 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f31a00c8ab0 con 0x7f31a006ba70 2024-03-18T16:21:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:22.108+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f31a0074fd0 con 0x7f31a006ba70 2024-03-18T16:21:23.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.537+0000 7f3197400640 1 -- 172.21.15.102:0/281087072 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f319002d040 con 0x7f31a006ba70 2024-03-18T16:21:23.540 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f318006bc20 msgr2=0x7f318006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f318006bc20 0x7f318006e0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f31a01e9d90 tx=0x7f318c007f10 comp rx=0 tx=0).stop 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a006ba70 msgr2=0x7f31a015ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a006ba70 0x7f31a015ac70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f31a0073630 tx=0x7f31900046c0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 shutdown_connections 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f318006bc20 0x7f318006e0e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31a0141700 0x7f31a0155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a006be90 0x7f31a0155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 --2- 172.21.15.102:0/281087072 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31a006ba70 0x7f31a015ac70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 >> 172.21.15.102:0/281087072 conn(0x7f31a00c83a0 msgr2=0x7f31a013f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:23.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 shutdown_connections 2024-03-18T16:21:23.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.541+0000 7f31a6400640 1 -- 172.21.15.102:0/281087072 wait complete. 2024-03-18T16:21:23.562 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-18T16:21:23.563 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-18T16:21:23.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.768+0000 7f338f600640 1 Processor -- start 2024-03-18T16:21:23.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.768+0000 7f338f600640 1 -- start start 2024-03-18T16:21:23.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.770+0000 7f338f600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33880c9ea0 0x7f338815de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:23.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.770+0000 7f338f600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33880cd1e0 0x7f338815e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:23.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.770+0000 7f338f600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3388154c90 con 0x7f338815da60 2024-03-18T16:21:23.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.770+0000 7f338f600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f33880ca670 con 0x7f33880c9ea0 2024-03-18T16:21:23.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.770+0000 7f338f600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f33880cc640 con 0x7f33880cd1e0 2024-03-18T16:21:23.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.770+0000 7f338d600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33880cd1e0 0x7f338815e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:23.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.771+0000 7f338d600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33880cd1e0 0x7f338815e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41480/0 (socket says 172.21.15.102:41480) 2024-03-18T16:21:23.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.771+0000 7f338d600640 1 -- 172.21.15.102:0/772597555 learned_addr learned my addr 172.21.15.102:0/772597555 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:23.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.771+0000 7f338cc00640 1 --1- 172.21.15.102:0/772597555 >> v1:172.21.15.102:6789/0 conn(0x7f338815da60 0x7f33880ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42390/0 (socket says 172.21.15.102:42390) 2024-03-18T16:21:23.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.771+0000 7f3387e00640 1 --2- 172.21.15.102:0/772597555 >> v2:172.21.15.102:3300/0 conn(0x7f33880c9ea0 0x7f338815de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:23.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.773+0000 7f338d600640 1 -- 172.21.15.102:0/772597555 >> v2:172.21.15.102:3300/0 conn(0x7f33880c9ea0 msgr2=0x7f338815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:23.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.773+0000 7f338d600640 1 --2- 172.21.15.102:0/772597555 >> v2:172.21.15.102:3300/0 conn(0x7f33880c9ea0 0x7f338815de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.773+0000 7f338d600640 1 -- 172.21.15.102:0/772597555 >> v1:172.21.15.102:6789/0 conn(0x7f338815da60 legacy=0x7f33880ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:23.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.773+0000 7f338d600640 1 -- 172.21.15.102:0/772597555 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338806d8d0 con 0x7f33880cd1e0 2024-03-18T16:21:23.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.773+0000 7f3387e00640 1 --2- 172.21.15.102:0/772597555 >> v2:172.21.15.102:3300/0 conn(0x7f33880c9ea0 0x7f338815de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:21:23.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.775+0000 7f338d600640 1 --2- 172.21.15.102:0/772597555 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33880cd1e0 0x7f338815e380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3380009e50 tx=0x7f338002bfe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e986d84201a80ac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:23.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.779+0000 7f3387400640 1 -- 172.21.15.102:0/772597555 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338002cb80 con 0x7f33880cd1e0 2024-03-18T16:21:23.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.779+0000 7f3387400640 1 -- 172.21.15.102:0/772597555 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f338002cd20 con 0x7f33880cd1e0 2024-03-18T16:21:23.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.779+0000 7f3387400640 1 -- 172.21.15.102:0/772597555 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380034610 con 0x7f33880cd1e0 2024-03-18T16:21:23.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.779+0000 7f338f600640 1 -- 172.21.15.102:0/772597555 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33880cd1e0 msgr2=0x7f338815e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:23.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.779+0000 7f338f600640 1 --2- 172.21.15.102:0/772597555 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33880cd1e0 0x7f338815e380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3380009e50 tx=0x7f338002bfe0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.779+0000 7f338f600640 1 -- 172.21.15.102:0/772597555 shutdown_connections 2024-03-18T16:21:23.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.779+0000 7f338f600640 1 --2- 172.21.15.102:0/772597555 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33880cd1e0 0x7f338815e380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.779+0000 7f338f600640 1 --2- 172.21.15.102:0/772597555 >> v2:172.21.15.102:3300/0 conn(0x7f33880c9ea0 0x7f338815de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.779+0000 7f338f600640 1 -- 172.21.15.102:0/772597555 >> 172.21.15.102:0/772597555 conn(0x7f33880b1840 msgr2=0x7f33880b1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:23.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.780+0000 7f338f600640 1 -- 172.21.15.102:0/772597555 shutdown_connections 2024-03-18T16:21:23.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.780+0000 7f338f600640 1 -- 172.21.15.102:0/772597555 wait complete. 2024-03-18T16:21:23.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.781+0000 7f338f600640 1 Processor -- start 2024-03-18T16:21:23.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.782+0000 7f338f600640 1 -- start start 2024-03-18T16:21:23.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.782+0000 7f338f600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33880c9ea0 0x7f33881e3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:23.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.782+0000 7f338f600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33880cd1e0 0x7f33881f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:23.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.782+0000 7f338f600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f338815da60 0x7f33880c5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:23.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.782+0000 7f338f600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f33880c9970 con 0x7f33880cd1e0 2024-03-18T16:21:23.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.782+0000 7f338f600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f33880caba0 con 0x7f33880c9ea0 2024-03-18T16:21:23.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.782+0000 7f338f600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f338815e8c0 con 0x7f338815da60 2024-03-18T16:21:23.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.783+0000 7f338d600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f338815da60 0x7f33880c5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:23.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.783+0000 7f338cc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33880c9ea0 0x7f33881e3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:23.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.783+0000 7f3387e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33880cd1e0 0x7f33881f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:23.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.784+0000 7f338cc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33880c9ea0 0x7f33881e3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40652/0 (socket says 172.21.15.102:40652) 2024-03-18T16:21:23.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.784+0000 7f338cc00640 1 -- 172.21.15.102:0/394348116 learned_addr learned my addr 172.21.15.102:0/394348116 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:23.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.787+0000 7f338cc00640 1 -- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f338815da60 msgr2=0x7f33880c5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:23.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.787+0000 7f338cc00640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f338815da60 0x7f33880c5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.787+0000 7f338cc00640 1 -- 172.21.15.102:0/394348116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33880cd1e0 msgr2=0x7f33881f3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:23.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.787+0000 7f338cc00640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33880cd1e0 0x7f33881f3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:23.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.787+0000 7f338cc00640 1 -- 172.21.15.102:0/394348116 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3388157730 con 0x7f33880c9ea0 2024-03-18T16:21:23.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.790+0000 7f338cc00640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33880c9ea0 0x7f33881e3980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f337800c400 tx=0x7f337800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:23.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.793+0000 7f3384c00640 1 -- 172.21.15.102:0/394348116 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33780074e0 con 0x7f33880c9ea0 2024-03-18T16:21:23.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.793+0000 7f3384c00640 1 -- 172.21.15.102:0/394348116 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3378007680 con 0x7f33880c9ea0 2024-03-18T16:21:23.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.793+0000 7f3384c00640 1 -- 172.21.15.102:0/394348116 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3378004a00 con 0x7f33880c9ea0 2024-03-18T16:21:23.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.793+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3388157a10 con 0x7f33880c9ea0 2024-03-18T16:21:23.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.793+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33880c5110 con 0x7f33880c9ea0 2024-03-18T16:21:23.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.795+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3388074fd0 con 0x7f33880c9ea0 2024-03-18T16:21:23.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.816+0000 7f3384c00640 1 -- 172.21.15.102:0/394348116 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f337801a020 con 0x7f33880c9ea0 2024-03-18T16:21:23.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.817+0000 7f3384c00640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f336406bc50 0x7f336406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:23.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.817+0000 7f3384c00640 1 -- 172.21.15.102:0/394348116 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f337808a770 con 0x7f33880c9ea0 2024-03-18T16:21:23.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.817+0000 7f3387e00640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f336406bc50 0x7f336406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:23.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:23.818+0000 7f3387e00640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f336406bc50 0x7f336406e110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3388156f70 tx=0x7f3370008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:24.069+0000 7f3384c00640 1 -- 172.21.15.102:0/394348116 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3378021070 con 0x7f33880c9ea0 2024-03-18T16:21:24.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:24.338+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f33880c6a00 con 0x7f33880c9ea0 2024-03-18T16:21:25.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.382+0000 7f3384c00640 1 -- 172.21.15.102:0/394348116 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v22) ==== 157+0+0 (secure 0 0 0) 0x7f3378058d80 con 0x7f33880c9ea0 2024-03-18T16:21:25.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f336406bc50 msgr2=0x7f336406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:25.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f336406bc50 0x7f336406e110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3388156f70 tx=0x7f3370008040 comp rx=0 tx=0).stop 2024-03-18T16:21:25.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33880c9ea0 msgr2=0x7f33881e3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:25.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33880c9ea0 0x7f33881e3980 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f337800c400 tx=0x7f337800c8c0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 shutdown_connections 2024-03-18T16:21:25.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f336406bc50 0x7f336406e110 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f338815da60 0x7f33880c5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33880cd1e0 0x7f33881f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 --2- 172.21.15.102:0/394348116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33880c9ea0 0x7f33881e3980 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 >> 172.21.15.102:0/394348116 conn(0x7f33880b1840 msgr2=0x7f33881e6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:25.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 shutdown_connections 2024-03-18T16:21:25.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.387+0000 7f338f600640 1 -- 172.21.15.102:0/394348116 wait complete. 2024-03-18T16:21:25.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T16:21:25.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.622+0000 7f85b5400640 1 Processor -- start 2024-03-18T16:21:25.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.622+0000 7f85b5400640 1 -- start start 2024-03-18T16:21:25.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.624+0000 7f85b5400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f85b0152410 0x7f85b0071c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:25.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.624+0000 7f85b5400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f85b015b8b0 0x7f85b00c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:25.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.624+0000 7f85b5400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85b015c220 con 0x7f85b0158e70 2024-03-18T16:21:25.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.624+0000 7f85b5400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f85b01583d0 con 0x7f85b015b8b0 2024-03-18T16:21:25.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.624+0000 7f85b5400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f85b0159280 con 0x7f85b0152410 2024-03-18T16:21:25.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.625+0000 7f85ae800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f85b0152410 0x7f85b0071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:25.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.625+0000 7f85a5e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f85b0158e70 0x7f85b006bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:54872/0 (socket says 172.21.15.102:54872) 2024-03-18T16:21:25.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.625+0000 7f85a5e00640 1 -- 172.21.15.102:0/1232128010 learned_addr learned my addr 172.21.15.102:0/1232128010 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:25.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.628+0000 7f85ae800640 1 -- 172.21.15.102:0/1232128010 >> v2:172.21.15.102:3300/0 conn(0x7f85b015b8b0 msgr2=0x7f85b00c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:25.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.628+0000 7f85ae800640 1 --2- 172.21.15.102:0/1232128010 >> v2:172.21.15.102:3300/0 conn(0x7f85b015b8b0 0x7f85b00c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.628+0000 7f85ae800640 1 -- 172.21.15.102:0/1232128010 >> v1:172.21.15.154:6789/0 conn(0x7f85b0158e70 legacy=0x7f85b006bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:25.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.628+0000 7f85ae800640 1 -- 172.21.15.102:0/1232128010 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b015c4c0 con 0x7f85b0152410 2024-03-18T16:21:25.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.630+0000 7f85ae800640 1 --2- 172.21.15.102:0/1232128010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 0x7f85b0071c80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f85b00c5da0 tx=0x7f85a002bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=557e64a8e40cf225 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:25.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.636+0000 7f85ade00640 1 -- 172.21.15.102:0/1232128010 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0039070 con 0x7f85b0152410 2024-03-18T16:21:25.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.636+0000 7f85ade00640 1 -- 172.21.15.102:0/1232128010 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a002cc70 con 0x7f85b0152410 2024-03-18T16:21:25.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.636+0000 7f85ade00640 1 -- 172.21.15.102:0/1232128010 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0034660 con 0x7f85b0152410 2024-03-18T16:21:25.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.637+0000 7f85b5400640 1 -- 172.21.15.102:0/1232128010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 msgr2=0x7f85b0071c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:25.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.637+0000 7f85b5400640 1 --2- 172.21.15.102:0/1232128010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 0x7f85b0071c80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f85b00c5da0 tx=0x7f85a002bec0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.637+0000 7f85b5400640 1 -- 172.21.15.102:0/1232128010 shutdown_connections 2024-03-18T16:21:25.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.637+0000 7f85b5400640 1 --2- 172.21.15.102:0/1232128010 >> v2:172.21.15.102:3300/0 conn(0x7f85b015b8b0 0x7f85b00c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.637+0000 7f85b5400640 1 --2- 172.21.15.102:0/1232128010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 0x7f85b0071c80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.637+0000 7f85b5400640 1 -- 172.21.15.102:0/1232128010 >> 172.21.15.102:0/1232128010 conn(0x7f85b006f450 msgr2=0x7f85b006f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:25.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.637+0000 7f85b5400640 1 -- 172.21.15.102:0/1232128010 shutdown_connections 2024-03-18T16:21:25.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.637+0000 7f85b5400640 1 -- 172.21.15.102:0/1232128010 wait complete. 2024-03-18T16:21:25.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.639+0000 7f85b5400640 1 Processor -- start 2024-03-18T16:21:25.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.639+0000 7f85b5400640 1 -- start start 2024-03-18T16:21:25.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.640+0000 7f85b5400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 0x7f85b013d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.640+0000 7f85b5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85b0158e70 0x7f85b00c3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.640+0000 7f85b5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85b015b8b0 0x7f85b01464e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.640+0000 7f85b5400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f85b015c7a0 con 0x7f85b015b8b0 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.640+0000 7f85b5400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85b0158950 con 0x7f85b0152410 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.640+0000 7f85b5400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85b01527f0 con 0x7f85b0158e70 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.641+0000 7f85ae800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 0x7f85b013d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.641+0000 7f85ae800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 0x7f85b013d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40678/0 (socket says 172.21.15.102:40678) 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.641+0000 7f85ae800640 1 -- 172.21.15.102:0/2062269655 learned_addr learned my addr 172.21.15.102:0/2062269655 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.641+0000 7f85a5e00640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85b0158e70 0x7f85b00c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.644+0000 7f85ae800640 1 -- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85b0158e70 msgr2=0x7f85b00c3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.644+0000 7f85ae800640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85b0158e70 0x7f85b00c3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.644+0000 7f85ae800640 1 -- 172.21.15.102:0/2062269655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85b015b8b0 msgr2=0x7f85b01464e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.644+0000 7f85ae800640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85b015b8b0 0x7f85b01464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:25.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.644+0000 7f85ae800640 1 -- 172.21.15.102:0/2062269655 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b01f3ad0 con 0x7f85b0152410 2024-03-18T16:21:25.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.646+0000 7f85ae800640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 0x7f85b013d850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009e80 tx=0x7f85a0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:25.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.650+0000 7f85a7400640 1 -- 172.21.15.102:0/2062269655 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0044020 con 0x7f85b0152410 2024-03-18T16:21:25.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.650+0000 7f85a7400640 1 -- 172.21.15.102:0/2062269655 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a003f670 con 0x7f85b0152410 2024-03-18T16:21:25.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.650+0000 7f85a7400640 1 -- 172.21.15.102:0/2062269655 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0039040 con 0x7f85b0152410 2024-03-18T16:21:25.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.650+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85b01f3d70 con 0x7f85b0152410 2024-03-18T16:21:25.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.651+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85b01f42f0 con 0x7f85b0152410 2024-03-18T16:21:25.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.652+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85b0074fd0 con 0x7f85b0152410 2024-03-18T16:21:25.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.913+0000 7f85a7400640 1 -- 172.21.15.102:0/2062269655 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f85a0047020 con 0x7f85b0152410 2024-03-18T16:21:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.915+0000 7f85a7400640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f858c06bc50 0x7f858c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.916+0000 7f85a7400640 1 -- 172.21.15.102:0/2062269655 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6007+0+0 (secure 0 0 0) 0x7f85a00adc60 con 0x7f85b0152410 2024-03-18T16:21:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.916+0000 7f85a5e00640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f858c06bc50 0x7f858c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:25.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.917+0000 7f85a5e00640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f858c06bc50 0x7f858c06e110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f85b0148a00 tx=0x7f8598009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:25.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:25.924+0000 7f85a7400640 1 -- 172.21.15.102:0/2062269655 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f85a004e070 con 0x7f85b0152410 2024-03-18T16:21:26.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:26.140+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f85b00b2210 con 0x7f85b0152410 2024-03-18T16:21:27.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.911+0000 7f85a7400640 1 -- 172.21.15.102:0/2062269655 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v24) ==== 228+0+0 (secure 0 0 0) 0x7f85a002ce20 con 0x7f85b0152410 2024-03-18T16:21:27.913 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data_ec' created 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.915+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f858c06bc50 msgr2=0x7f858c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.915+0000 7f85b5400640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f858c06bc50 0x7f858c06e110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f85b0148a00 tx=0x7f8598009040 comp rx=0 tx=0).stop 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.915+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 msgr2=0x7f85b013d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.915+0000 7f85b5400640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 0x7f85b013d850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009e80 tx=0x7f85a0004a40 comp rx=0 tx=0).stop 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.916+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 shutdown_connections 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.916+0000 7f85b5400640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f858c06bc50 0x7f858c06e110 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.916+0000 7f85b5400640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85b015b8b0 0x7f85b01464e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.916+0000 7f85b5400640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85b0158e70 0x7f85b00c3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.916+0000 7f85b5400640 1 --2- 172.21.15.102:0/2062269655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85b0152410 0x7f85b013d850 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.916+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 >> 172.21.15.102:0/2062269655 conn(0x7f85b006f450 msgr2=0x7f85b0071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.916+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 shutdown_connections 2024-03-18T16:21:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:27.916+0000 7f85b5400640 1 -- 172.21.15.102:0/2062269655 wait complete. 2024-03-18T16:21:27.941 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-18T16:21:28.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.134+0000 7fabe6c00640 1 Processor -- start 2024-03-18T16:21:28.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.134+0000 7fabe6c00640 1 -- start start 2024-03-18T16:21:28.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.136+0000 7fabe6c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe00c7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:28.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.136+0000 7fabe6c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fabe00c7bd0 0x7fabe015bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:28.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.136+0000 7fabe6c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabe01584d0 con 0x7fabe015c280 2024-03-18T16:21:28.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.136+0000 7fabe6c00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fabe00c3b90 con 0x7fabe00c7bd0 2024-03-18T16:21:28.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.136+0000 7fabe6c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fabe01587b0 con 0x7fabe0152490 2024-03-18T16:21:28.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.136+0000 7fabe4e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fabe00c7bd0 0x7fabe015bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:28.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.137+0000 7fabe6200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fabe015c280 0x7fabe00c68c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42398/0 (socket says 172.21.15.102:42398) 2024-03-18T16:21:28.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.137+0000 7fabe6200640 1 -- 172.21.15.102:0/1966930669 learned_addr learned my addr 172.21.15.102:0/1966930669 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:28.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.137+0000 7fabe5800640 1 --2- 172.21.15.102:0/1966930669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe00c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:28.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.139+0000 7fabe4e00640 1 -- 172.21.15.102:0/1966930669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 msgr2=0x7fabe00c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:28.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.139+0000 7fabe4e00640 1 --2- 172.21.15.102:0/1966930669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe00c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:28.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.139+0000 7fabe4e00640 1 -- 172.21.15.102:0/1966930669 >> v1:172.21.15.102:6789/0 conn(0x7fabe015c280 legacy=0x7fabe00c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:28.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.139+0000 7fabe4e00640 1 -- 172.21.15.102:0/1966930669 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe0158a90 con 0x7fabe00c7bd0 2024-03-18T16:21:28.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.139+0000 7fabe5800640 1 --2- 172.21.15.102:0/1966930669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe00c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:28.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.142+0000 7fabe4e00640 1 --2- 172.21.15.102:0/1966930669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe00c7bd0 0x7fabe015bd40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0097a0 tx=0x7fabcc02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8b94c70b7b0b656 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:28.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.146+0000 7fabdfe00640 1 -- 172.21.15.102:0/1966930669 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc034070 con 0x7fabe00c7bd0 2024-03-18T16:21:28.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.146+0000 7fabdfe00640 1 -- 172.21.15.102:0/1966930669 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabcc030800 con 0x7fabe00c7bd0 2024-03-18T16:21:28.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.146+0000 7fabdfe00640 1 -- 172.21.15.102:0/1966930669 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc030bd0 con 0x7fabe00c7bd0 2024-03-18T16:21:28.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.147+0000 7fabe6c00640 1 -- 172.21.15.102:0/1966930669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe00c7bd0 msgr2=0x7fabe015bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:28.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.147+0000 7fabe6c00640 1 --2- 172.21.15.102:0/1966930669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe00c7bd0 0x7fabe015bd40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabcc0097a0 tx=0x7fabcc02b940 comp rx=0 tx=0).stop 2024-03-18T16:21:28.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.147+0000 7fabe6c00640 1 -- 172.21.15.102:0/1966930669 shutdown_connections 2024-03-18T16:21:28.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.147+0000 7fabe6c00640 1 --2- 172.21.15.102:0/1966930669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe00c7bd0 0x7fabe015bd40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:28.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.147+0000 7fabe6c00640 1 --2- 172.21.15.102:0/1966930669 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe00c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:28.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.147+0000 7fabe6c00640 1 -- 172.21.15.102:0/1966930669 >> 172.21.15.102:0/1966930669 conn(0x7fabe00675b0 msgr2=0x7fabe006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:28.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.147+0000 7fabe6c00640 1 -- 172.21.15.102:0/1966930669 shutdown_connections 2024-03-18T16:21:28.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.148+0000 7fabe6c00640 1 -- 172.21.15.102:0/1966930669 wait complete. 2024-03-18T16:21:28.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.150+0000 7fabe6c00640 1 Processor -- start 2024-03-18T16:21:28.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.150+0000 7fabe6c00640 1 -- start start 2024-03-18T16:21:28.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.151+0000 7fabe6c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe00c7bd0 0x7fabe01f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:28.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.151+0000 7fabe6c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:28.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.151+0000 7fabe6c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe015c280 0x7fabe0153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:28.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.151+0000 7fabe6c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fabe00c8dd0 con 0x7fabe00c7bd0 2024-03-18T16:21:28.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.151+0000 7fabe6c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fabe015b820 con 0x7fabe015c280 2024-03-18T16:21:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.151+0000 7fabe6c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fabe00c6fd0 con 0x7fabe0152490 2024-03-18T16:21:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.152+0000 7fabe4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.152+0000 7fabe6200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe015c280 0x7fabe0153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.152+0000 7fabe5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe00c7bd0 0x7fabe01f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.152+0000 7fabe6200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe015c280 0x7fabe0153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40694/0 (socket says 172.21.15.102:40694) 2024-03-18T16:21:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.152+0000 7fabe6200640 1 -- 172.21.15.102:0/227843995 learned_addr learned my addr 172.21.15.102:0/227843995 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:28.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.156+0000 7fabe4e00640 1 -- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe015c280 msgr2=0x7fabe0153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.156+0000 7fabe4e00640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe015c280 0x7fabe0153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.156+0000 7fabe4e00640 1 -- 172.21.15.102:0/227843995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe00c7bd0 msgr2=0x7fabe01f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.156+0000 7fabe4e00640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe00c7bd0 0x7fabe01f2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.156+0000 7fabe4e00640 1 -- 172.21.15.102:0/227843995 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe00cb250 con 0x7fabe0152490 2024-03-18T16:21:28.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.158+0000 7fabe4e00640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe015acc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fabcc033000 tx=0x7fabcc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.162+0000 7fabdd600640 1 -- 172.21.15.102:0/227843995 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc034040 con 0x7fabe0152490 2024-03-18T16:21:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.163+0000 7fabdd600640 1 -- 172.21.15.102:0/227843995 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabcc03ad20 con 0x7fabe0152490 2024-03-18T16:21:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.163+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe0154bc0 con 0x7fabe0152490 2024-03-18T16:21:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.163+0000 7fabdd600640 1 -- 172.21.15.102:0/227843995 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc044610 con 0x7fabe0152490 2024-03-18T16:21:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.163+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe00cb4f0 con 0x7fabe0152490 2024-03-18T16:21:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.164+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe004f540 con 0x7fabe0152490 2024-03-18T16:21:28.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.188+0000 7fabdd600640 1 -- 172.21.15.102:0/227843995 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fabcc04b060 con 0x7fabe0152490 2024-03-18T16:21:28.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.189+0000 7fabdd600640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fabc006bc50 0x7fabc006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:28.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.189+0000 7fabe5800640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fabc006bc50 0x7fabc006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:28.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.190+0000 7fabe5800640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fabc006bc50 0x7fabc006e110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009560 tx=0x7fabd4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:28.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.447+0000 7fabdd600640 1 -- 172.21.15.102:0/227843995 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7fabcc079050 con 0x7fabe0152490 2024-03-18T16:21:28.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.459+0000 7fabdd600640 1 -- 172.21.15.102:0/227843995 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fabcc03f070 con 0x7fabe0152490 2024-03-18T16:21:28.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:28.659+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fabe00c3520 con 0x7fabe0152490 2024-03-18T16:21:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.001+0000 7fabdd600640 1 -- 172.21.15.102:0/227843995 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v25) ==== 169+0+0 (secure 0 0 0) 0x7fabcc07afb0 con 0x7fabe0152490 2024-03-18T16:21:29.003 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 4 allow_ec_overwrites to true 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.004+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fabc006bc50 msgr2=0x7fabc006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.004+0000 7fabe6c00640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fabc006bc50 0x7fabc006e110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009560 tx=0x7fabd4009190 comp rx=0 tx=0).stop 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.004+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 msgr2=0x7fabe015acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.004+0000 7fabe6c00640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe015acc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fabcc033000 tx=0x7fabcc0046c0 comp rx=0 tx=0).stop 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.004+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 shutdown_connections 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.004+0000 7fabe6c00640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fabc006bc50 0x7fabc006e110 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.004+0000 7fabe6c00640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe015c280 0x7fabe0153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.004+0000 7fabe6c00640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fabe0152490 0x7fabe015acc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.004+0000 7fabe6c00640 1 --2- 172.21.15.102:0/227843995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabe00c7bd0 0x7fabe01f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.005+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 >> 172.21.15.102:0/227843995 conn(0x7fabe00675b0 msgr2=0x7fabe00c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.005+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 shutdown_connections 2024-03-18T16:21:29.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.005+0000 7fabe6c00640 1 -- 172.21.15.102:0/227843995 wait complete. 2024-03-18T16:21:29.026 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-18T16:21:29.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.230+0000 7fde91000640 1 Processor -- start 2024-03-18T16:21:29.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.230+0000 7fde91000640 1 -- start start 2024-03-18T16:21:29.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.231+0000 7fde91000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 0x7fde8c140620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:29.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.231+0000 7fde91000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8c158060 con 0x7fde8c06f590 2024-03-18T16:21:29.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.231+0000 7fde91000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8c071cc0 con 0x7fde8c06f970 2024-03-18T16:21:29.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.231+0000 7fde91000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde8c140b60 con 0x7fde8c1401b0 2024-03-18T16:21:29.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.264+0000 7fde81400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fde8c06f970 0x7fde8c157950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:54878/0 (socket says 172.21.15.102:54878) 2024-03-18T16:21:29.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.264+0000 7fde81400640 1 -- 172.21.15.102:0/1382614798 learned_addr learned my addr 172.21.15.102:0/1382614798 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:29.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.267+0000 7fde8a800640 1 --2- 172.21.15.102:0/1382614798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 0x7fde8c140620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:29.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.270+0000 7fde8a800640 1 -- 172.21.15.102:0/1382614798 >> v1:172.21.15.154:6789/0 conn(0x7fde8c06f970 legacy=0x7fde8c157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:29.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.270+0000 7fde8a800640 1 -- 172.21.15.102:0/1382614798 >> v1:172.21.15.102:6789/0 conn(0x7fde8c06f590 legacy=0x7fde8c0c7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:29.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.270+0000 7fde8a800640 1 -- 172.21.15.102:0/1382614798 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c071f60 con 0x7fde8c1401b0 2024-03-18T16:21:29.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.273+0000 7fde8a800640 1 --2- 172.21.15.102:0/1382614798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 0x7fde8c140620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde8c06c450 tx=0x7fde8402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4630665c0e5ff930 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:29.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.278+0000 7fde89400640 1 -- 172.21.15.102:0/1382614798 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84031760 con 0x7fde8c1401b0 2024-03-18T16:21:29.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.278+0000 7fde89400640 1 -- 172.21.15.102:0/1382614798 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde84037070 con 0x7fde8c1401b0 2024-03-18T16:21:29.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.278+0000 7fde89400640 1 -- 172.21.15.102:0/1382614798 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84031b20 con 0x7fde8c1401b0 2024-03-18T16:21:29.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.279+0000 7fde91000640 1 -- 172.21.15.102:0/1382614798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 msgr2=0x7fde8c140620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:29.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.279+0000 7fde91000640 1 --2- 172.21.15.102:0/1382614798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 0x7fde8c140620 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde8c06c450 tx=0x7fde8402b800 comp rx=0 tx=0).stop 2024-03-18T16:21:29.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.279+0000 7fde91000640 1 -- 172.21.15.102:0/1382614798 shutdown_connections 2024-03-18T16:21:29.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.279+0000 7fde91000640 1 --2- 172.21.15.102:0/1382614798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 0x7fde8c140620 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:29.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.279+0000 7fde91000640 1 -- 172.21.15.102:0/1382614798 >> 172.21.15.102:0/1382614798 conn(0x7fde8c0cb9a0 msgr2=0x7fde8c0cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:29.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.279+0000 7fde91000640 1 -- 172.21.15.102:0/1382614798 shutdown_connections 2024-03-18T16:21:29.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.280+0000 7fde91000640 1 -- 172.21.15.102:0/1382614798 wait complete. 2024-03-18T16:21:29.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.282+0000 7fde91000640 1 Processor -- start 2024-03-18T16:21:29.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.282+0000 7fde91000640 1 -- start start 2024-03-18T16:21:29.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.283+0000 7fde91000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde8c06f590 0x7fde8c136280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:29.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.283+0000 7fde91000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c06f970 0x7fde8c1f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:29.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.283+0000 7fde91000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 0x7fde8c136b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:29.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.283+0000 7fde91000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c072500 con 0x7fde8c06f590 2024-03-18T16:21:29.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.283+0000 7fde91000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c072220 con 0x7fde8c06f970 2024-03-18T16:21:29.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.283+0000 7fde91000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde8c157430 con 0x7fde8c1401b0 2024-03-18T16:21:29.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.286+0000 7fde81400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c06f970 0x7fde8c1f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:29.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.286+0000 7fde81400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c06f970 0x7fde8c1f42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40696/0 (socket says 172.21.15.102:40696) 2024-03-18T16:21:29.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.286+0000 7fde81400640 1 -- 172.21.15.102:0/1342549927 learned_addr learned my addr 172.21.15.102:0/1342549927 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:29.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.292+0000 7fde8a800640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 0x7fde8c136b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:29.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.293+0000 7fde81400640 1 -- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 msgr2=0x7fde8c136b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:29.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.293+0000 7fde81400640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 0x7fde8c136b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.293+0000 7fde81400640 1 -- 172.21.15.102:0/1342549927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde8c06f590 msgr2=0x7fde8c136280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.293+0000 7fde81400640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde8c06f590 0x7fde8c136280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:29.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.293+0000 7fde81400640 1 -- 172.21.15.102:0/1342549927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c0cb330 con 0x7fde8c06f970 2024-03-18T16:21:29.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.297+0000 7fde81400640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c06f970 0x7fde8c1f42e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde7400bab0 tx=0x7fde7400bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:29.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.302+0000 7fde82a00640 1 -- 172.21.15.102:0/1342549927 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde74019070 con 0x7fde8c06f970 2024-03-18T16:21:29.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.302+0000 7fde82a00640 1 -- 172.21.15.102:0/1342549927 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde7400cc80 con 0x7fde8c06f970 2024-03-18T16:21:29.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.302+0000 7fde82a00640 1 -- 172.21.15.102:0/1342549927 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde74014680 con 0x7fde8c06f970 2024-03-18T16:21:29.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.302+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8c1f0360 con 0x7fde8c06f970 2024-03-18T16:21:29.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.302+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8c1f0620 con 0x7fde8c06f970 2024-03-18T16:21:29.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.303+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8c074fd0 con 0x7fde8c06f970 2024-03-18T16:21:29.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.333+0000 7fde82a00640 1 -- 172.21.15.102:0/1342549927 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fde7401c070 con 0x7fde8c06f970 2024-03-18T16:21:29.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.335+0000 7fde82a00640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:29.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.335+0000 7fde89e00640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:29.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.336+0000 7fde89e00640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009650 tx=0x7fde7c006ba0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:29.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.337+0000 7fde82a00640 1 -- 172.21.15.102:0/1342549927 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6500+0+0 (secure 0 0 0) 0x7fde740571c0 con 0x7fde8c06f970 2024-03-18T16:21:29.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.666+0000 7fde82a00640 1 -- 172.21.15.102:0/1342549927 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fde74023020 con 0x7fde8c06f970 2024-03-18T16:21:29.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:29.912+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fde8c157be0 con 0x7fde8c06f970 2024-03-18T16:21:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.146+0000 7fde82a00640 1 -- 172.21.15.102:0/1342549927 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7fde74059820 con 0x7fde8c06f970 2024-03-18T16:21:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:added data pool 4 to fsmap 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.150+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 msgr2=0x7fde6806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.150+0000 7fde91000640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009650 tx=0x7fde7c006ba0 comp rx=0 tx=0).stop 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.150+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c06f970 msgr2=0x7fde8c1f42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.150+0000 7fde91000640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c06f970 0x7fde8c1f42e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde7400bab0 tx=0x7fde7400bf70 comp rx=0 tx=0).stop 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.151+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 shutdown_connections 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.151+0000 7fde91000640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.151+0000 7fde91000640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8c1401b0 0x7fde8c136b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.151+0000 7fde91000640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde8c06f970 0x7fde8c1f42e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.151+0000 7fde91000640 1 --2- 172.21.15.102:0/1342549927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde8c06f590 0x7fde8c136280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.151+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 >> 172.21.15.102:0/1342549927 conn(0x7fde8c0cb9a0 msgr2=0x7fde8c13f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.151+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 shutdown_connections 2024-03-18T16:21:31.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.151+0000 7fde91000640 1 -- 172.21.15.102:0/1342549927 wait complete. 2024-03-18T16:21:31.181 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:21:31.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.375+0000 7f9796800640 1 Processor -- start 2024-03-18T16:21:31.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.375+0000 7f9796800640 1 -- start start 2024-03-18T16:21:31.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.377+0000 7f9796800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97900cc4f0 0x7f97900cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:31.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.377+0000 7f9796800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9790154ab0 con 0x7f979015ae70 2024-03-18T16:21:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.377+0000 7f9796800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f979015b250 con 0x7f97900c9be0 2024-03-18T16:21:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.377+0000 7f9796800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f97900caba0 con 0x7f97900cc4f0 2024-03-18T16:21:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.377+0000 7f9794a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f97900c9be0 0x7f97900d0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33126/0 (socket says 172.21.15.102:33126) 2024-03-18T16:21:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.377+0000 7f9794a00640 1 -- 172.21.15.102:0/2447104667 learned_addr learned my addr 172.21.15.102:0/2447104667 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:31.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.378+0000 7f9795e00640 1 --2- 172.21.15.102:0/2447104667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97900cc4f0 0x7f97900cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.380+0000 7f978fe00640 1 -- 172.21.15.102:0/2447104667 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1315847790 0 0) 0x7f979015b250 con 0x7f97900c9be0 2024-03-18T16:21:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.380+0000 7f978fe00640 1 -- 172.21.15.102:0/2447104667 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9778003520 con 0x7f97900c9be0 2024-03-18T16:21:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.381+0000 7f9795e00640 1 -- 172.21.15.102:0/2447104667 >> v1:172.21.15.154:6789/0 conn(0x7f97900c9be0 legacy=0x7f97900d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.381+0000 7f9795e00640 1 -- 172.21.15.102:0/2447104667 >> v1:172.21.15.102:6789/0 conn(0x7f979015ae70 legacy=0x7f97900ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:31.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.381+0000 7f9795e00640 1 -- 172.21.15.102:0/2447104667 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97900cce80 con 0x7f97900cc4f0 2024-03-18T16:21:31.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.383+0000 7f9795e00640 1 --2- 172.21.15.102:0/2447104667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97900cc4f0 0x7f97900cc940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f97880097d0 tx=0x7f978802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4911f1604db5374c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:31.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.387+0000 7f978fe00640 1 -- 172.21.15.102:0/2447104667 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9788035020 con 0x7f97900cc4f0 2024-03-18T16:21:31.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.387+0000 7f978fe00640 1 -- 172.21.15.102:0/2447104667 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9788031890 con 0x7f97900cc4f0 2024-03-18T16:21:31.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.387+0000 7f978fe00640 1 -- 172.21.15.102:0/2447104667 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9788031c60 con 0x7f97900cc4f0 2024-03-18T16:21:31.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.388+0000 7f9796800640 1 -- 172.21.15.102:0/2447104667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97900cc4f0 msgr2=0x7f97900cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:31.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.388+0000 7f9796800640 1 --2- 172.21.15.102:0/2447104667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97900cc4f0 0x7f97900cc940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f97880097d0 tx=0x7f978802b930 comp rx=0 tx=0).stop 2024-03-18T16:21:31.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.388+0000 7f9796800640 1 -- 172.21.15.102:0/2447104667 shutdown_connections 2024-03-18T16:21:31.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.388+0000 7f9796800640 1 --2- 172.21.15.102:0/2447104667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97900cc4f0 0x7f97900cc940 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:31.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.388+0000 7f9796800640 1 -- 172.21.15.102:0/2447104667 >> 172.21.15.102:0/2447104667 conn(0x7f97900673a0 msgr2=0x7f97900677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:31.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.388+0000 7f9796800640 1 -- 172.21.15.102:0/2447104667 shutdown_connections 2024-03-18T16:21:31.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.388+0000 7f9796800640 1 -- 172.21.15.102:0/2447104667 wait complete. 2024-03-18T16:21:31.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.390+0000 7f9796800640 1 Processor -- start 2024-03-18T16:21:31.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.390+0000 7f9796800640 1 -- start start 2024-03-18T16:21:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.391+0000 7f9796800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97900c9be0 0x7f9790155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.391+0000 7f9796800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97900cc4f0 0x7f97900b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.391+0000 7f9796800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f979015ae70 0x7f9790153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.391+0000 7f9796800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f979006d8b0 con 0x7f97900cc4f0 2024-03-18T16:21:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.391+0000 7f9796800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f979006d5d0 con 0x7f97900c9be0 2024-03-18T16:21:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.391+0000 7f9796800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f979006db90 con 0x7f979015ae70 2024-03-18T16:21:31.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.392+0000 7f9795400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97900c9be0 0x7f9790155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.392+0000 7f9795400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97900c9be0 0x7f9790155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:32820/0 (socket says 172.21.15.102:32820) 2024-03-18T16:21:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.392+0000 7f9795e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f979015ae70 0x7f9790153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:31.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.392+0000 7f9795400640 1 -- 172.21.15.102:0/1760662931 learned_addr learned my addr 172.21.15.102:0/1760662931 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.395+0000 7f9795400640 1 -- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f979015ae70 msgr2=0x7f9790153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.395+0000 7f9795400640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f979015ae70 0x7f9790153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.395+0000 7f9795400640 1 -- 172.21.15.102:0/1760662931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97900cc4f0 msgr2=0x7f97900b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.395+0000 7f9795400640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97900cc4f0 0x7f97900b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.395+0000 7f9795400640 1 -- 172.21.15.102:0/1760662931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97901edde0 con 0x7f97900c9be0 2024-03-18T16:21:31.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.398+0000 7f9795400640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97900c9be0 0x7f9790155070 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f978400c430 tx=0x7f978400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:31.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.402+0000 7f978d600640 1 -- 172.21.15.102:0/1760662931 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784017070 con 0x7f97900c9be0 2024-03-18T16:21:31.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.402+0000 7f978d600640 1 -- 172.21.15.102:0/1760662931 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97840052d0 con 0x7f97900c9be0 2024-03-18T16:21:31.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.402+0000 7f978d600640 1 -- 172.21.15.102:0/1760662931 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9784003850 con 0x7f97900c9be0 2024-03-18T16:21:31.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.402+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97901f0110 con 0x7f97900c9be0 2024-03-18T16:21:31.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.402+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97901f0990 con 0x7f97900c9be0 2024-03-18T16:21:31.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.403+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9790074fd0 con 0x7f97900c9be0 2024-03-18T16:21:31.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.709+0000 7f978d600640 1 -- 172.21.15.102:0/1760662931 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f978401a070 con 0x7f97900c9be0 2024-03-18T16:21:31.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.711+0000 7f978d600640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f977406bc20 0x7f977406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:31.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.711+0000 7f9794a00640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f977406bc20 0x7f977406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:31.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.713+0000 7f9794a00640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f977406bc20 0x7f977406e0e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f97901545b0 tx=0x7f97800015f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:31.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.714+0000 7f978d600640 1 -- 172.21.15.102:0/1760662931 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f9784013e10 con 0x7f97900c9be0 2024-03-18T16:21:31.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.728+0000 7f978d600640 1 -- 172.21.15.102:0/1760662931 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9784021020 con 0x7f97900c9be0 2024-03-18T16:21:31.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:31.975+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f979006de40 con 0x7f97900c9be0 2024-03-18T16:21:32.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.001+0000 7f978d600640 1 -- 172.21.15.102:0/1760662931 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6837 (secure 0 0 0) 0x7f9784058620 con 0x7f97900c9be0 2024-03-18T16:21:32.004 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:32.004 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4250,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6834/3423054813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3423054813},{"type":"v1","addr":"172.21.15.102:6834","nonce":3423054813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4253,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4256,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:21:30.104961+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1}]} 2024-03-18T16:21:32.004 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 7 2024-03-18T16:21:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f977406bc20 msgr2=0x7f977406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f977406bc20 0x7f977406e0e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f97901545b0 tx=0x7f97800015f0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97900c9be0 msgr2=0x7f9790155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:32.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97900c9be0 0x7f9790155070 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f978400c430 tx=0x7f978400c8f0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 shutdown_connections 2024-03-18T16:21:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f977406bc20 0x7f977406e0e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f979015ae70 0x7f9790153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97900cc4f0 0x7f97900b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 --2- 172.21.15.102:0/1760662931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97900c9be0 0x7f9790155070 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 >> 172.21.15.102:0/1760662931 conn(0x7f97900673a0 msgr2=0x7f9790158f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 shutdown_connections 2024-03-18T16:21:32.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.004+0000 7f9796800640 1 -- 172.21.15.102:0/1760662931 wait complete. 2024-03-18T16:21:32.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:32.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.230+0000 7fd5c5600640 1 Processor -- start 2024-03-18T16:21:32.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.230+0000 7fd5c5600640 1 -- start start 2024-03-18T16:21:32.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.231+0000 7fd5c5600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c0154780 0x7fd5c015a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:32.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.231+0000 7fd5c5600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd5c00cc1f0 0x7fd5c00caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:32.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.232+0000 7fd5c5600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5c015daa0 con 0x7fd5c00c5050 2024-03-18T16:21:32.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.232+0000 7fd5c5600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd5c015e970 con 0x7fd5c00cc1f0 2024-03-18T16:21:32.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.232+0000 7fd5c5600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5c00cc600 con 0x7fd5c0154780 2024-03-18T16:21:32.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.232+0000 7fd5b7e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd5c00c5050 0x7fd5c015e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47940/0 (socket says 172.21.15.102:47940) 2024-03-18T16:21:32.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.232+0000 7fd5bea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c0154780 0x7fd5c015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:32.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.232+0000 7fd5b7e00640 1 -- 172.21.15.102:0/3860641407 learned_addr learned my addr 172.21.15.102:0/3860641407 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:32.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.232+0000 7fd5bf400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd5c00cc1f0 0x7fd5c00caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:32.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.235+0000 7fd5bf400640 1 -- 172.21.15.102:0/3860641407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c0154780 msgr2=0x7fd5c015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:32.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.235+0000 7fd5bf400640 1 --2- 172.21.15.102:0/3860641407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c0154780 0x7fd5c015a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.235+0000 7fd5bf400640 1 -- 172.21.15.102:0/3860641407 >> v1:172.21.15.102:6789/0 conn(0x7fd5c00c5050 legacy=0x7fd5c015e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:32.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.235+0000 7fd5bf400640 1 -- 172.21.15.102:0/3860641407 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c015aad0 con 0x7fd5c00cc1f0 2024-03-18T16:21:32.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.237+0000 7fd5bf400640 1 --2- 172.21.15.102:0/3860641407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c00cc1f0 0x7fd5c00caca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009d90 tx=0x7fd5b802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba15cef47337bc3e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:32.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.239+0000 7fd5be000640 1 -- 172.21.15.102:0/3860641407 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8039070 con 0x7fd5c00cc1f0 2024-03-18T16:21:32.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.240+0000 7fd5be000640 1 -- 172.21.15.102:0/3860641407 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5b802cce0 con 0x7fd5c00cc1f0 2024-03-18T16:21:32.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.240+0000 7fd5be000640 1 -- 172.21.15.102:0/3860641407 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8034660 con 0x7fd5c00cc1f0 2024-03-18T16:21:32.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.241+0000 7fd5c5600640 1 -- 172.21.15.102:0/3860641407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c00cc1f0 msgr2=0x7fd5c00caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:32.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.241+0000 7fd5c5600640 1 --2- 172.21.15.102:0/3860641407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c00cc1f0 0x7fd5c00caca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8009d90 tx=0x7fd5b802bf30 comp rx=0 tx=0).stop 2024-03-18T16:21:32.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.241+0000 7fd5c5600640 1 -- 172.21.15.102:0/3860641407 shutdown_connections 2024-03-18T16:21:32.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.241+0000 7fd5c5600640 1 --2- 172.21.15.102:0/3860641407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c00cc1f0 0x7fd5c00caca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.241+0000 7fd5c5600640 1 --2- 172.21.15.102:0/3860641407 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c0154780 0x7fd5c015a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.241+0000 7fd5c5600640 1 -- 172.21.15.102:0/3860641407 >> 172.21.15.102:0/3860641407 conn(0x7fd5c0067160 msgr2=0x7fd5c0067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:32.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.241+0000 7fd5c5600640 1 -- 172.21.15.102:0/3860641407 shutdown_connections 2024-03-18T16:21:32.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.241+0000 7fd5c5600640 1 -- 172.21.15.102:0/3860641407 wait complete. 2024-03-18T16:21:32.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.243+0000 7fd5c5600640 1 Processor -- start 2024-03-18T16:21:32.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.243+0000 7fd5c5600640 1 -- start start 2024-03-18T16:21:32.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.244+0000 7fd5c5600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00c5050 0x7fd5c01f0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:32.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.244+0000 7fd5c5600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c00cc1f0 0x7fd5c0158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:32.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.244+0000 7fd5c5600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c0154780 0x7fd5c014a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:32.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.244+0000 7fd5c5600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c015dd40 con 0x7fd5c00c5050 2024-03-18T16:21:32.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.244+0000 7fd5c5600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c015b090 con 0x7fd5c0154780 2024-03-18T16:21:32.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.244+0000 7fd5c5600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5c015b370 con 0x7fd5c00cc1f0 2024-03-18T16:21:32.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.244+0000 7fd5bf400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c0154780 0x7fd5c014a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:32.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.245+0000 7fd5b7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c00cc1f0 0x7fd5c0158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:32.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.245+0000 7fd5bea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00c5050 0x7fd5c01f0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:32.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.245+0000 7fd5bf400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c0154780 0x7fd5c014a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:32850/0 (socket says 172.21.15.102:32850) 2024-03-18T16:21:32.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.245+0000 7fd5bf400640 1 -- 172.21.15.102:0/2466293791 learned_addr learned my addr 172.21.15.102:0/2466293791 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:32.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.248+0000 7fd5bf400640 1 -- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c00cc1f0 msgr2=0x7fd5c0158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:32.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.248+0000 7fd5bf400640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c00cc1f0 0x7fd5c0158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.248+0000 7fd5bf400640 1 -- 172.21.15.102:0/2466293791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00c5050 msgr2=0x7fd5c01f0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:32.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.248+0000 7fd5bf400640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00c5050 0x7fd5c01f0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.248+0000 7fd5bf400640 1 -- 172.21.15.102:0/2466293791 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5c01584c0 con 0x7fd5c0154780 2024-03-18T16:21:32.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.250+0000 7fd5bf400640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c0154780 0x7fd5c014a140 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5b802c540 tx=0x7fd5b8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.254+0000 7fd5b6a00640 1 -- 172.21.15.102:0/2466293791 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8042020 con 0x7fd5c0154780 2024-03-18T16:21:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.254+0000 7fd5b6a00640 1 -- 172.21.15.102:0/2466293791 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5b803a040 con 0x7fd5c0154780 2024-03-18T16:21:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.254+0000 7fd5b6a00640 1 -- 172.21.15.102:0/2466293791 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8039070 con 0x7fd5c0154780 2024-03-18T16:21:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.254+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c00cf400 con 0x7fd5c0154780 2024-03-18T16:21:32.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.254+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c00cf9c0 con 0x7fd5c0154780 2024-03-18T16:21:32.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.255+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5c0074fd0 con 0x7fd5c0154780 2024-03-18T16:21:32.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.276+0000 7fd5b6a00640 1 -- 172.21.15.102:0/2466293791 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd5b8005d10 con 0x7fd5c0154780 2024-03-18T16:21:32.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.277+0000 7fd5b6a00640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd59c06bc20 0x7fd59c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:32.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.277+0000 7fd5b6a00640 1 -- 172.21.15.102:0/2466293791 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7fd5b8051de0 con 0x7fd5c0154780 2024-03-18T16:21:32.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.277+0000 7fd5bea00640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd59c06bc20 0x7fd59c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:32.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.278+0000 7fd5bea00640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd59c06bc20 0x7fd59c06e0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009aa0 tx=0x7fd5b0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:32.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.526+0000 7fd5b6a00640 1 -- 172.21.15.102:0/2466293791 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd5b804c030 con 0x7fd5c0154780 2024-03-18T16:21:32.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.741+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5c014afa0 con 0x7fd5c0154780 2024-03-18T16:21:32.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.809+0000 7fd5b6a00640 1 -- 172.21.15.102:0/2466293791 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18574 (secure 0 0 0) 0x7fd5b807b760 con 0x7fd5c0154780 2024-03-18T16:21:32.812 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:32.813 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:30.100713+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:32.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd59c06bc20 msgr2=0x7fd59c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:32.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd59c06bc20 0x7fd59c06e0e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009aa0 tx=0x7fd5b0009190 comp rx=0 tx=0).stop 2024-03-18T16:21:32.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c0154780 msgr2=0x7fd5c014a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:32.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c0154780 0x7fd5c014a140 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd5b802c540 tx=0x7fd5b8004a40 comp rx=0 tx=0).stop 2024-03-18T16:21:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 shutdown_connections 2024-03-18T16:21:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd59c06bc20 0x7fd59c06e0e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5c0154780 0x7fd5c014a140 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5c00cc1f0 0x7fd5c0158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 --2- 172.21.15.102:0/2466293791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c00c5050 0x7fd5c01f0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 >> 172.21.15.102:0/2466293791 conn(0x7fd5c0067160 msgr2=0x7fd5c006c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 shutdown_connections 2024-03-18T16:21:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:32.812+0000 7fd5c5600640 1 -- 172.21.15.102:0/2466293791 wait complete. 2024-03-18T16:21:32.835 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:33.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.033+0000 7f3cfc600640 1 Processor -- start 2024-03-18T16:21:33.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.033+0000 7f3cfc600640 1 -- start start 2024-03-18T16:21:33.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.035+0000 7f3cfc600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf4154780 0x7f3cf415a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:33.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.035+0000 7f3cfc600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3cf40c5050 0x7f3cf40cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:33.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.035+0000 7f3cfc600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf415daa0 con 0x7f3cf415e260 2024-03-18T16:21:33.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.035+0000 7f3cfc600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf415aad0 con 0x7f3cf40c5050 2024-03-18T16:21:33.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.035+0000 7f3cfc600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf415adb0 con 0x7f3cf4154780 2024-03-18T16:21:33.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.035+0000 7f3cf9200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3cf40c5050 0x7f3cf40cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:33.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.036+0000 7f3cf9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf4154780 0x7f3cf415a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:33.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.036+0000 7f3cfa600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3cf415e260 0x7f3cf40caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47942/0 (socket says 172.21.15.102:47942) 2024-03-18T16:21:33.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.036+0000 7f3cfa600640 1 -- 172.21.15.102:0/1483629042 learned_addr learned my addr 172.21.15.102:0/1483629042 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:33.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.038+0000 7f3cf9200640 1 -- 172.21.15.102:0/1483629042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf4154780 msgr2=0x7f3cf415a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:33.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.038+0000 7f3cf9200640 1 --2- 172.21.15.102:0/1483629042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf4154780 0x7f3cf415a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:33.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.038+0000 7f3cf9200640 1 -- 172.21.15.102:0/1483629042 >> v1:172.21.15.102:6789/0 conn(0x7f3cf415e260 legacy=0x7f3cf40caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:33.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.038+0000 7f3cf9200640 1 -- 172.21.15.102:0/1483629042 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf415b090 con 0x7f3cf40c5050 2024-03-18T16:21:33.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.040+0000 7f3cf9200640 1 --2- 172.21.15.102:0/1483629042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf40c5050 0x7f3cf40cc1f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00097a0 tx=0x7f3ce002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b15e644692bf783 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:33.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.045+0000 7f3cf3e00640 1 -- 172.21.15.102:0/1483629042 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0034070 con 0x7f3cf40c5050 2024-03-18T16:21:33.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.045+0000 7f3cf3e00640 1 -- 172.21.15.102:0/1483629042 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce0030610 con 0x7f3cf40c5050 2024-03-18T16:21:33.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.045+0000 7f3cf3e00640 1 -- 172.21.15.102:0/1483629042 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce00309e0 con 0x7f3cf40c5050 2024-03-18T16:21:33.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.045+0000 7f3cfc600640 1 -- 172.21.15.102:0/1483629042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf40c5050 msgr2=0x7f3cf40cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:33.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.045+0000 7f3cfc600640 1 --2- 172.21.15.102:0/1483629042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf40c5050 0x7f3cf40cc1f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00097a0 tx=0x7f3ce002b960 comp rx=0 tx=0).stop 2024-03-18T16:21:33.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.046+0000 7f3cfc600640 1 -- 172.21.15.102:0/1483629042 shutdown_connections 2024-03-18T16:21:33.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.046+0000 7f3cfc600640 1 --2- 172.21.15.102:0/1483629042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf40c5050 0x7f3cf40cc1f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:33.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.046+0000 7f3cfc600640 1 --2- 172.21.15.102:0/1483629042 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf4154780 0x7f3cf415a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:33.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.046+0000 7f3cfc600640 1 -- 172.21.15.102:0/1483629042 >> 172.21.15.102:0/1483629042 conn(0x7f3cf4067160 msgr2=0x7f3cf4067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:33.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.046+0000 7f3cfc600640 1 -- 172.21.15.102:0/1483629042 shutdown_connections 2024-03-18T16:21:33.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.046+0000 7f3cfc600640 1 -- 172.21.15.102:0/1483629042 wait complete. 2024-03-18T16:21:33.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.047+0000 7f3cfc600640 1 Processor -- start 2024-03-18T16:21:33.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.047+0000 7f3cfc600640 1 -- start start 2024-03-18T16:21:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.048+0000 7f3cfc600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf40c5050 0x7f3cf414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.048+0000 7f3cfc600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf4154780 0x7f3cf415c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.048+0000 7f3cfc600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf415e260 0x7f3cf415cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.048+0000 7f3cfc600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf4154b60 con 0x7f3cf4154780 2024-03-18T16:21:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.048+0000 7f3cfc600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf415b370 con 0x7f3cf415e260 2024-03-18T16:21:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.048+0000 7f3cfc600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf40c55a0 con 0x7f3cf40c5050 2024-03-18T16:21:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.049+0000 7f3cfa600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf415e260 0x7f3cf415cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:33.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.049+0000 7f3cf9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf4154780 0x7f3cf415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:33.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.049+0000 7f3cf9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf40c5050 0x7f3cf414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:33.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.049+0000 7f3cfa600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf415e260 0x7f3cf415cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:32870/0 (socket says 172.21.15.102:32870) 2024-03-18T16:21:33.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.049+0000 7f3cfa600640 1 -- 172.21.15.102:0/1552911587 learned_addr learned my addr 172.21.15.102:0/1552911587 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:33.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.052+0000 7f3cfa600640 1 -- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf40c5050 msgr2=0x7f3cf414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:33.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.052+0000 7f3cfa600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf40c5050 0x7f3cf414aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:33.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.052+0000 7f3cfa600640 1 -- 172.21.15.102:0/1552911587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf4154780 msgr2=0x7f3cf415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:33.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.052+0000 7f3cfa600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf4154780 0x7f3cf415c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:33.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.052+0000 7f3cfa600640 1 -- 172.21.15.102:0/1552911587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf415a790 con 0x7f3cf415e260 2024-03-18T16:21:33.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.055+0000 7f3cfa600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf415e260 0x7f3cf415cd90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00cb40 tx=0x7f3cec00a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.059+0000 7f3cf1600640 1 -- 172.21.15.102:0/1552911587 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec017070 con 0x7f3cf415e260 2024-03-18T16:21:33.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.059+0000 7f3cf1600640 1 -- 172.21.15.102:0/1552911587 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cec00aca0 con 0x7f3cf415e260 2024-03-18T16:21:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.059+0000 7f3cf1600640 1 -- 172.21.15.102:0/1552911587 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cec012620 con 0x7f3cf415e260 2024-03-18T16:21:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.059+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf414afd0 con 0x7f3cf415e260 2024-03-18T16:21:33.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.060+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf415a210 con 0x7f3cf415e260 2024-03-18T16:21:33.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.061+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf4074fd0 con 0x7f3cf415e260 2024-03-18T16:21:33.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.085+0000 7f3cf1600640 1 -- 172.21.15.102:0/1552911587 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3cec01a070 con 0x7f3cf415e260 2024-03-18T16:21:33.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.086+0000 7f3cf1600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cd406bc20 0x7f3cd406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:33.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.086+0000 7f3cf9c00640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cd406bc20 0x7f3cd406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:33.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.087+0000 7f3cf9c00640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cd406bc20 0x7f3cd406e0e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009b40 tx=0x7f3ce8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:33.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.345+0000 7f3cf1600640 1 -- 172.21.15.102:0/1552911587 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f3cec0570e0 con 0x7f3cf415e260 2024-03-18T16:21:33.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.357+0000 7f3cf1600640 1 -- 172.21.15.102:0/1552911587 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3cec021020 con 0x7f3cf415e260 2024-03-18T16:21:33.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.561+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3cf414e460 con 0x7f3cf415e260 2024-03-18T16:21:33.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.600+0000 7f3cf1600640 1 -- 172.21.15.102:0/1552911587 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18574 (secure 0 0 0) 0x7f3cec059040 con 0x7f3cf415e260 2024-03-18T16:21:33.602 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:33.602 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:30.100713+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cd406bc20 msgr2=0x7f3cd406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cd406bc20 0x7f3cd406e0e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009b40 tx=0x7f3ce8009190 comp rx=0 tx=0).stop 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf415e260 msgr2=0x7f3cf415cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf415e260 0x7f3cf415cd90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3cec00cb40 tx=0x7f3cec00a060 comp rx=0 tx=0).stop 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 shutdown_connections 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cd406bc20 0x7f3cd406e0e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf415e260 0x7f3cf415cd90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3cf4154780 0x7f3cf415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 --2- 172.21.15.102:0/1552911587 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3cf40c5050 0x7f3cf414aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 >> 172.21.15.102:0/1552911587 conn(0x7f3cf4067160 msgr2=0x7f3cf406cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:33.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 shutdown_connections 2024-03-18T16:21:33.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:33.603+0000 7f3cfc600640 1 -- 172.21.15.102:0/1552911587 wait complete. 2024-03-18T16:21:33.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:21:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.171+0000 7f562c600640 1 Processor -- start 2024-03-18T16:21:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.171+0000 7f562c600640 1 -- start start 2024-03-18T16:21:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.173+0000 7f562c600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f56241524c0 0x7f5624072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.173+0000 7f562c600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f562415b8e0 0x7f56240c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.173+0000 7f562c600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f562415c200 con 0x7f5624158ee0 2024-03-18T16:21:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.173+0000 7f562c600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5624159300 con 0x7f562415b8e0 2024-03-18T16:21:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.173+0000 7f562c600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f562415c4e0 con 0x7f56241524c0 2024-03-18T16:21:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.173+0000 7f5629c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f56241524c0 0x7f5624072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.174+0000 7f5629c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f56241524c0 0x7f5624072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:32876/0 (socket says 172.21.15.102:32876) 2024-03-18T16:21:34.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.174+0000 7f5629c00640 1 -- 172.21.15.102:0/1620736874 learned_addr learned my addr 172.21.15.102:0/1620736874 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:34.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.174+0000 7f562a600640 1 --2- 172.21.15.102:0/1620736874 >> v2:172.21.15.102:3300/0 conn(0x7f562415b8e0 0x7f56240c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.176+0000 7f5629c00640 1 -- 172.21.15.102:0/1620736874 >> v2:172.21.15.102:3300/0 conn(0x7f562415b8e0 msgr2=0x7f56240c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:34.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.176+0000 7f5629c00640 1 --2- 172.21.15.102:0/1620736874 >> v2:172.21.15.102:3300/0 conn(0x7f562415b8e0 0x7f56240c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.176+0000 7f5629c00640 1 -- 172.21.15.102:0/1620736874 >> v1:172.21.15.102:6789/0 conn(0x7f5624158ee0 legacy=0x7f562406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.176+0000 7f5629c00640 1 -- 172.21.15.102:0/1620736874 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562415c7c0 con 0x7f56241524c0 2024-03-18T16:21:34.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.178+0000 7f5629c00640 1 --2- 172.21.15.102:0/1620736874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 0x7f5624072700 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f562406bf90 tx=0x7f561802b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3b7d59665733b13 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:34.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.181+0000 7f5623e00640 1 -- 172.21.15.102:0/1620736874 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618034070 con 0x7f56241524c0 2024-03-18T16:21:34.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.183+0000 7f5623e00640 1 -- 172.21.15.102:0/1620736874 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5618030870 con 0x7f56241524c0 2024-03-18T16:21:34.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.183+0000 7f5623e00640 1 -- 172.21.15.102:0/1620736874 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618030c40 con 0x7f56241524c0 2024-03-18T16:21:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.183+0000 7f562c600640 1 -- 172.21.15.102:0/1620736874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 msgr2=0x7f5624072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.183+0000 7f562c600640 1 --2- 172.21.15.102:0/1620736874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 0x7f5624072700 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f562406bf90 tx=0x7f561802b9b0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.184+0000 7f562c600640 1 -- 172.21.15.102:0/1620736874 shutdown_connections 2024-03-18T16:21:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.184+0000 7f562c600640 1 --2- 172.21.15.102:0/1620736874 >> v2:172.21.15.102:3300/0 conn(0x7f562415b8e0 0x7f56240c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.184+0000 7f562c600640 1 --2- 172.21.15.102:0/1620736874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 0x7f5624072700 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.184+0000 7f562c600640 1 -- 172.21.15.102:0/1620736874 >> 172.21.15.102:0/1620736874 conn(0x7f56240700d0 msgr2=0x7f562406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.184+0000 7f562c600640 1 -- 172.21.15.102:0/1620736874 shutdown_connections 2024-03-18T16:21:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.184+0000 7f562c600640 1 -- 172.21.15.102:0/1620736874 wait complete. 2024-03-18T16:21:34.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.185+0000 7f562c600640 1 Processor -- start 2024-03-18T16:21:34.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.186+0000 7f562c600640 1 -- start start 2024-03-18T16:21:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.186+0000 7f562c600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 0x7f562414cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.187+0000 7f562c600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5624158ee0 0x7f56240c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.187+0000 7f562c600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f562415b8e0 0x7f56240c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.187+0000 7f562c600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f56241589c0 con 0x7f562415b8e0 2024-03-18T16:21:34.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.187+0000 7f562c600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56241586e0 con 0x7f56241524c0 2024-03-18T16:21:34.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.187+0000 7f562c600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f56241528a0 con 0x7f5624158ee0 2024-03-18T16:21:34.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.187+0000 7f5629c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 0x7f562414cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.187+0000 7f5629200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5624158ee0 0x7f56240c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.187+0000 7f5629c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 0x7f562414cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:32884/0 (socket says 172.21.15.102:32884) 2024-03-18T16:21:34.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.187+0000 7f5629c00640 1 -- 172.21.15.102:0/3447447034 learned_addr learned my addr 172.21.15.102:0/3447447034 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:34.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.188+0000 7f562a600640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f562415b8e0 0x7f56240c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.191+0000 7f5629c00640 1 -- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5624158ee0 msgr2=0x7f56240c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.191+0000 7f5629c00640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5624158ee0 0x7f56240c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.191+0000 7f5629c00640 1 -- 172.21.15.102:0/3447447034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f562415b8e0 msgr2=0x7f56240c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.191+0000 7f5629c00640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f562415b8e0 0x7f56240c7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.191+0000 7f5629c00640 1 -- 172.21.15.102:0/3447447034 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56240b1350 con 0x7f56241524c0 2024-03-18T16:21:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.191+0000 7f5629200640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5624158ee0 0x7f56240c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:34.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.194+0000 7f5629c00640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 0x7f562414cb00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f561802b9f0 tx=0x7f56180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:34.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.198+0000 7f5621600640 1 -- 172.21.15.102:0/3447447034 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618034040 con 0x7f56241524c0 2024-03-18T16:21:34.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.198+0000 7f5621600640 1 -- 172.21.15.102:0/3447447034 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5618039d90 con 0x7f56241524c0 2024-03-18T16:21:34.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.198+0000 7f5621600640 1 -- 172.21.15.102:0/3447447034 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5618043630 con 0x7f56241524c0 2024-03-18T16:21:34.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.198+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5624148c60 con 0x7f56241524c0 2024-03-18T16:21:34.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.198+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5624148980 con 0x7f56241524c0 2024-03-18T16:21:34.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.199+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5624074fd0 con 0x7f56241524c0 2024-03-18T16:21:34.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.220+0000 7f5621600640 1 -- 172.21.15.102:0/3447447034 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5618037070 con 0x7f56241524c0 2024-03-18T16:21:34.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.221+0000 7f5621600640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f560406bc20 0x7f560406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.222+0000 7f5621600640 1 -- 172.21.15.102:0/3447447034 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f56180ad8a0 con 0x7f56241524c0 2024-03-18T16:21:34.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.222+0000 7f5629200640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f560406bc20 0x7f560406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.222+0000 7f5629200640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f560406bc20 0x7f560406e0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f56240c84a0 tx=0x7f5610009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:34.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.481+0000 7f5621600640 1 -- 172.21.15.102:0/3447447034 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f561803e070 con 0x7f56241524c0 2024-03-18T16:21:34.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.735+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56240c5430 con 0x7f56241524c0 2024-03-18T16:21:34.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.749+0000 7f5621600640 1 -- 172.21.15.102:0/3447447034 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6837 (secure 0 0 0) 0x7f561807bb70 con 0x7f56241524c0 2024-03-18T16:21:34.752 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:34.752 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4250,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6834/3423054813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3423054813},{"type":"v1","addr":"172.21.15.102:6834","nonce":3423054813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4253,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4256,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:21:30.104961+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1}]} 2024-03-18T16:21:34.752 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 7 2024-03-18T16:21:34.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f560406bc20 msgr2=0x7f560406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f560406bc20 0x7f560406e0e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f56240c84a0 tx=0x7f5610009040 comp rx=0 tx=0).stop 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 msgr2=0x7f562414cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 0x7f562414cb00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f561802b9f0 tx=0x7f56180046c0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 shutdown_connections 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f560406bc20 0x7f560406e0e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f562415b8e0 0x7f56240c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5624158ee0 0x7f56240c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 --2- 172.21.15.102:0/3447447034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56241524c0 0x7f562414cb00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.752+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 >> 172.21.15.102:0/3447447034 conn(0x7f56240700d0 msgr2=0x7f5624071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.753+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 shutdown_connections 2024-03-18T16:21:34.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.753+0000 7f562c600640 1 -- 172.21.15.102:0/3447447034 wait complete. 2024-03-18T16:21:34.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:34.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.973+0000 7f82c9800640 1 Processor -- start 2024-03-18T16:21:34.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.974+0000 7f82c9800640 1 -- start start 2024-03-18T16:21:34.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.975+0000 7f82c9800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f82c41524c0 0x7f82c4072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.975+0000 7f82c9800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f82c415b8e0 0x7f82c40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.975+0000 7f82c9800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c415c200 con 0x7f82c4158ee0 2024-03-18T16:21:34.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.975+0000 7f82c9800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f82c4159300 con 0x7f82c41524c0 2024-03-18T16:21:34.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.975+0000 7f82c9800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f82c415c4e0 con 0x7f82c415b8e0 2024-03-18T16:21:34.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.976+0000 7f82c3400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f82c415b8e0 0x7f82c40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.976+0000 7f82c2a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f82c41524c0 0x7f82c4072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.976+0000 7f82ba000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f82c4158ee0 0x7f82c406e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33140/0 (socket says 172.21.15.102:33140) 2024-03-18T16:21:34.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.976+0000 7f82ba000640 1 -- 172.21.15.102:0/3314569565 learned_addr learned my addr 172.21.15.102:0/3314569565 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:34.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.979+0000 7f82c3400640 1 -- 172.21.15.102:0/3314569565 >> v2:172.21.15.102:3300/0 conn(0x7f82c41524c0 msgr2=0x7f82c4072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:34.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.979+0000 7f82c3400640 1 --2- 172.21.15.102:0/3314569565 >> v2:172.21.15.102:3300/0 conn(0x7f82c41524c0 0x7f82c4072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.979+0000 7f82c3400640 1 -- 172.21.15.102:0/3314569565 >> v1:172.21.15.154:6789/0 conn(0x7f82c4158ee0 legacy=0x7f82c406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.979+0000 7f82c3400640 1 -- 172.21.15.102:0/3314569565 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c415c7c0 con 0x7f82c415b8e0 2024-03-18T16:21:34.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.983+0000 7f82c3400640 1 --2- 172.21.15.102:0/3314569565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c415b8e0 0x7f82c40c51e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f82c406c090 tx=0x7f82bc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6421162de022c4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:34.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.987+0000 7f82c2000640 1 -- 172.21.15.102:0/3314569565 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82bc035020 con 0x7f82c415b8e0 2024-03-18T16:21:34.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.987+0000 7f82c2000640 1 -- 172.21.15.102:0/3314569565 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82bc031830 con 0x7f82c415b8e0 2024-03-18T16:21:34.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.987+0000 7f82c2000640 1 -- 172.21.15.102:0/3314569565 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82bc031c00 con 0x7f82c415b8e0 2024-03-18T16:21:34.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.987+0000 7f82c9800640 1 -- 172.21.15.102:0/3314569565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c415b8e0 msgr2=0x7f82c40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.987+0000 7f82c9800640 1 --2- 172.21.15.102:0/3314569565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c415b8e0 0x7f82c40c51e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f82c406c090 tx=0x7f82bc02b8d0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.988+0000 7f82c9800640 1 -- 172.21.15.102:0/3314569565 shutdown_connections 2024-03-18T16:21:34.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.988+0000 7f82c9800640 1 --2- 172.21.15.102:0/3314569565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c415b8e0 0x7f82c40c51e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.988+0000 7f82c9800640 1 --2- 172.21.15.102:0/3314569565 >> v2:172.21.15.102:3300/0 conn(0x7f82c41524c0 0x7f82c4072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.988+0000 7f82c9800640 1 -- 172.21.15.102:0/3314569565 >> 172.21.15.102:0/3314569565 conn(0x7f82c40700d0 msgr2=0x7f82c406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:34.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.988+0000 7f82c9800640 1 -- 172.21.15.102:0/3314569565 shutdown_connections 2024-03-18T16:21:34.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.988+0000 7f82c9800640 1 -- 172.21.15.102:0/3314569565 wait complete. 2024-03-18T16:21:34.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.989+0000 7f82c9800640 1 Processor -- start 2024-03-18T16:21:34.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.990+0000 7f82c9800640 1 -- start start 2024-03-18T16:21:34.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c9800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c41524c0 0x7f82c41f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c9800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82c4158ee0 0x7f82c40afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c9800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415b8e0 0x7f82c40b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:34.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c9800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f82c41589c0 con 0x7f82c415b8e0 2024-03-18T16:21:34.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c9800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82c41586e0 con 0x7f82c41524c0 2024-03-18T16:21:34.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c9800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f82c41528a0 con 0x7f82c4158ee0 2024-03-18T16:21:34.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c41524c0 0x7f82c41f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415b8e0 0x7f82c40b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82ba000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82c4158ee0 0x7f82c40afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:34.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c41524c0 0x7f82c41f33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:32912/0 (socket says 172.21.15.102:32912) 2024-03-18T16:21:34.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.991+0000 7f82c2a00640 1 -- 172.21.15.102:0/69604809 learned_addr learned my addr 172.21.15.102:0/69604809 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:34.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.995+0000 7f82c2a00640 1 -- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82c4158ee0 msgr2=0x7f82c40afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.995+0000 7f82c2a00640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82c4158ee0 0x7f82c40afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.995+0000 7f82c2a00640 1 -- 172.21.15.102:0/69604809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415b8e0 msgr2=0x7f82c40b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:34.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.995+0000 7f82c2a00640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415b8e0 0x7f82c40b0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:34.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.995+0000 7f82c2a00640 1 -- 172.21.15.102:0/69604809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c4157990 con 0x7f82c41524c0 2024-03-18T16:21:34.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:34.997+0000 7f82c2a00640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c41524c0 0x7f82c41f33c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f82b400efc0 tx=0x7f82b400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:35.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.001+0000 7f82bb400640 1 -- 172.21.15.102:0/69604809 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4018020 con 0x7f82c41524c0 2024-03-18T16:21:35.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.001+0000 7f82bb400640 1 -- 172.21.15.102:0/69604809 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b4014480 con 0x7f82c41524c0 2024-03-18T16:21:35.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.001+0000 7f82bb400640 1 -- 172.21.15.102:0/69604809 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4014850 con 0x7f82c41524c0 2024-03-18T16:21:35.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.001+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c41f3e30 con 0x7f82c41524c0 2024-03-18T16:21:35.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.001+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c414e7b0 con 0x7f82c41524c0 2024-03-18T16:21:35.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.003+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c4074fd0 con 0x7f82c41524c0 2024-03-18T16:21:35.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.266+0000 7f82bb400640 1 -- 172.21.15.102:0/69604809 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f82b4005d10 con 0x7f82c41524c0 2024-03-18T16:21:35.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.267+0000 7f82bb400640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82a006bc50 0x7f82a006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:35.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.268+0000 7f82ba000640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82a006bc50 0x7f82a006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:35.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.268+0000 7f82ba000640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82a006bc50 0x7f82a006e110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f82c4157640 tx=0x7f82ac007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:35.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.270+0000 7f82bb400640 1 -- 172.21.15.102:0/69604809 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f82b40583e0 con 0x7f82c41524c0 2024-03-18T16:21:35.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.289+0000 7f82bb400640 1 -- 172.21.15.102:0/69604809 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f82b4022020 con 0x7f82c41524c0 2024-03-18T16:21:35.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.493+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82c40cad10 con 0x7f82c41524c0 2024-03-18T16:21:35.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.531+0000 7f82bb400640 1 -- 172.21.15.102:0/69604809 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18574 (secure 0 0 0) 0x7f82b4010040 con 0x7f82c41524c0 2024-03-18T16:21:35.533 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:35.533 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:30.100713+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82a006bc50 msgr2=0x7f82a006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82a006bc50 0x7f82a006e110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f82c4157640 tx=0x7f82ac007e80 comp rx=0 tx=0).stop 2024-03-18T16:21:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c41524c0 msgr2=0x7f82c41f33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c41524c0 0x7f82c41f33c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f82b400efc0 tx=0x7f82b400c490 comp rx=0 tx=0).stop 2024-03-18T16:21:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 shutdown_connections 2024-03-18T16:21:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82a006bc50 0x7f82a006e110 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415b8e0 0x7f82c40b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82c4158ee0 0x7f82c40afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:35.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 --2- 172.21.15.102:0/69604809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82c41524c0 0x7f82c41f33c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:35.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 >> 172.21.15.102:0/69604809 conn(0x7f82c40700d0 msgr2=0x7f82c40ca3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:35.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 shutdown_connections 2024-03-18T16:21:35.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:35.534+0000 7f82c9800640 1 -- 172.21.15.102:0/69604809 wait complete. 2024-03-18T16:21:36.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:21:30.104961+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4270, 'qdb_cluster': [4270]} 2024-03-18T16:21:36.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T16:21:36.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-18T16:21:36.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.376+0000 7ffa43c00640 1 Processor -- start 2024-03-18T16:21:36.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.376+0000 7ffa43c00640 1 -- start start 2024-03-18T16:21:36.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.377+0000 7ffa43c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffa3c0c5830 0x7ffa3c141580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.377+0000 7ffa43c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c1582f0 0x7ffa3c158780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.377+0000 7ffa43c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa3c072050 con 0x7ffa3c0c5450 2024-03-18T16:21:36.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.377+0000 7ffa43c00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ffa3c15c8b0 con 0x7ffa3c0c5830 2024-03-18T16:21:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.377+0000 7ffa43c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ffa3c141ac0 con 0x7ffa3c1582f0 2024-03-18T16:21:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.378+0000 7ffa33e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffa3c0c5830 0x7ffa3c141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.378+0000 7ffa41c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c1582f0 0x7ffa3c158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.378+0000 7ffa41200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffa3c0c5450 0x7ffa3c071940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47962/0 (socket says 172.21.15.102:47962) 2024-03-18T16:21:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.378+0000 7ffa41200640 1 -- 172.21.15.102:0/441918053 learned_addr learned my addr 172.21.15.102:0/441918053 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:36.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.381+0000 7ffa33e00640 1 -- 172.21.15.102:0/441918053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c1582f0 msgr2=0x7ffa3c158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.381+0000 7ffa33e00640 1 --2- 172.21.15.102:0/441918053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c1582f0 0x7ffa3c158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.381+0000 7ffa33e00640 1 -- 172.21.15.102:0/441918053 >> v1:172.21.15.102:6789/0 conn(0x7ffa3c0c5450 legacy=0x7ffa3c071940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.381+0000 7ffa33e00640 1 -- 172.21.15.102:0/441918053 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c0722f0 con 0x7ffa3c0c5830 2024-03-18T16:21:36.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.384+0000 7ffa33e00640 1 --2- 172.21.15.102:0/441918053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 0x7ffa3c141580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffa240097a0 tx=0x7ffa2402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b72d2f03137b1b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:36.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.387+0000 7ffa3be00640 1 -- 172.21.15.102:0/441918053 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24035020 con 0x7ffa3c0c5830 2024-03-18T16:21:36.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.387+0000 7ffa3be00640 1 -- 172.21.15.102:0/441918053 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa24031830 con 0x7ffa3c0c5830 2024-03-18T16:21:36.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.387+0000 7ffa3be00640 1 -- 172.21.15.102:0/441918053 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24031c00 con 0x7ffa3c0c5830 2024-03-18T16:21:36.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.388+0000 7ffa43c00640 1 -- 172.21.15.102:0/441918053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 msgr2=0x7ffa3c141580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.388+0000 7ffa43c00640 1 --2- 172.21.15.102:0/441918053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 0x7ffa3c141580 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffa240097a0 tx=0x7ffa2402b8d0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.388+0000 7ffa43c00640 1 -- 172.21.15.102:0/441918053 shutdown_connections 2024-03-18T16:21:36.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.388+0000 7ffa43c00640 1 --2- 172.21.15.102:0/441918053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c1582f0 0x7ffa3c158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.388+0000 7ffa43c00640 1 --2- 172.21.15.102:0/441918053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 0x7ffa3c141580 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.388+0000 7ffa43c00640 1 -- 172.21.15.102:0/441918053 >> 172.21.15.102:0/441918053 conn(0x7ffa3c0700d0 msgr2=0x7ffa3c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:36.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.388+0000 7ffa43c00640 1 -- 172.21.15.102:0/441918053 shutdown_connections 2024-03-18T16:21:36.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.388+0000 7ffa43c00640 1 -- 172.21.15.102:0/441918053 wait complete. 2024-03-18T16:21:36.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.390+0000 7ffa43c00640 1 Processor -- start 2024-03-18T16:21:36.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.390+0000 7ffa43c00640 1 -- start start 2024-03-18T16:21:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.391+0000 7ffa43c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c0c5450 0x7ffa3c0b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.391+0000 7ffa43c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 0x7ffa3c13dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.391+0000 7ffa43c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa3c1582f0 0x7ffa3c1f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.391+0000 7ffa43c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c15c0b0 con 0x7ffa3c1582f0 2024-03-18T16:21:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.391+0000 7ffa43c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c15bdd0 con 0x7ffa3c0c5830 2024-03-18T16:21:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.391+0000 7ffa43c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ffa3c151ed0 con 0x7ffa3c0c5450 2024-03-18T16:21:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.392+0000 7ffa41200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c0c5450 0x7ffa3c0b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.392+0000 7ffa41c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa3c1582f0 0x7ffa3c1f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.392+0000 7ffa33e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 0x7ffa3c13dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.392+0000 7ffa33e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 0x7ffa3c13dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:32934/0 (socket says 172.21.15.102:32934) 2024-03-18T16:21:36.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.392+0000 7ffa33e00640 1 -- 172.21.15.102:0/1469194854 learned_addr learned my addr 172.21.15.102:0/1469194854 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:36.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.396+0000 7ffa33e00640 1 -- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c0c5450 msgr2=0x7ffa3c0b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.396+0000 7ffa33e00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c0c5450 0x7ffa3c0b1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.396+0000 7ffa33e00640 1 -- 172.21.15.102:0/1469194854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa3c1582f0 msgr2=0x7ffa3c1f8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.396+0000 7ffa33e00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa3c1582f0 0x7ffa3c1f8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.396+0000 7ffa33e00640 1 -- 172.21.15.102:0/1469194854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c1f76c0 con 0x7ffa3c0c5830 2024-03-18T16:21:36.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.398+0000 7ffa33e00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 0x7ffa3c13dcf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffa2402bee0 tx=0x7ffa240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:36.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.402+0000 7ffa39600640 1 -- 172.21.15.102:0/1469194854 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24041070 con 0x7ffa3c0c5830 2024-03-18T16:21:36.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.402+0000 7ffa39600640 1 -- 172.21.15.102:0/1469194854 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2403b920 con 0x7ffa3c0c5830 2024-03-18T16:21:36.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.402+0000 7ffa39600640 1 -- 172.21.15.102:0/1469194854 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24035040 con 0x7ffa3c0c5830 2024-03-18T16:21:36.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.402+0000 7ffa43c00640 1 -- 172.21.15.102:0/1469194854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3c13d6f0 con 0x7ffa3c0c5830 2024-03-18T16:21:36.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.402+0000 7ffa43c00640 1 -- 172.21.15.102:0/1469194854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa3c1f7980 con 0x7ffa3c0c5830 2024-03-18T16:21:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.406+0000 7ffa4680d740 1 Processor -- start 2024-03-18T16:21:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.406+0000 7ffa4680d740 1 -- start start 2024-03-18T16:21:36.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.407+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x558071868510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.407+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558071867210 0x5580718a9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.407+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8b50 0x5580718a7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.407+0000 7ffa4680d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x5580717cbc50 con 0x5580718a8770 2024-03-18T16:21:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.407+0000 7ffa4680d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5580717749d0 con 0x5580718a8b50 2024-03-18T16:21:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.407+0000 7ffa4680d740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5580717c7cf0 con 0x558071867210 2024-03-18T16:21:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.407+0000 7ffa41200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558071867210 0x5580718a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.407+0000 7ffa41c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x558071868510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.408+0000 7ffa41200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558071867210 0x5580718a9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56796/0 (socket says 172.21.15.102:56796) 2024-03-18T16:21:36.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.408+0000 7ffa41200640 1 -- 172.21.15.102:0/1140912584 learned_addr learned my addr 172.21.15.102:0/1140912584 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:36.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.410+0000 7ffa41200640 1 -- 172.21.15.102:0/1140912584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8b50 msgr2=0x5580718a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:36.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.410+0000 7ffa41200640 1 --2- 172.21.15.102:0/1140912584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8b50 0x5580718a7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.410+0000 7ffa41200640 1 -- 172.21.15.102:0/1140912584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 msgr2=0x558071868510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:36.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.410+0000 7ffa41200640 1 --2- 172.21.15.102:0/1140912584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x558071868510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.410+0000 7ffa41200640 1 -- 172.21.15.102:0/1140912584 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55807178ef10 con 0x558071867210 2024-03-18T16:21:36.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.412+0000 7ffa41200640 1 --2- 172.21.15.102:0/1140912584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558071867210 0x5580718a9130 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c002090 tx=0x7ffa2c007430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e59a3f424d0e98e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.417+0000 7ffa31600640 1 -- 172.21.15.102:0/1140912584 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c004e50 con 0x558071867210 2024-03-18T16:21:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.417+0000 7ffa31600640 1 -- 172.21.15.102:0/1140912584 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2c018070 con 0x558071867210 2024-03-18T16:21:36.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.417+0000 7ffa31600640 1 -- 172.21.15.102:0/1140912584 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c012780 con 0x558071867210 2024-03-18T16:21:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.418+0000 7ffa4680d740 1 -- 172.21.15.102:0/1140912584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558071867210 msgr2=0x5580718a9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.418+0000 7ffa4680d740 1 --2- 172.21.15.102:0/1140912584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558071867210 0x5580718a9130 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c002090 tx=0x7ffa2c007430 comp rx=0 tx=0).stop 2024-03-18T16:21:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.418+0000 7ffa4680d740 1 -- 172.21.15.102:0/1140912584 shutdown_connections 2024-03-18T16:21:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.418+0000 7ffa4680d740 1 --2- 172.21.15.102:0/1140912584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8b50 0x5580718a7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.418+0000 7ffa4680d740 1 --2- 172.21.15.102:0/1140912584 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x558071867210 0x5580718a9130 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.418+0000 7ffa4680d740 1 --2- 172.21.15.102:0/1140912584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x558071868510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.418+0000 7ffa4680d740 1 -- 172.21.15.102:0/1140912584 >> 172.21.15.102:0/1140912584 conn(0x558071864380 msgr2=0x558071867b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:36.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.418+0000 7ffa4680d740 1 -- 172.21.15.102:0/1140912584 shutdown_connections 2024-03-18T16:21:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.418+0000 7ffa4680d740 1 -- 172.21.15.102:0/1140912584 wait complete. 2024-03-18T16:21:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.419+0000 7ffa4680d740 1 -- start start 2024-03-18T16:21:36.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.419+0000 7ffa4680d740 1 Processor -- start 2024-03-18T16:21:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.420+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 0x55807188e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.420+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 0x55807188ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.420+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x558071886380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.420+0000 7ffa4680d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x558071787e20 con 0x5580718a8b50 2024-03-18T16:21:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.420+0000 7ffa4680d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5580718b1aa0 con 0x5580718a8770 2024-03-18T16:21:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.420+0000 7ffa4680d740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x55807188f890 con 0x5580718a5f40 2024-03-18T16:21:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.420+0000 7ffa4680d740 10 client.0 _ll_register_callbacks cb 0x5580717877f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:21:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.421+0000 7ffa33e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x558071886380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.421+0000 7ffa41200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 0x55807188ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.421+0000 7ffa33e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x558071886380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56810/0 (socket says 172.21.15.102:56810) 2024-03-18T16:21:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.421+0000 7ffa33e00640 1 -- 172.21.15.102:0/2401911650 learned_addr learned my addr 172.21.15.102:0/2401911650 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.424+0000 7ffa39600640 1 -- 172.21.15.102:0/1469194854 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ffa24038020 con 0x7ffa3c0c5830 2024-03-18T16:21:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.425+0000 7ffa39600640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ffa1806bbc0 0x7ffa1806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.425+0000 7ffa33e00640 1 -- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 msgr2=0x55807188e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.425+0000 7ffa33e00640 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 0x55807188e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.425+0000 7ffa33e00640 1 -- 172.21.15.102:0/2401911650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 msgr2=0x55807188ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.425+0000 7ffa33e00640 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 0x55807188ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.425+0000 7ffa33e00640 1 -- 172.21.15.102:0/2401911650 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x5580714e02d0 con 0x5580718a5f40 2024-03-18T16:21:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.425+0000 7ffa41c00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ffa1806bbc0 0x7ffa1806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.425+0000 7ffa41c00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ffa1806bbc0 0x7ffa1806e080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c072da0 tx=0x7ffa34008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.427+0000 7ffa33e00640 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x558071886380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa240b1260 tx=0x7ffa2402eb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65f9be9346b80056 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.427+0000 7ffa2a000640 10 client.0 ms_handle_connect on v2:172.21.15.154:3301/0 2024-03-18T16:21:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.428+0000 7ffa39600640 1 -- 172.21.15.102:0/1469194854 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7ffa24043ab0 con 0x7ffa3c0c5830 2024-03-18T16:21:36.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.432+0000 7ffa2a000640 1 -- 172.21.15.102:0/2401911650 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24080f50 con 0x5580718a5f40 2024-03-18T16:21:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.432+0000 7ffa2a000640 1 -- 172.21.15.102:0/2401911650 <== mon.2 v2:172.21.15.154:3301/0 2 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7ffa24083df0 con 0x5580718a5f40 2024-03-18T16:21:36.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.432+0000 7ffa4680d740 1 -- 172.21.15.102:0/2401911650 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa2c018070 con 0x5580718a5f40 2024-03-18T16:21:36.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.434+0000 7ffa2a000640 1 -- 172.21.15.102:0/2401911650 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7ffa2404a210 con 0x5580718a5f40 2024-03-18T16:21:36.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.434+0000 7ffa4680d740 10 client.4346 fetch_fsmap learned FSMap version 7 2024-03-18T16:21:36.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.434+0000 7ffa4680d740 1 -- 172.21.15.102:0/2401911650 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x5580715aa6b0 con 0x5580718a5f40 2024-03-18T16:21:36.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.436+0000 7ffa2a000640 1 -- 172.21.15.102:0/2401911650 <== mon.2 v2:172.21.15.154:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2c018070 con 0x5580718a5f40 2024-03-18T16:21:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.451+0000 7ffa2a000640 1 -- 172.21.15.102:0/2401911650 <== mon.2 v2:172.21.15.154:3301/0 5 ==== fsmap(e 7) ==== 4712+0+0 (secure 0 0 0) 0x7ffa24034170 con 0x5580718a5f40 2024-03-18T16:21:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.451+0000 7ffa4680d740 10 client.4346 fetch_fsmap finished waiting for FSMap version 7 2024-03-18T16:21:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.451+0000 7ffa4680d740 10 client.4346 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T16:21:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.451+0000 7ffa4680d740 20 client.4346 populate_metadata read hostname 'smithi102' 2024-03-18T16:21:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.451+0000 7ffa4680d740 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] conn(0x558071865580 0x5580718c7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.451+0000 7ffa4680d740 4 client.4346 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:21:36.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.451+0000 7ffa4680d740 1 -- 172.21.15.102:0/2401911650 --> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580718292b0 con 0x558071865580 2024-03-18T16:21:36.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.460+0000 7ffa41200640 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] conn(0x558071865580 0x5580718c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.545+0000 7ffa41200640 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] conn(0x558071865580 0x5580718c7780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e17aae86f4d1349 server_cookie=2b3cd5e7a10bfb11 in_seq=0 out_seq=0 2024-03-18T16:21:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.545+0000 7ffa2a000640 10 client.4346 ms_handle_connect on v2:172.21.15.154:6834/2841941336 2024-03-18T16:21:36.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.666+0000 7ffa2a000640 1 -- 172.21.15.102:0/2401911650 <== mds.0 v2:172.21.15.154:6834/2841941336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5580718292b0 con 0x558071865580 2024-03-18T16:21:36.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.666+0000 7ffa2a000640 10 client.4346 handle_command_reply: tid=0 2024-03-18T16:21:36.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.666+0000 7ffa4680d740 1 client.4346 shutdown 2024-03-18T16:21:36.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.667+0000 7ffa4680d740 1 -- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 msgr2=0x558071886380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.667+0000 7ffa4680d740 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x558071886380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa240b1260 tx=0x7ffa2402eb60 comp rx=0 tx=0).stop 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.667+0000 7ffa4680d740 1 -- 172.21.15.102:0/2401911650 shutdown_connections 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.667+0000 7ffa4680d740 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x558071886380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.667+0000 7ffa4680d740 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] conn(0x558071865580 0x5580718c7780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.667+0000 7ffa4680d740 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 0x55807188ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.667+0000 7ffa4680d740 1 --2- 172.21.15.102:0/2401911650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 0x55807188e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.667+0000 7ffa2a000640 0 client.4346 ms_handle_reset on v2:172.21.15.154:6834/2841941336 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.667+0000 7ffa4680d740 1 -- 172.21.15.102:0/2401911650 >> 172.21.15.102:0/2401911650 conn(0x558071867210 msgr2=0x558071861a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.668+0000 7ffa4680d740 1 -- 172.21.15.102:0/2401911650 shutdown_connections 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.668+0000 7ffa4680d740 1 -- 172.21.15.102:0/2401911650 wait complete. 2024-03-18T16:21:36.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.668+0000 7ffa4680d740 20 client.4346 trim_cache size 0 max 16384 2024-03-18T16:21:36.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.726+0000 7ffa4680d740 1 Processor -- start 2024-03-18T16:21:36.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.726+0000 7ffa4680d740 1 -- start start 2024-03-18T16:21:36.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.729+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x5580718613d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.729+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 0x5580718ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.729+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 0x5580718abb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.729+0000 7ffa4680d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c201290 con 0x5580718a8b50 2024-03-18T16:21:36.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.729+0000 7ffa4680d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5580718bc590 con 0x5580718a8770 2024-03-18T16:21:36.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.729+0000 7ffa4680d740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x5580718bcb50 con 0x5580718a5f40 2024-03-18T16:21:36.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.730+0000 7ffa41200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x5580718613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.730+0000 7ffa41c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 0x5580718abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.730+0000 7ffa41200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x5580718613d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56826/0 (socket says 172.21.15.102:56826) 2024-03-18T16:21:36.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.730+0000 7ffa41200640 1 -- 172.21.15.102:0/4189596138 learned_addr learned my addr 172.21.15.102:0/4189596138 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:36.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.730+0000 7ffa33e00640 1 --2- 172.21.15.102:0/4189596138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 0x5580718ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.733+0000 7ffa33e00640 1 -- 172.21.15.102:0/4189596138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 msgr2=0x5580718613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:36.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.733+0000 7ffa33e00640 1 --2- 172.21.15.102:0/4189596138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x5580718613d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.733+0000 7ffa33e00640 1 -- 172.21.15.102:0/4189596138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 msgr2=0x5580718abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:36.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.733+0000 7ffa33e00640 1 --2- 172.21.15.102:0/4189596138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 0x5580718abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.733+0000 7ffa33e00640 1 -- 172.21.15.102:0/4189596138 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55807185ce80 con 0x5580718a8770 2024-03-18T16:21:36.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.735+0000 7ffa33e00640 1 --2- 172.21.15.102:0/4189596138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 0x5580718ab5f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x5580718b63f0 tx=0x7ffa2403f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f741ea3a3404054 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:36.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.738+0000 7ffa30c00640 1 -- 172.21.15.102:0/4189596138 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2407f2c0 con 0x5580718a8770 2024-03-18T16:21:36.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.739+0000 7ffa30c00640 1 -- 172.21.15.102:0/4189596138 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa24041e40 con 0x5580718a8770 2024-03-18T16:21:36.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.739+0000 7ffa30c00640 1 -- 172.21.15.102:0/4189596138 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24080ae0 con 0x5580718a8770 2024-03-18T16:21:36.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.740+0000 7ffa4680d740 1 -- 172.21.15.102:0/4189596138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 msgr2=0x5580718ab5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.740+0000 7ffa4680d740 1 --2- 172.21.15.102:0/4189596138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 0x5580718ab5f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x5580718b63f0 tx=0x7ffa2403f000 comp rx=0 tx=0).stop 2024-03-18T16:21:36.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.740+0000 7ffa4680d740 1 -- 172.21.15.102:0/4189596138 shutdown_connections 2024-03-18T16:21:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.740+0000 7ffa4680d740 1 --2- 172.21.15.102:0/4189596138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8b50 0x5580718abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.740+0000 7ffa4680d740 1 --2- 172.21.15.102:0/4189596138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a8770 0x5580718ab5f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.740+0000 7ffa4680d740 1 --2- 172.21.15.102:0/4189596138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a5f40 0x5580718613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.740+0000 7ffa4680d740 1 -- 172.21.15.102:0/4189596138 >> 172.21.15.102:0/4189596138 conn(0x558071867210 msgr2=0x5580718676d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.740+0000 7ffa4680d740 1 -- 172.21.15.102:0/4189596138 shutdown_connections 2024-03-18T16:21:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.740+0000 7ffa4680d740 1 -- 172.21.15.102:0/4189596138 wait complete. 2024-03-18T16:21:36.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.742+0000 7ffa4680d740 1 -- start start 2024-03-18T16:21:36.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.743+0000 7ffa4680d740 1 Processor -- start 2024-03-18T16:21:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.745+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a5f40 0x5580718ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.745+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x5580718cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.746+0000 7ffa4680d740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a8b50 0x5580718cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.746+0000 7ffa4680d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x558071865400 con 0x5580718a8770 2024-03-18T16:21:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.746+0000 7ffa4680d740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x558071887320 con 0x5580718a5f40 2024-03-18T16:21:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.746+0000 7ffa4680d740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x558071865a50 con 0x5580718a8b50 2024-03-18T16:21:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.746+0000 7ffa4680d740 10 client.0 _ll_register_callbacks cb 0x7ffa24051070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:21:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.746+0000 7ffa33e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x5580718cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.746+0000 7ffa41c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a8b50 0x5580718cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.746+0000 7ffa41200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a5f40 0x5580718ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.747+0000 7ffa33e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x5580718cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59378/0 (socket says 172.21.15.102:59378) 2024-03-18T16:21:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.747+0000 7ffa33e00640 1 -- 172.21.15.102:0/3073735522 learned_addr learned my addr 172.21.15.102:0/3073735522 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:36.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.750+0000 7ffa33e00640 1 -- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a8b50 msgr2=0x5580718cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:36.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.750+0000 7ffa33e00640 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a8b50 0x5580718cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.750+0000 7ffa33e00640 1 -- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a5f40 msgr2=0x5580718ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:36.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.750+0000 7ffa33e00640 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a5f40 0x5580718ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.750+0000 7ffa33e00640 1 -- 172.21.15.102:0/3073735522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558071838750 con 0x5580718a8770 2024-03-18T16:21:36.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.753+0000 7ffa33e00640 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x5580718cb7a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa24080e80 tx=0x7ffa24031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d1cac54b679bf7f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:36.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.753+0000 7ffa2a000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-18T16:21:36.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.761+0000 7ffa2a000640 1 -- 172.21.15.102:0/3073735522 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24080cc0 con 0x5580718a8770 2024-03-18T16:21:36.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.761+0000 7ffa2a000640 1 -- 172.21.15.102:0/3073735522 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7ffa2407f480 con 0x5580718a8770 2024-03-18T16:21:36.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.761+0000 7ffa4680d740 1 -- 172.21.15.102:0/3073735522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5580717d9190 con 0x5580718a8770 2024-03-18T16:21:36.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.765+0000 7ffa2a000640 1 -- 172.21.15.102:0/3073735522 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7ffa24053030 con 0x5580718a8770 2024-03-18T16:21:36.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.765+0000 7ffa4680d740 10 client.4281 fetch_fsmap learned FSMap version 7 2024-03-18T16:21:36.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.765+0000 7ffa4680d740 1 -- 172.21.15.102:0/3073735522 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x5580718bbb60 con 0x5580718a8770 2024-03-18T16:21:36.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.769+0000 7ffa2a000640 1 -- 172.21.15.102:0/3073735522 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5580717d9190 con 0x5580718a8770 2024-03-18T16:21:36.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.792+0000 7ffa2a000640 1 -- 172.21.15.102:0/3073735522 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 7) ==== 4712+0+0 (secure 0 0 0) 0x7ffa240813e0 con 0x5580718a8770 2024-03-18T16:21:36.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.792+0000 7ffa4680d740 10 client.4281 fetch_fsmap finished waiting for FSMap version 7 2024-03-18T16:21:36.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.792+0000 7ffa4680d740 10 client.4281 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T16:21:36.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.792+0000 7ffa4680d740 20 client.4281 populate_metadata read hostname 'smithi102' 2024-03-18T16:21:36.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.792+0000 7ffa4680d740 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] conn(0x558071926400 0x5580719267e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:36.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.792+0000 7ffa4680d740 4 client.4281 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:21:36.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.792+0000 7ffa4680d740 1 -- 172.21.15.102:0/3073735522 --> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] -- command(tid 0: {"prefix": "status"}) -- 0x5580717d34e0 con 0x558071926400 2024-03-18T16:21:36.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.793+0000 7ffa41200640 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] conn(0x558071926400 0x5580719267e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:36.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.797+0000 7ffa41200640 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] conn(0x558071926400 0x5580719267e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9a7339979bfe69e server_cookie=8028b3dbdfcf9ef3 in_seq=0 out_seq=0 2024-03-18T16:21:36.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.797+0000 7ffa2a000640 10 client.4281 ms_handle_connect on v2:172.21.15.154:6834/2841941336 2024-03-18T16:21:36.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa2a000640 1 -- 172.21.15.102:0/3073735522 <== mds.0 v2:172.21.15.154:6834/2841941336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5580717d34e0 con 0x558071926400 2024-03-18T16:21:36.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa2a000640 10 client.4281 handle_command_reply: tid=0 2024-03-18T16:21:36.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa4680d740 1 client.4281 shutdown 2024-03-18T16:21:36.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa4680d740 1 -- 172.21.15.102:0/3073735522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 msgr2=0x5580718cb7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa4680d740 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x5580718cb7a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa24080e80 tx=0x7ffa24031000 comp rx=0 tx=0).stop 2024-03-18T16:21:36.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa4680d740 1 -- 172.21.15.102:0/3073735522 shutdown_connections 2024-03-18T16:21:36.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa4680d740 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x5580718a8b50 0x5580718cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa4680d740 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:6834/2841941336,v1:172.21.15.154:6835/2841941336] conn(0x558071926400 0x5580719267e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa4680d740 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5580718a8770 0x5580718cb7a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.816+0000 7ffa2a000640 0 client.4281 ms_handle_reset on v2:172.21.15.154:6834/2841941336 2024-03-18T16:21:36.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.815+0000 7ffa4680d740 1 --2- 172.21.15.102:0/3073735522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580718a5f40 0x5580718ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.816+0000 7ffa4680d740 1 -- 172.21.15.102:0/3073735522 >> 172.21.15.102:0/3073735522 conn(0x558071867210 msgr2=0x5580718613d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:36.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.816+0000 7ffa4680d740 1 -- 172.21.15.102:0/3073735522 shutdown_connections 2024-03-18T16:21:36.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.816+0000 7ffa4680d740 1 -- 172.21.15.102:0/3073735522 wait complete. 2024-03-18T16:21:36.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.816+0000 7ffa4680d740 20 client.4281 trim_cache size 0 max 16384 2024-03-18T16:21:36.819 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:21:36.819 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "0179d728-7c99-4c29-b345-3950a2c4b164", 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 4270, 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 13.241468656, 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 7, 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 27, 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 27, 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 23.102738250000002 2024-03-18T16:21:36.820 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:21:36.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 -- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ffa1806bbc0 msgr2=0x7ffa1806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ffa1806bbc0 0x7ffa1806e080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c072da0 tx=0x7ffa34008040 comp rx=0 tx=0).stop 2024-03-18T16:21:36.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 -- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 msgr2=0x7ffa3c13dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 0x7ffa3c13dcf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffa2402bee0 tx=0x7ffa240046c0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 -- 172.21.15.102:0/1469194854 shutdown_connections 2024-03-18T16:21:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ffa1806bbc0 0x7ffa1806e080 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa3c1582f0 0x7ffa3c1f8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa3c0c5830 0x7ffa3c13dcf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 --2- 172.21.15.102:0/1469194854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ffa3c0c5450 0x7ffa3c0b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 -- 172.21.15.102:0/1469194854 >> 172.21.15.102:0/1469194854 conn(0x7ffa3c0700d0 msgr2=0x7ffa3c072fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 -- 172.21.15.102:0/1469194854 shutdown_connections 2024-03-18T16:21:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:36.818+0000 7ffa43c00640 1 -- 172.21.15.102:0/1469194854 wait complete. 2024-03-18T16:21:36.849 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T16:21:36.850 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:21:36.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-18T16:21:37.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.074+0000 7fd455c00640 1 Processor -- start 2024-03-18T16:21:37.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.074+0000 7fd455c00640 1 -- start start 2024-03-18T16:21:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.076+0000 7fd455c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd4500c5450 0x7fd450141580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.076+0000 7fd455c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd450072050 con 0x7fd4501582f0 2024-03-18T16:21:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.076+0000 7fd455c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4500722f0 con 0x7fd4500c5830 2024-03-18T16:21:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.076+0000 7fd455c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd45015c8b0 con 0x7fd4500c5450 2024-03-18T16:21:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.077+0000 7fd44e000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd4500c5830 0x7fd450071940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33152/0 (socket says 172.21.15.102:33152) 2024-03-18T16:21:37.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.077+0000 7fd44e000640 1 -- 172.21.15.102:0/1996967727 learned_addr learned my addr 172.21.15.102:0/1996967727 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:37.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.130+0000 7fd44ea00640 1 --2- 172.21.15.102:0/1996967727 >> v2:172.21.15.102:3300/0 conn(0x7fd4500c5450 0x7fd450141580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.135+0000 7fd44ea00640 1 -- 172.21.15.102:0/1996967727 >> v1:172.21.15.154:6789/0 conn(0x7fd4500c5830 legacy=0x7fd450071940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.135+0000 7fd44ea00640 1 -- 172.21.15.102:0/1996967727 >> v1:172.21.15.102:6789/0 conn(0x7fd4501582f0 legacy=0x7fd450151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.135+0000 7fd44ea00640 1 -- 172.21.15.102:0/1996967727 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45015baf0 con 0x7fd4500c5450 2024-03-18T16:21:37.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.141+0000 7fd44ea00640 1 --2- 172.21.15.102:0/1996967727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5450 0x7fd450141580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd440009d90 tx=0x7fd44002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f744194552bbd68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:37.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.149+0000 7fd44d600640 1 -- 172.21.15.102:0/1996967727 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440039070 con 0x7fd4500c5450 2024-03-18T16:21:37.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.149+0000 7fd44d600640 1 -- 172.21.15.102:0/1996967727 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd44002ccf0 con 0x7fd4500c5450 2024-03-18T16:21:37.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.149+0000 7fd44d600640 1 -- 172.21.15.102:0/1996967727 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd440034660 con 0x7fd4500c5450 2024-03-18T16:21:37.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.150+0000 7fd455c00640 1 -- 172.21.15.102:0/1996967727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5450 msgr2=0x7fd450141580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.150+0000 7fd455c00640 1 --2- 172.21.15.102:0/1996967727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5450 0x7fd450141580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd440009d90 tx=0x7fd44002bf40 comp rx=0 tx=0).stop 2024-03-18T16:21:37.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.150+0000 7fd455c00640 1 -- 172.21.15.102:0/1996967727 shutdown_connections 2024-03-18T16:21:37.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.150+0000 7fd455c00640 1 --2- 172.21.15.102:0/1996967727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5450 0x7fd450141580 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.150+0000 7fd455c00640 1 -- 172.21.15.102:0/1996967727 >> 172.21.15.102:0/1996967727 conn(0x7fd4500700d0 msgr2=0x7fd45006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:37.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.150+0000 7fd455c00640 1 -- 172.21.15.102:0/1996967727 shutdown_connections 2024-03-18T16:21:37.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.150+0000 7fd455c00640 1 -- 172.21.15.102:0/1996967727 wait complete. 2024-03-18T16:21:37.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.153+0000 7fd455c00640 1 Processor -- start 2024-03-18T16:21:37.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.154+0000 7fd455c00640 1 -- start start 2024-03-18T16:21:37.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.155+0000 7fd455c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4500c5450 0x7fd45015b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.155+0000 7fd455c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5830 0x7fd4500c6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.155+0000 7fd455c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4501582f0 0x7fd4500c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.155+0000 7fd455c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd45015c0b0 con 0x7fd4500c5830 2024-03-18T16:21:37.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.155+0000 7fd455c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd450141ac0 con 0x7fd4501582f0 2024-03-18T16:21:37.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.155+0000 7fd455c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd4501525e0 con 0x7fd4500c5450 2024-03-18T16:21:37.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.156+0000 7fd44e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5830 0x7fd4500c6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.156+0000 7fd44e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5830 0x7fd4500c6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59394/0 (socket says 172.21.15.102:59394) 2024-03-18T16:21:37.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.156+0000 7fd44e000640 1 -- 172.21.15.102:0/2205813618 learned_addr learned my addr 172.21.15.102:0/2205813618 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:37.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.160+0000 7fd44e000640 1 -- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4500c5450 msgr2=0x7fd45015b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.160+0000 7fd44e000640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4500c5450 0x7fd45015b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.160+0000 7fd44e000640 1 -- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4501582f0 msgr2=0x7fd4500c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.160+0000 7fd44e000640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4501582f0 0x7fd4500c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.160+0000 7fd44e000640 1 -- 172.21.15.102:0/2205813618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4501530d0 con 0x7fd4500c5830 2024-03-18T16:21:37.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.164+0000 7fd44e000640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5830 0x7fd4500c6f90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00c500 tx=0x7fd43c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:37.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.172+0000 7fd446a00640 1 -- 172.21.15.102:0/2205813618 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c017070 con 0x7fd4500c5830 2024-03-18T16:21:37.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.172+0000 7fd446a00640 1 -- 172.21.15.102:0/2205813618 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd43c005ce0 con 0x7fd4500c5830 2024-03-18T16:21:37.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.172+0000 7fd446a00640 1 -- 172.21.15.102:0/2205813618 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0072f0 con 0x7fd4500c5830 2024-03-18T16:21:37.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.172+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4501533b0 con 0x7fd4500c5830 2024-03-18T16:21:37.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.173+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd450153970 con 0x7fd4500c5830 2024-03-18T16:21:37.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.174+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd450074fd0 con 0x7fd4500c5830 2024-03-18T16:21:37.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.200+0000 7fd446a00640 1 -- 172.21.15.102:0/2205813618 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd43c01a070 con 0x7fd4500c5830 2024-03-18T16:21:37.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.201+0000 7fd446a00640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd43006bc50 0x7fd43006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.202+0000 7fd44ea00640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd43006bc50 0x7fd43006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.202+0000 7fd44ea00640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd43006bc50 0x7fd43006e110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd440036000 tx=0x7fd44002e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:37.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.508+0000 7fd446a00640 1 -- 172.21.15.102:0/2205813618 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7fd43c01de30 con 0x7fd4500c5830 2024-03-18T16:21:37.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.509+0000 7fd446a00640 1 -- 172.21.15.102:0/2205813618 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd43c058720 con 0x7fd4500c5830 2024-03-18T16:21:37.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.759+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd4500c93c0 con 0x7fd4500c5830 2024-03-18T16:21:37.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.775+0000 7fd446a00640 1 -- 172.21.15.102:0/2205813618 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7fd43c05c7c0 con 0x7fd4500c5830 2024-03-18T16:21:37.777 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T16:21:37.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.777+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd43006bc50 msgr2=0x7fd43006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.777+0000 7fd455c00640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd43006bc50 0x7fd43006e110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd440036000 tx=0x7fd44002e000 comp rx=0 tx=0).stop 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.777+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5830 msgr2=0x7fd4500c6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.777+0000 7fd455c00640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5830 0x7fd4500c6f90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd43c00c500 tx=0x7fd43c00c9c0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.777+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 shutdown_connections 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.777+0000 7fd455c00640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd43006bc50 0x7fd43006e110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.777+0000 7fd455c00640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4501582f0 0x7fd4500c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.777+0000 7fd455c00640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4500c5830 0x7fd4500c6f90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.777+0000 7fd455c00640 1 --2- 172.21.15.102:0/2205813618 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd4500c5450 0x7fd45015b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.778+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 >> 172.21.15.102:0/2205813618 conn(0x7fd4500700d0 msgr2=0x7fd4500c98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.778+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 shutdown_connections 2024-03-18T16:21:37.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37.778+0000 7fd455c00640 1 -- 172.21.15.102:0/2205813618 wait complete. 2024-03-18T16:21:37.799 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-18T16:21:37.799 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T16:21:37.799 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-18T16:21:37.799 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-18T16:21:37.799 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:21:37.799 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-18T16:21:37.799 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T16:21:37.799 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T16:21:37.800 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T16:21:37.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:37.800 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-18T16:21:37.849 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T16:21:37.850 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:21:37.850 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T16:21:37.850 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:21:37.850 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-18T16:21:37.850 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 41734sec preferred_lft 41734sec 2024-03-18T16:21:37.850 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-18T16:21:37.850 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-18T16:21:37.851 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T16:21:37.851 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:21:37.851 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-18T16:21:37.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2024-03-18T16:21:37.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2024-03-18T16:21:37.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2024-03-18T16:21:37.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T16:21:37.852 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-18T16:21:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:21:38.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:21:38.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:38.340 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T16:21:38.409 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-18T16:21:38.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:38.413 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-18T16:21:38.472 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-18T16:21:38.472 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T16:21:38.472 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-18T16:21:38.472 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-18T16:21:38.472 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T16:21:38.473 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:21:38.473 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-18T16:21:38.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-18T16:21:38.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-18T16:21:38.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-18T16:21:38.473 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-18T16:21:38.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:21:39.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:21:39.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:39.180 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-18T16:21:39.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:39.220 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-03-18T16:21:39.306 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:21:39.306 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-18T16:21:39.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:21:39.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-18T16:21:39.306 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-18T16:21:39.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:21:39.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:21:39.537 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-18T16:21:39.537 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:21:39.537 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-18T16:21:39.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-18T16:21:39.537 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T16:21:39.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-18T16:21:39.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-18T16:21:39.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-18T16:21:39.538 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-18T16:21:39.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:21:39.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:21:39.854 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:21:39.854 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-18T16:21:39.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-18T16:21:39.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T16:21:39.855 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-18T16:21:39.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:21:40.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:21:40.070 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-18T16:21:40.070 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T16:21:40.070 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:21:40.110 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T16:21:40.110 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T16:21:40.110 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:21:40.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-18T16:21:40.323 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T16:21:40.371 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/selinux 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T16:21:40.372 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T16:21:40.373 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-03-18T16:21:40.374 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:21:40.374 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:21:40.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:40.375 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-18T16:21:40.422 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T16:21:40.422 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-18T16:21:40.426 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-18T16:21:40.498 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/selinux 2024-03-18T16:21:40.537 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T16:21:40.538 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T16:21:40.539 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-03-18T16:21:40.539 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:21:40.539 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:21:40.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:40.540 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-18T16:21:40.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:2024-03-18T16:21:40.612+0000 7f47293d9480 -1 init, newargv = 0x55da4b409990 newargc=15 2024-03-18T16:21:40.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:ceph-fuse[42266]: starting ceph client 2024-03-18T16:21:40.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:ceph-fuse[42266]: starting fuse 2024-03-18T16:21:41.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-18T16:21:41.824 DEBUG:teuthology.orchestra.run.smithi102:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/proc 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/sys 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/dev 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/security 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/shm 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/pts 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/run 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/cgroup 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/pstore 2024-03-18T16:21:41.880 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/bpf 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/ 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/selinux 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/mqueue 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/hugepages 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/debug 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/tracing 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/fs/fuse/connections 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/sys/kernel/config 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/run/user/1000 2024-03-18T16:21:41.881 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph 2024-03-18T16:21:41.882 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-0 2024-03-18T16:21:41.882 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-1 2024-03-18T16:21:41.882 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-2 2024-03-18T16:21:41.882 INFO:teuthology.orchestra.run.smithi102.stdout:/var/lib/ceph/osd/ceph-3 2024-03-18T16:21:41.882 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns 2024-03-18T16:21:41.882 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:21:41.882 INFO:teuthology.orchestra.run.smithi102.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:21:41.882 INFO:teuthology.orchestra.run.smithi102.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-18T16:21:41.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:41.883 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-18T16:21:41.935 INFO:teuthology.orchestra.run.smithi102.stdout:42 2024-03-18T16:21:41.935 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-18T16:21:41.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo stdin-killer -- python3 -c ' 2024-03-18T16:21:41.935 DEBUG:teuthology.orchestra.run.smithi102:> import glob 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> import re 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> import os 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> import subprocess 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> def _find_admin_socket(client_name): 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> files = glob.glob(asok_path) 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> # Given a non-glob path, it better be there 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> if "*" not in asok_path: 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> assert(len(files) == 1) 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> return files[0] 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> for f in files: 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T16:21:41.936 DEBUG:teuthology.orchestra.run.smithi102:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T16:21:41.937 DEBUG:teuthology.orchestra.run.smithi102:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T16:21:41.937 DEBUG:teuthology.orchestra.run.smithi102:> contents = proc_f.read() 2024-03-18T16:21:41.937 DEBUG:teuthology.orchestra.run.smithi102:> if mountpoint in contents: 2024-03-18T16:21:41.937 DEBUG:teuthology.orchestra.run.smithi102:> return f 2024-03-18T16:21:41.937 DEBUG:teuthology.orchestra.run.smithi102:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T16:21:41.937 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-18T16:21:41.937 DEBUG:teuthology.orchestra.run.smithi102:> print(_find_admin_socket("client.admin")) 2024-03-18T16:21:41.937 DEBUG:teuthology.orchestra.run.smithi102:> ' 2024-03-18T16:21:42.113 INFO:teuthology.orchestra.run.smithi102.stdout:/var/run/ceph/ceph-client.admin.42266.asok 2024-03-18T16:21:42.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:21:42.132 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.42266.asok 2024-03-18T16:21:42.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:42.132 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.42266.asok status 2024-03-18T16:21:42.367 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:21:42.367 INFO:teuthology.orchestra.run.smithi102.stdout: "metadata": { 2024-03-18T16:21:42.367 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T16:21:42.367 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "entity_id": "admin", 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "hostname": "smithi102", 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "pid": "42266", 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "root": "/" 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_count": 0, 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_pinned_count": 0, 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 4358, 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "inst": { 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "name": { 2024-03-18T16:21:42.368 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "client", 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4358 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 1443975527 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "172.21.15.102:0", 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 1443975527 2024-03-18T16:21:42.369 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-18T16:21:42.370 INFO:teuthology.orchestra.run.smithi102.stdout: "inst_str": "client.4358 172.21.15.102:0/1443975527", 2024-03-18T16:21:42.370 INFO:teuthology.orchestra.run.smithi102.stdout: "addr_str": "172.21.15.102:0/1443975527", 2024-03-18T16:21:42.370 INFO:teuthology.orchestra.run.smithi102.stdout: "inode_count": 1, 2024-03-18T16:21:42.370 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_epoch": 7, 2024-03-18T16:21:42.370 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch": 27, 2024-03-18T16:21:42.370 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch_barrier": 0, 2024-03-18T16:21:42.370 INFO:teuthology.orchestra.run.smithi102.stdout: "blocklisted": false, 2024-03-18T16:21:42.370 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs" 2024-03-18T16:21:42.370 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:21:42.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:42.382 DEBUG:teuthology.orchestra.run.smithi102:> stat --file-system '--printf=%T 2024-03-18T16:21:42.382 DEBUG:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:21:42.444 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk 2024-03-18T16:21:42.444 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:21:42.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:42.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:21:42.623 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-18T16:21:42.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:42 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T16:21:42.982 INFO:teuthology.orchestra.run.smithi102.stdout:# file: . 2024-03-18T16:21:42.982 INFO:teuthology.orchestra.run.smithi102.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-18T16:21:42.982 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:42.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:21:42.994 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:21:42.994 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T16:21:43.045 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.298+0000 7f05b7200640 1 Processor -- start 2024-03-18T16:21:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.298+0000 7f05b7200640 1 -- start start 2024-03-18T16:21:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.300+0000 7f05b7200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f05b8154750 0x7f05b8072f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.300+0000 7f05b7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b8073480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.300+0000 7f05b7200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05b80c2ea0 con 0x7f05b8070860 2024-03-18T16:21:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.300+0000 7f05b7200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f05b80c7b80 con 0x7f05b8154750 2024-03-18T16:21:43.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.300+0000 7f05b7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f05b80c3160 con 0x7f05b815ac50 2024-03-18T16:21:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.301+0000 7f05b6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b8073480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.301+0000 7f05b5e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f05b8154750 0x7f05b8072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.301+0000 7f05b6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b8073480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52808/0 (socket says 172.21.15.102:52808) 2024-03-18T16:21:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.301+0000 7f05b6800640 1 -- 172.21.15.102:0/1911915224 learned_addr learned my addr 172.21.15.102:0/1911915224 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.301+0000 7f05afe00640 1 --1- 172.21.15.102:0/1911915224 >> v1:172.21.15.154:6789/0 conn(0x7f05b8070860 0x7f05b815a540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33212/0 (socket says 172.21.15.102:33212) 2024-03-18T16:21:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.304+0000 7f05b6800640 1 -- 172.21.15.102:0/1911915224 >> v2:172.21.15.154:3300/0 conn(0x7f05b8154750 msgr2=0x7f05b8072f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.304+0000 7f05b6800640 1 --2- 172.21.15.102:0/1911915224 >> v2:172.21.15.154:3300/0 conn(0x7f05b8154750 0x7f05b8072f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.304+0000 7f05b6800640 1 -- 172.21.15.102:0/1911915224 >> v1:172.21.15.154:6789/0 conn(0x7f05b8070860 legacy=0x7f05b815a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.304+0000 7f05b6800640 1 -- 172.21.15.102:0/1911915224 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b80c3440 con 0x7f05b815ac50 2024-03-18T16:21:43.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.306+0000 7f05b6800640 1 --2- 172.21.15.102:0/1911915224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b8073480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009e50 tx=0x7f05b002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=678d714c4fde566e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:43.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.311+0000 7f05b5400640 1 -- 172.21.15.102:0/1911915224 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b002cb90 con 0x7f05b815ac50 2024-03-18T16:21:43.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.311+0000 7f05b5400640 1 -- 172.21.15.102:0/1911915224 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b002cd30 con 0x7f05b815ac50 2024-03-18T16:21:43.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.311+0000 7f05b5400640 1 -- 172.21.15.102:0/1911915224 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0034610 con 0x7f05b815ac50 2024-03-18T16:21:43.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.312+0000 7f05b7200640 1 -- 172.21.15.102:0/1911915224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 msgr2=0x7f05b8073480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.312+0000 7f05b7200640 1 --2- 172.21.15.102:0/1911915224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b8073480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009e50 tx=0x7f05b002bff0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.312+0000 7f05b7200640 1 -- 172.21.15.102:0/1911915224 shutdown_connections 2024-03-18T16:21:43.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.312+0000 7f05b7200640 1 --2- 172.21.15.102:0/1911915224 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b8073480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.312+0000 7f05b7200640 1 --2- 172.21.15.102:0/1911915224 >> v2:172.21.15.154:3300/0 conn(0x7f05b8154750 0x7f05b8072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.312+0000 7f05b7200640 1 -- 172.21.15.102:0/1911915224 >> 172.21.15.102:0/1911915224 conn(0x7f05b80ccb50 msgr2=0x7f05b80ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:43.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.312+0000 7f05b7200640 1 -- 172.21.15.102:0/1911915224 shutdown_connections 2024-03-18T16:21:43.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.313+0000 7f05b7200640 1 -- 172.21.15.102:0/1911915224 wait complete. 2024-03-18T16:21:43.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.314+0000 7f05b7200640 1 Processor -- start 2024-03-18T16:21:43.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.315+0000 7f05b7200640 1 -- start start 2024-03-18T16:21:43.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.316+0000 7f05b7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b8070860 0x7f05b814ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.316+0000 7f05b7200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8154750 0x7f05b81574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.316+0000 7f05b7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b81f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.316+0000 7f05b7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f05b8070280 con 0x7f05b8070860 2024-03-18T16:21:43.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.316+0000 7f05b7200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f05b80c3a00 con 0x7f05b8154750 2024-03-18T16:21:43.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.316+0000 7f05b7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f05b8070400 con 0x7f05b815ac50 2024-03-18T16:21:43.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.316+0000 7f05afe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8154750 0x7f05b81574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.316+0000 7f05b5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b8070860 0x7f05b814ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.317+0000 7f05afe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8154750 0x7f05b81574d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59696/0 (socket says 172.21.15.102:59696) 2024-03-18T16:21:43.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.317+0000 7f05b6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b81f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.317+0000 7f05afe00640 1 -- 172.21.15.102:0/2853787764 learned_addr learned my addr 172.21.15.102:0/2853787764 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:43.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.320+0000 7f05afe00640 1 -- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 msgr2=0x7f05b81f3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.320+0000 7f05afe00640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b81f3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.320+0000 7f05afe00640 1 -- 172.21.15.102:0/2853787764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b8070860 msgr2=0x7f05b814ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.320+0000 7f05afe00640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b8070860 0x7f05b814ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.320+0000 7f05afe00640 1 -- 172.21.15.102:0/2853787764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b8151540 con 0x7f05b8154750 2024-03-18T16:21:43.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.322+0000 7f05afe00640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8154750 0x7f05b81574d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c400 tx=0x7f05a000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:43.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.326+0000 7f05ae000640 1 -- 172.21.15.102:0/2853787764 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a00074e0 con 0x7f05b8154750 2024-03-18T16:21:43.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.326+0000 7f05ae000640 1 -- 172.21.15.102:0/2853787764 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a0007680 con 0x7f05b8154750 2024-03-18T16:21:43.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.326+0000 7f05ae000640 1 -- 172.21.15.102:0/2853787764 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0004a00 con 0x7f05b8154750 2024-03-18T16:21:43.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.326+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b8150a00 con 0x7f05b8154750 2024-03-18T16:21:43.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.326+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b8150ca0 con 0x7f05b8154750 2024-03-18T16:21:43.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.327+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05b8074fd0 con 0x7f05b8154750 2024-03-18T16:21:43.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.352+0000 7f05ae000640 1 -- 172.21.15.102:0/2853787764 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f05a001a020 con 0x7f05b8154750 2024-03-18T16:21:43.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.352+0000 7f05ae000640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f059406bc20 0x7f059406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.353+0000 7f05ae000640 1 -- 172.21.15.102:0/2853787764 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f05a008abc0 con 0x7f05b8154750 2024-03-18T16:21:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.353+0000 7f05b5e00640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f059406bc20 0x7f059406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.354+0000 7f05b5e00640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f059406bc20 0x7f059406e0e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009b40 tx=0x7f05a8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.593+0000 7f05ae000640 1 -- 172.21.15.102:0/2853787764 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f05a0021070 con 0x7f05b8154750 2024-03-18T16:21:43.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.800+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f05b8070060 con 0x7f05b8154750 2024-03-18T16:21:43.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.840+0000 7f05ae000640 1 -- 172.21.15.102:0/2853787764 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18574 (secure 0 0 0) 0x7f05a0058e00 con 0x7f05b8154750 2024-03-18T16:21:43.842 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:43.842 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:30.100713+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f059406bc20 msgr2=0x7f059406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f059406bc20 0x7f059406e0e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009b40 tx=0x7f05a8009190 comp rx=0 tx=0).stop 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8154750 msgr2=0x7f05b81574d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8154750 0x7f05b81574d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c400 tx=0x7f05a000c8c0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 shutdown_connections 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f059406bc20 0x7f059406e0e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05b815ac50 0x7f05b81f3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05b8154750 0x7f05b81574d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 --2- 172.21.15.102:0/2853787764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05b8070860 0x7f05b814ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.842+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 >> 172.21.15.102:0/2853787764 conn(0x7f05b80ccb50 msgr2=0x7f05b8153380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.843+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 shutdown_connections 2024-03-18T16:21:43.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:43.843+0000 7f05b7200640 1 -- 172.21.15.102:0/2853787764 wait complete. 2024-03-18T16:21:43.863 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:21:05.204661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:21:19.524753+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:21:21.031843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:21:26.818690+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:21:43.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:21:44.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.075+0000 7f9dfd400640 1 Processor -- start 2024-03-18T16:21:44.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.075+0000 7f9dfd400640 1 -- start start 2024-03-18T16:21:44.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.076+0000 7f9dfd400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9df8141700 0x7f9df80723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.076+0000 7f9dfd400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9df806ba70 0x7f9df806be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.076+0000 7f9dfd400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df8158390 con 0x7f9df80c4fb0 2024-03-18T16:21:44.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.076+0000 7f9dfd400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9df8141ae0 con 0x7f9df8141700 2024-03-18T16:21:44.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.076+0000 7f9dfd400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9df8158670 con 0x7f9df806ba70 2024-03-18T16:21:44.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.077+0000 7f9df5e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9df806ba70 0x7f9df806be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.077+0000 7f9df6800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9df8141700 0x7f9df80723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.077+0000 7f9df7200640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f9df80c4fb0 0x7f9df815bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33218/0 (socket says 172.21.15.102:33218) 2024-03-18T16:21:44.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.078+0000 7f9df7200640 1 -- 172.21.15.102:0/2486430499 learned_addr learned my addr 172.21.15.102:0/2486430499 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:44.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.080+0000 7f9df5e00640 1 -- 172.21.15.102:0/2486430499 >> v2:172.21.15.102:3300/0 conn(0x7f9df8141700 msgr2=0x7f9df80723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:44.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.080+0000 7f9df5e00640 1 --2- 172.21.15.102:0/2486430499 >> v2:172.21.15.102:3300/0 conn(0x7f9df8141700 0x7f9df80723b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.080+0000 7f9df5e00640 1 -- 172.21.15.102:0/2486430499 >> v1:172.21.15.154:6789/0 conn(0x7f9df80c4fb0 legacy=0x7f9df815bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.080+0000 7f9df5e00640 1 -- 172.21.15.102:0/2486430499 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df8158950 con 0x7f9df806ba70 2024-03-18T16:21:44.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.084+0000 7f9df5e00640 1 --2- 172.21.15.102:0/2486430499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 0x7f9df806be90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9df8072d90 tx=0x7f9de802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a81e98ff41257059 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:44.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.088+0000 7f9df5400640 1 -- 172.21.15.102:0/2486430499 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8034070 con 0x7f9df806ba70 2024-03-18T16:21:44.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.088+0000 7f9df5400640 1 -- 172.21.15.102:0/2486430499 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de8030610 con 0x7f9df806ba70 2024-03-18T16:21:44.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.088+0000 7f9df5400640 1 -- 172.21.15.102:0/2486430499 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de80309e0 con 0x7f9df806ba70 2024-03-18T16:21:44.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.089+0000 7f9dfd400640 1 -- 172.21.15.102:0/2486430499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 msgr2=0x7f9df806be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.089+0000 7f9dfd400640 1 --2- 172.21.15.102:0/2486430499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 0x7f9df806be90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9df8072d90 tx=0x7f9de802b960 comp rx=0 tx=0).stop 2024-03-18T16:21:44.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.089+0000 7f9dfd400640 1 -- 172.21.15.102:0/2486430499 shutdown_connections 2024-03-18T16:21:44.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.089+0000 7f9dfd400640 1 --2- 172.21.15.102:0/2486430499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 0x7f9df806be90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.089+0000 7f9dfd400640 1 --2- 172.21.15.102:0/2486430499 >> v2:172.21.15.102:3300/0 conn(0x7f9df8141700 0x7f9df80723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.089+0000 7f9dfd400640 1 -- 172.21.15.102:0/2486430499 >> 172.21.15.102:0/2486430499 conn(0x7f9df80c83a0 msgr2=0x7f9df80c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:44.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.089+0000 7f9dfd400640 1 -- 172.21.15.102:0/2486430499 shutdown_connections 2024-03-18T16:21:44.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.089+0000 7f9dfd400640 1 -- 172.21.15.102:0/2486430499 wait complete. 2024-03-18T16:21:44.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.090+0000 7f9dfd400640 1 Processor -- start 2024-03-18T16:21:44.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.091+0000 7f9dfd400640 1 -- start start 2024-03-18T16:21:44.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.091+0000 7f9dfd400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 0x7f9df81e3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9dfd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df80c4fb0 0x7f9df81e1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9dfd400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9df8141700 0x7f9df806f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9dfd400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9df815c260 con 0x7f9df80c4fb0 2024-03-18T16:21:44.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9dfd400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9df8158f10 con 0x7f9df806ba70 2024-03-18T16:21:44.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9dfd400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9df815c3e0 con 0x7f9df8141700 2024-03-18T16:21:44.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9df5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df80c4fb0 0x7f9df81e1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9df6800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 0x7f9df81e3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9df7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9df8141700 0x7f9df806f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9df6800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 0x7f9df81e3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59716/0 (socket says 172.21.15.102:59716) 2024-03-18T16:21:44.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.092+0000 7f9df6800640 1 -- 172.21.15.102:0/2245486809 learned_addr learned my addr 172.21.15.102:0/2245486809 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:44.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.096+0000 7f9df6800640 1 -- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9df8141700 msgr2=0x7f9df806f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.096+0000 7f9df6800640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9df8141700 0x7f9df806f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.096+0000 7f9df6800640 1 -- 172.21.15.102:0/2245486809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df80c4fb0 msgr2=0x7f9df81e1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.096+0000 7f9df6800640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df80c4fb0 0x7f9df81e1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.096+0000 7f9df6800640 1 -- 172.21.15.102:0/2245486809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df81e4170 con 0x7f9df806ba70 2024-03-18T16:21:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.099+0000 7f9df6800640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 0x7f9df81e3250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9de000e8a0 tx=0x7f9de000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:44.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.103+0000 7f9deea00640 1 -- 172.21.15.102:0/2245486809 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0017070 con 0x7f9df806ba70 2024-03-18T16:21:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.103+0000 7f9deea00640 1 -- 172.21.15.102:0/2245486809 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de000c940 con 0x7f9df806ba70 2024-03-18T16:21:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.103+0000 7f9deea00640 1 -- 172.21.15.102:0/2245486809 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de000cd30 con 0x7f9df806ba70 2024-03-18T16:21:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.103+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df80c4810 con 0x7f9df806ba70 2024-03-18T16:21:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.103+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df80c3b00 con 0x7f9df806ba70 2024-03-18T16:21:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.104+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df8074fd0 con 0x7f9df806ba70 2024-03-18T16:21:44.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.126+0000 7f9deea00640 1 -- 172.21.15.102:0/2245486809 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9de001a070 con 0x7f9df806ba70 2024-03-18T16:21:44.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.126+0000 7f9deea00640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9dd406bc20 0x7f9dd406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.127+0000 7f9df5e00640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9dd406bc20 0x7f9dd406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.127+0000 7f9df5e00640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9dd406bc20 0x7f9dd406e0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9de8036000 tx=0x7f9de803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:44.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.370+0000 7f9deea00640 1 -- 172.21.15.102:0/2245486809 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f9de00575a0 con 0x7f9df806ba70 2024-03-18T16:21:44.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.381+0000 7f9deea00640 1 -- 172.21.15.102:0/2245486809 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9de0059500 con 0x7f9df806ba70 2024-03-18T16:21:44.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.589+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9df80c27d0 con 0x7f9df806ba70 2024-03-18T16:21:44.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.610+0000 7f9deea00640 1 -- 172.21.15.102:0/2245486809 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f9de005d5a0 con 0x7f9df806ba70 2024-03-18T16:21:44.612 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-18T16:21:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9dd406bc20 msgr2=0x7f9dd406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9dd406bc20 0x7f9dd406e0e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9de8036000 tx=0x7f9de803b040 comp rx=0 tx=0).stop 2024-03-18T16:21:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 msgr2=0x7f9df81e3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 0x7f9df81e3250 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9de000e8a0 tx=0x7f9de000ed60 comp rx=0 tx=0).stop 2024-03-18T16:21:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 shutdown_connections 2024-03-18T16:21:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9dd406bc20 0x7f9dd406e0e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9df8141700 0x7f9df806f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9df80c4fb0 0x7f9df81e1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 --2- 172.21.15.102:0/2245486809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9df806ba70 0x7f9df81e3250 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 >> 172.21.15.102:0/2245486809 conn(0x7f9df80c83a0 msgr2=0x7f9df813e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:44.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 shutdown_connections 2024-03-18T16:21:44.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.613+0000 7f9dfd400640 1 -- 172.21.15.102:0/2245486809 wait complete. 2024-03-18T16:21:44.635 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:21:44.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.841+0000 7fc0ec400640 1 Processor -- start 2024-03-18T16:21:44.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.842+0000 7fc0ec400640 1 -- start start 2024-03-18T16:21:44.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.843+0000 7fc0ec400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0e40ca6e0 0x7fc0e40caac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.843+0000 7fc0ec400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc0e40cf240 0x7fc0e406cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.843+0000 7fc0ec400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e406d650 con 0x7fc0e406d1c0 2024-03-18T16:21:44.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.843+0000 7fc0ec400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e415b5b0 con 0x7fc0e40ca6e0 2024-03-18T16:21:44.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.843+0000 7fc0ec400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e415a770 con 0x7fc0e40cf240 2024-03-18T16:21:44.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.844+0000 7fc0e9000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc0e40cf240 0x7fc0e406cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.844+0000 7fc0e9000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc0e40cf240 0x7fc0e406cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59726/0 (socket says 172.21.15.102:59726) 2024-03-18T16:21:44.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.844+0000 7fc0e9000640 1 -- 172.21.15.102:0/2702870393 learned_addr learned my addr 172.21.15.102:0/2702870393 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:44.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.846+0000 7fc0e9000640 1 -- 172.21.15.102:0/2702870393 >> v2:172.21.15.102:3300/0 conn(0x7fc0e40ca6e0 msgr2=0x7fc0e40caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:44.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.846+0000 7fc0e9000640 1 --2- 172.21.15.102:0/2702870393 >> v2:172.21.15.102:3300/0 conn(0x7fc0e40ca6e0 0x7fc0e40caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.846+0000 7fc0e9000640 1 -- 172.21.15.102:0/2702870393 >> v1:172.21.15.102:6789/0 conn(0x7fc0e406d1c0 legacy=0x7fc0e415dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:44.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.846+0000 7fc0e9000640 1 -- 172.21.15.102:0/2702870393 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e415aa30 con 0x7fc0e40cf240 2024-03-18T16:21:44.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.849+0000 7fc0e9000640 1 --2- 172.21.15.102:0/2702870393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 0x7fc0e406cc80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc0e408d190 tx=0x7fc0d402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1bd8fdee16e7d90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:44.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.853+0000 7fc0e3e00640 1 -- 172.21.15.102:0/2702870393 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d402e070 con 0x7fc0e40cf240 2024-03-18T16:21:44.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.853+0000 7fc0e3e00640 1 -- 172.21.15.102:0/2702870393 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d4037070 con 0x7fc0e40cf240 2024-03-18T16:21:44.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.853+0000 7fc0e3e00640 1 -- 172.21.15.102:0/2702870393 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4031a00 con 0x7fc0e40cf240 2024-03-18T16:21:44.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.854+0000 7fc0ec400640 1 -- 172.21.15.102:0/2702870393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 msgr2=0x7fc0e406cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.854+0000 7fc0ec400640 1 --2- 172.21.15.102:0/2702870393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 0x7fc0e406cc80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc0e408d190 tx=0x7fc0d402b880 comp rx=0 tx=0).stop 2024-03-18T16:21:44.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.854+0000 7fc0ec400640 1 -- 172.21.15.102:0/2702870393 shutdown_connections 2024-03-18T16:21:44.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.854+0000 7fc0ec400640 1 --2- 172.21.15.102:0/2702870393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 0x7fc0e406cc80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.854+0000 7fc0ec400640 1 --2- 172.21.15.102:0/2702870393 >> v2:172.21.15.102:3300/0 conn(0x7fc0e40ca6e0 0x7fc0e40caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.854+0000 7fc0ec400640 1 -- 172.21.15.102:0/2702870393 >> 172.21.15.102:0/2702870393 conn(0x7fc0e40b18b0 msgr2=0x7fc0e40b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:44.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.854+0000 7fc0ec400640 1 -- 172.21.15.102:0/2702870393 shutdown_connections 2024-03-18T16:21:44.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.854+0000 7fc0ec400640 1 -- 172.21.15.102:0/2702870393 wait complete. 2024-03-18T16:21:44.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.856+0000 7fc0ec400640 1 Processor -- start 2024-03-18T16:21:44.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.856+0000 7fc0ec400640 1 -- start start 2024-03-18T16:21:44.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.857+0000 7fc0ec400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0e406d1c0 0x7fc0e40b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.857+0000 7fc0ec400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0e40ca6e0 0x7fc0e40b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.857+0000 7fc0ec400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 0x7fc0e40b4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.857+0000 7fc0ec400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e415b2d0 con 0x7fc0e406d1c0 2024-03-18T16:21:44.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.857+0000 7fc0ec400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e415aff0 con 0x7fc0e40cf240 2024-03-18T16:21:44.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.857+0000 7fc0ec400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e40cfa30 con 0x7fc0e40ca6e0 2024-03-18T16:21:44.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.858+0000 7fc0ea400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 0x7fc0e40b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.858+0000 7fc0e9000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0e40ca6e0 0x7fc0e40b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.858+0000 7fc0e9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0e406d1c0 0x7fc0e40b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.858+0000 7fc0ea400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 0x7fc0e40b4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59740/0 (socket says 172.21.15.102:59740) 2024-03-18T16:21:44.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.858+0000 7fc0ea400640 1 -- 172.21.15.102:0/3956787020 learned_addr learned my addr 172.21.15.102:0/3956787020 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:44.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.861+0000 7fc0ea400640 1 -- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0e40ca6e0 msgr2=0x7fc0e40b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.861+0000 7fc0ea400640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0e40ca6e0 0x7fc0e40b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.861+0000 7fc0ea400640 1 -- 172.21.15.102:0/3956787020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0e406d1c0 msgr2=0x7fc0e40b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.861+0000 7fc0ea400640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0e406d1c0 0x7fc0e40b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.861+0000 7fc0ea400640 1 -- 172.21.15.102:0/3956787020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e40b2ef0 con 0x7fc0e40cf240 2024-03-18T16:21:44.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.864+0000 7fc0ea400640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 0x7fc0e40b4890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00b9e0 tx=0x7fc0dc00bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:44.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.867+0000 7fc0e1600640 1 -- 172.21.15.102:0/3956787020 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc019070 con 0x7fc0e40cf240 2024-03-18T16:21:44.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.868+0000 7fc0e1600640 1 -- 172.21.15.102:0/3956787020 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0dc00caa0 con 0x7fc0e40cf240 2024-03-18T16:21:44.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.868+0000 7fc0e1600640 1 -- 172.21.15.102:0/3956787020 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0dc014440 con 0x7fc0e40cf240 2024-03-18T16:21:44.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.868+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e4156a30 con 0x7fc0e40cf240 2024-03-18T16:21:44.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.868+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e4156ff0 con 0x7fc0e40cf240 2024-03-18T16:21:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:44.869+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e4074fd0 con 0x7fc0e40cf240 2024-03-18T16:21:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.126+0000 7fc0e1600640 1 -- 172.21.15.102:0/3956787020 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc0dc003970 con 0x7fc0e40cf240 2024-03-18T16:21:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.129+0000 7fc0e1600640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc0c806bc50 0x7fc0c806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.129+0000 7fc0e1600640 1 -- 172.21.15.102:0/3956787020 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7fc0dc08a940 con 0x7fc0e40cf240 2024-03-18T16:21:45.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.129+0000 7fc0e9a00640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc0c806bc50 0x7fc0c806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.130+0000 7fc0e9a00640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc0c806bc50 0x7fc0c806e110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80098e0 tx=0x7fc0d8009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:45.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.138+0000 7fc0e1600640 1 -- 172.21.15.102:0/3956787020 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc0dc058b80 con 0x7fc0e40cf240 2024-03-18T16:21:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.346+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc0e406d940 con 0x7fc0e40cf240 2024-03-18T16:21:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.354+0000 7fc0e1600640 1 -- 172.21.15.102:0/3956787020 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fc0dc05cc20 con 0x7fc0e40cf240 2024-03-18T16:21:45.356 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-18T16:21:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc0c806bc50 msgr2=0x7fc0c806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc0c806bc50 0x7fc0c806e110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0d80098e0 tx=0x7fc0d8009060 comp rx=0 tx=0).stop 2024-03-18T16:21:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 msgr2=0x7fc0e40b4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 0x7fc0e40b4890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc00b9e0 tx=0x7fc0dc00bea0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 shutdown_connections 2024-03-18T16:21:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc0c806bc50 0x7fc0c806e110 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0e40cf240 0x7fc0e40b4890 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0e40ca6e0 0x7fc0e40b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 --2- 172.21.15.102:0/3956787020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0e406d1c0 0x7fc0e40b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 >> 172.21.15.102:0/3956787020 conn(0x7fc0e40b18b0 msgr2=0x7fc0e40b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 shutdown_connections 2024-03-18T16:21:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.357+0000 7fc0ec400640 1 -- 172.21.15.102:0/3956787020 wait complete. 2024-03-18T16:21:45.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.582+0000 7fc0b5a00640 1 Processor -- start 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.583+0000 7fc0b5a00640 1 -- start start 2024-03-18T16:21:45.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.585+0000 7fc0b5a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0b006d390 0x7fc0b0165f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.585+0000 7fc0b5a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0b015b5b0 con 0x7fc0b00ca6e0 2024-03-18T16:21:45.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.585+0000 7fc0b5a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0b015a770 con 0x7fc0b00caac0 2024-03-18T16:21:45.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.585+0000 7fc0b5a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc0b015aa30 con 0x7fc0b006d390 2024-03-18T16:21:45.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.598+0000 7fc0ae000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc0b00caac0 0x7fc0b006cc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33234/0 (socket says 172.21.15.102:33234) 2024-03-18T16:21:45.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.598+0000 7fc0ae000640 1 -- 172.21.15.102:0/2726683030 learned_addr learned my addr 172.21.15.102:0/2726683030 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:45.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.632+0000 7fc0af400640 1 --2- 172.21.15.102:0/2726683030 >> v2:172.21.15.102:3300/0 conn(0x7fc0b006d390 0x7fc0b0165f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.635+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 384557897 0 0) 0x7fc0b015a770 con 0x7fc0b00caac0 2024-03-18T16:21:45.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.635+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc094003520 con 0x7fc0b00caac0 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.639+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3715313583 0 0) 0x7fc094003520 con 0x7fc0b00caac0 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.639+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 >> v2:172.21.15.102:3300/0 conn(0x7fc0b006d390 msgr2=0x7fc0b0165f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.639+0000 7fc0ad600640 1 --2- 172.21.15.102:0/2726683030 >> v2:172.21.15.102:3300/0 conn(0x7fc0b006d390 0x7fc0b0165f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.639+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 >> v1:172.21.15.102:6789/0 conn(0x7fc0b00ca6e0 legacy=0x7fc0b00cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.639+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b015ad10 con 0x7fc0b00caac0 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.639+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7fc09c001670 con 0x7fc0b00caac0 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.639+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 >> v1:172.21.15.154:6789/0 conn(0x7fc0b00caac0 legacy=0x7fc0b006cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.639+0000 7fc0ad600640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc094003a00 0x7fc094003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.640+0000 7fc0ad600640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc094004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.640+0000 7fc0ad600640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0940052e0 0x7fc09400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.640+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b015aa30 con 0x7fc0940052e0 2024-03-18T16:21:45.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.640+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc09400c200 con 0x7fc094003a00 2024-03-18T16:21:45.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.640+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc09400c400 con 0x7fc094004800 2024-03-18T16:21:45.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.640+0000 7fc0ae000640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc094004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.643+0000 7fc0aea00640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc094003a00 0x7fc094003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.643+0000 7fc0ae000640 1 -- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc094003a00 msgr2=0x7fc094003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:45.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.643+0000 7fc0ae000640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc094003a00 0x7fc094003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.644+0000 7fc0ae000640 1 -- 172.21.15.102:0/2726683030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0940052e0 msgr2=0x7fc09400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:45.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.644+0000 7fc0ae000640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0940052e0 0x7fc09400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.644+0000 7fc0ae000640 1 -- 172.21.15.102:0/2726683030 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc094003520 con 0x7fc094004800 2024-03-18T16:21:45.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.644+0000 7fc0af400640 1 -- 172.21.15.102:0/2726683030 reap_dead start 2024-03-18T16:21:45.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.647+0000 7fc0ae000640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc094004c80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00c190 tx=0x7fc094005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37cfb1e16a108c3b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:45.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.651+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c007c60 con 0x7fc094004800 2024-03-18T16:21:45.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.651+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc09c030070 con 0x7fc094004800 2024-03-18T16:21:45.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.651+0000 7fc0ad600640 1 -- 172.21.15.102:0/2726683030 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c005d90 con 0x7fc094004800 2024-03-18T16:21:45.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.652+0000 7fc0b5a00640 1 -- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 msgr2=0x7fc094004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.652+0000 7fc0b5a00640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc094004c80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00c190 tx=0x7fc094005830 comp rx=0 tx=0).stop 2024-03-18T16:21:45.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.653+0000 7fc0b5a00640 1 -- 172.21.15.102:0/2726683030 shutdown_connections 2024-03-18T16:21:45.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.653+0000 7fc0b5a00640 1 --2- 172.21.15.102:0/2726683030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc094004c80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.653+0000 7fc0b5a00640 1 -- 172.21.15.102:0/2726683030 >> 172.21.15.102:0/2726683030 conn(0x7fc0b00b18b0 msgr2=0x7fc0b00b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:45.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.653+0000 7fc0b5a00640 1 -- 172.21.15.102:0/2726683030 shutdown_connections 2024-03-18T16:21:45.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.653+0000 7fc0b5a00640 1 -- 172.21.15.102:0/2726683030 wait complete. 2024-03-18T16:21:45.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.656+0000 7fc0b5a00640 1 Processor -- start 2024-03-18T16:21:45.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.656+0000 7fc0b5a00640 1 -- start start 2024-03-18T16:21:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.657+0000 7fc0b5a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc0b01f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.658+0000 7fc0b5a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01574f0 0x7fc0b01f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.658+0000 7fc0b5a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0b01f12f0 0x7fc0b0158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.658+0000 7fc0b5a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b00caf20 con 0x7fc0b01f12f0 2024-03-18T16:21:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.658+0000 7fc0b5a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b015b2d0 con 0x7fc0b01574f0 2024-03-18T16:21:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.658+0000 7fc0b5a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc0b00cfa30 con 0x7fc094004800 2024-03-18T16:21:45.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.658+0000 7fc0aea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc0b01f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.658+0000 7fc0ae000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01574f0 0x7fc0b01f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.658+0000 7fc0af400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0b01f12f0 0x7fc0b0158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.659+0000 7fc0aea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc0b01f1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52850/0 (socket says 172.21.15.102:52850) 2024-03-18T16:21:45.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.659+0000 7fc0aea00640 1 -- 172.21.15.102:0/711468459 learned_addr learned my addr 172.21.15.102:0/711468459 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:45.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.662+0000 7fc0aea00640 1 -- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01574f0 msgr2=0x7fc0b01f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.662+0000 7fc0aea00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01574f0 0x7fc0b01f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.662+0000 7fc0aea00640 1 -- 172.21.15.102:0/711468459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0b01f12f0 msgr2=0x7fc0b0158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.662+0000 7fc0aea00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0b01f12f0 0x7fc0b0158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.662+0000 7fc0aea00640 1 -- 172.21.15.102:0/711468459 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b00c87c0 con 0x7fc094004800 2024-03-18T16:21:45.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.664+0000 7fc0aea00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc0b01f1b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0007b90 tx=0x7fc0a000eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:45.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.668+0000 7fc0a6a00640 1 -- 172.21.15.102:0/711468459 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0003b80 con 0x7fc094004800 2024-03-18T16:21:45.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.668+0000 7fc0a6a00640 1 -- 172.21.15.102:0/711468459 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0a0018070 con 0x7fc094004800 2024-03-18T16:21:45.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.668+0000 7fc0a6a00640 1 -- 172.21.15.102:0/711468459 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a00050d0 con 0x7fc094004800 2024-03-18T16:21:45.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.668+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0b0158010 con 0x7fc094004800 2024-03-18T16:21:45.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.668+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0b0157a50 con 0x7fc094004800 2024-03-18T16:21:45.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.669+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0b0074fd0 con 0x7fc094004800 2024-03-18T16:21:45.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.920+0000 7fc0a6a00640 1 -- 172.21.15.102:0/711468459 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc0a0005270 con 0x7fc094004800 2024-03-18T16:21:45.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.922+0000 7fc0a6a00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09006bc20 0x7fc09006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.922+0000 7fc0a6a00640 1 -- 172.21.15.102:0/711468459 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7fc0a008b650 con 0x7fc094004800 2024-03-18T16:21:45.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.923+0000 7fc0ae000640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09006bc20 0x7fc09006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.924+0000 7fc0ae000640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09006bc20 0x7fc09006e0e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00c160 tx=0x7fc09c009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:45.932+0000 7fc0a6a00640 1 -- 172.21.15.102:0/711468459 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc0a0059800 con 0x7fc094004800 2024-03-18T16:21:46.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.157+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc0b015a770 con 0x7fc094004800 2024-03-18T16:21:46.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.164+0000 7fc0a6a00640 1 -- 172.21.15.102:0/711468459 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fc0a005d8a0 con 0x7fc094004800 2024-03-18T16:21:46.166 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09006bc20 msgr2=0x7fc09006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09006bc20 0x7fc09006e0e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00c160 tx=0x7fc09c009030 comp rx=0 tx=0).stop 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 msgr2=0x7fc0b01f1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc0b01f1b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0007b90 tx=0x7fc0a000eca0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 shutdown_connections 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09006bc20 0x7fc09006e0e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0b01f12f0 0x7fc0b0158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01574f0 0x7fc0b01f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 --2- 172.21.15.102:0/711468459 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc094004800 0x7fc0b01f1b20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 >> 172.21.15.102:0/711468459 conn(0x7fc0b00b18b0 msgr2=0x7fc0b0159f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:46.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 shutdown_connections 2024-03-18T16:21:46.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.167+0000 7fc0b5a00640 1 -- 172.21.15.102:0/711468459 wait complete. 2024-03-18T16:21:46.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:21:46.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.397+0000 7faa69200640 1 Processor -- start 2024-03-18T16:21:46.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.398+0000 7faa69200640 1 -- start start 2024-03-18T16:21:46.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.399+0000 7faa69200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faa640c3280 0x7faa640c3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:46.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.399+0000 7faa69200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa640c5160 con 0x7faa64140010 2024-03-18T16:21:46.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.399+0000 7faa69200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa641409b0 con 0x7faa6413fc30 2024-03-18T16:21:46.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.399+0000 7faa69200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7faa64140470 con 0x7faa640c3280 2024-03-18T16:21:46.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.400+0000 7faa61e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7faa6413fc30 0x7faa64071cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33238/0 (socket says 172.21.15.102:33238) 2024-03-18T16:21:46.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.400+0000 7faa61e00640 1 -- 172.21.15.102:0/3443801976 learned_addr learned my addr 172.21.15.102:0/3443801976 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:46.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.401+0000 7faa62800640 1 --2- 172.21.15.102:0/3443801976 >> v2:172.21.15.102:3300/0 conn(0x7faa640c3280 0x7faa640c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:46.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.402+0000 7faa61400640 1 -- 172.21.15.102:0/3443801976 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3526414684 0 0) 0x7faa641409b0 con 0x7faa6413fc30 2024-03-18T16:21:46.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.402+0000 7faa61400640 1 -- 172.21.15.102:0/3443801976 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa48003520 con 0x7faa6413fc30 2024-03-18T16:21:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.405+0000 7faa62800640 1 -- 172.21.15.102:0/3443801976 >> v1:172.21.15.154:6789/0 conn(0x7faa6413fc30 legacy=0x7faa64071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.405+0000 7faa62800640 1 -- 172.21.15.102:0/3443801976 >> v1:172.21.15.102:6789/0 conn(0x7faa64140010 legacy=0x7faa6406f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.405+0000 7faa62800640 1 -- 172.21.15.102:0/3443801976 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa640c2da0 con 0x7faa640c3280 2024-03-18T16:21:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.410+0000 7faa62800640 1 --2- 172.21.15.102:0/3443801976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa640c3280 0x7faa640c3710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009d90 tx=0x7faa5c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3578e5c852b30fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:46.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.416+0000 7faa61400640 1 -- 172.21.15.102:0/3443801976 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c039070 con 0x7faa640c3280 2024-03-18T16:21:46.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.416+0000 7faa61400640 1 -- 172.21.15.102:0/3443801976 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa5c02cc60 con 0x7faa640c3280 2024-03-18T16:21:46.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.416+0000 7faa61400640 1 -- 172.21.15.102:0/3443801976 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c034660 con 0x7faa640c3280 2024-03-18T16:21:46.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.417+0000 7faa69200640 1 -- 172.21.15.102:0/3443801976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa640c3280 msgr2=0x7faa640c3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.417+0000 7faa69200640 1 --2- 172.21.15.102:0/3443801976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa640c3280 0x7faa640c3710 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009d90 tx=0x7faa5c02beb0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.417+0000 7faa69200640 1 -- 172.21.15.102:0/3443801976 shutdown_connections 2024-03-18T16:21:46.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.417+0000 7faa69200640 1 --2- 172.21.15.102:0/3443801976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa640c3280 0x7faa640c3710 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.417+0000 7faa69200640 1 -- 172.21.15.102:0/3443801976 >> 172.21.15.102:0/3443801976 conn(0x7faa640c83a0 msgr2=0x7faa640c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:46.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.417+0000 7faa69200640 1 -- 172.21.15.102:0/3443801976 shutdown_connections 2024-03-18T16:21:46.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.417+0000 7faa69200640 1 -- 172.21.15.102:0/3443801976 wait complete. 2024-03-18T16:21:46.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.419+0000 7faa69200640 1 Processor -- start 2024-03-18T16:21:46.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.419+0000 7faa69200640 1 -- start start 2024-03-18T16:21:46.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.420+0000 7faa69200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa640c3280 0x7faa640caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:46.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.420+0000 7faa69200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa6413fc30 0x7faa641f1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:46.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.420+0000 7faa69200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa64140010 0x7faa641f1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:46.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.420+0000 7faa69200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faa640c56b0 con 0x7faa64140010 2024-03-18T16:21:46.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.420+0000 7faa69200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa640c3c50 con 0x7faa640c3280 2024-03-18T16:21:46.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.420+0000 7faa69200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faa640c5990 con 0x7faa6413fc30 2024-03-18T16:21:46.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.421+0000 7faa61e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa640c3280 0x7faa640caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:46.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.421+0000 7faa59400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa6413fc30 0x7faa641f1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:46.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.421+0000 7faa62800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa64140010 0x7faa641f1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:46.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.421+0000 7faa61e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa640c3280 0x7faa640caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59764/0 (socket says 172.21.15.102:59764) 2024-03-18T16:21:46.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.421+0000 7faa61e00640 1 -- 172.21.15.102:0/729529280 learned_addr learned my addr 172.21.15.102:0/729529280 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:46.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.424+0000 7faa61e00640 1 -- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa6413fc30 msgr2=0x7faa641f1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.424+0000 7faa61e00640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa6413fc30 0x7faa641f1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.424+0000 7faa61e00640 1 -- 172.21.15.102:0/729529280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa64140010 msgr2=0x7faa641f1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.424+0000 7faa61e00640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa64140010 0x7faa641f1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.424+0000 7faa61e00640 1 -- 172.21.15.102:0/729529280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa64150db0 con 0x7faa640c3280 2024-03-18T16:21:46.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.427+0000 7faa61e00640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa640c3280 0x7faa640caab0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faa5400c500 tx=0x7faa5400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:46.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.431+0000 7faa5aa00640 1 -- 172.21.15.102:0/729529280 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54017070 con 0x7faa640c3280 2024-03-18T16:21:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.431+0000 7faa5aa00640 1 -- 172.21.15.102:0/729529280 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa54005ce0 con 0x7faa640c3280 2024-03-18T16:21:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.431+0000 7faa5aa00640 1 -- 172.21.15.102:0/729529280 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54007330 con 0x7faa640c3280 2024-03-18T16:21:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.431+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa6413de60 con 0x7faa640c3280 2024-03-18T16:21:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.431+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6413e420 con 0x7faa640c3280 2024-03-18T16:21:46.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.432+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa64074fd0 con 0x7faa640c3280 2024-03-18T16:21:46.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.452+0000 7faa5aa00640 1 -- 172.21.15.102:0/729529280 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faa5401a070 con 0x7faa640c3280 2024-03-18T16:21:46.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.453+0000 7faa5aa00640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faa4006bc50 0x7faa4006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.454+0000 7faa5aa00640 1 -- 172.21.15.102:0/729529280 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7faa540568e0 con 0x7faa640c3280 2024-03-18T16:21:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.454+0000 7faa59400640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faa4006bc50 0x7faa4006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.454+0000 7faa59400640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faa4006bc50 0x7faa4006e110 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faa640c1740 tx=0x7faa4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:46.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.709+0000 7faa5aa00640 1 -- 172.21.15.102:0/729529280 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faa54021020 con 0x7faa640c3280 2024-03-18T16:21:46.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.920+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7faa64071aa0 con 0x7faa640c3280 2024-03-18T16:21:46.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.925+0000 7faa5aa00640 1 -- 172.21.15.102:0/729529280 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7faa54058840 con 0x7faa640c3280 2024-03-18T16:21:46.927 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.927+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faa4006bc50 msgr2=0x7faa4006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.927+0000 7faa69200640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faa4006bc50 0x7faa4006e110 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faa640c1740 tx=0x7faa4c009040 comp rx=0 tx=0).stop 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa640c3280 msgr2=0x7faa640caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa640c3280 0x7faa640caab0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faa5400c500 tx=0x7faa5400c9c0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 shutdown_connections 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faa4006bc50 0x7faa4006e110 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa64140010 0x7faa641f1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faa6413fc30 0x7faa641f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 --2- 172.21.15.102:0/729529280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa640c3280 0x7faa640caab0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 >> 172.21.15.102:0/729529280 conn(0x7faa640c83a0 msgr2=0x7faa641584d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 shutdown_connections 2024-03-18T16:21:46.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:46.928+0000 7faa69200640 1 -- 172.21.15.102:0/729529280 wait complete. 2024-03-18T16:21:46.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T16:21:47.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.154+0000 7f5a6e000640 1 Processor -- start 2024-03-18T16:21:47.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.154+0000 7f5a6e000640 1 -- start start 2024-03-18T16:21:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.155+0000 7f5a6e000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a68067310 0x7f5a680c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.155+0000 7f5a6e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5a6815ad90 0x7f5a68164c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:47.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.155+0000 7f5a6e000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a6815b280 con 0x7f5a681542d0 2024-03-18T16:21:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.155+0000 7f5a6e000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5a6806beb0 con 0x7f5a6815ad90 2024-03-18T16:21:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.155+0000 7f5a6e000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5a6806b2a0 con 0x7f5a68067310 2024-03-18T16:21:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.156+0000 7f5a6cc00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5a681542d0 0x7f5a6806f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59394/0 (socket says 172.21.15.102:59394) 2024-03-18T16:21:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.156+0000 7f5a6cc00640 1 -- 172.21.15.102:0/2669310617 learned_addr learned my addr 172.21.15.102:0/2669310617 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.156+0000 7f5a67e00640 1 --2- 172.21.15.102:0/2669310617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a68067310 0x7f5a680c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.159+0000 7f5a67e00640 1 -- 172.21.15.102:0/2669310617 >> v2:172.21.15.102:3300/0 conn(0x7f5a6815ad90 msgr2=0x7f5a68164c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.159+0000 7f5a67e00640 1 --2- 172.21.15.102:0/2669310617 >> v2:172.21.15.102:3300/0 conn(0x7f5a6815ad90 0x7f5a68164c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.159+0000 7f5a67e00640 1 -- 172.21.15.102:0/2669310617 >> v1:172.21.15.102:6789/0 conn(0x7f5a681542d0 legacy=0x7f5a6806f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:47.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.159+0000 7f5a67e00640 1 -- 172.21.15.102:0/2669310617 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6806b580 con 0x7f5a68067310 2024-03-18T16:21:47.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.161+0000 7f5a67e00640 1 --2- 172.21.15.102:0/2669310617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a68067310 0x7f5a680c6f20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5a540097a0 tx=0x7f5a5402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=745caceb0f47fc88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.165+0000 7f5a67400640 1 -- 172.21.15.102:0/2669310617 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54034070 con 0x7f5a68067310 2024-03-18T16:21:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.165+0000 7f5a67400640 1 -- 172.21.15.102:0/2669310617 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a54030610 con 0x7f5a68067310 2024-03-18T16:21:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.165+0000 7f5a67400640 1 -- 172.21.15.102:0/2669310617 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a540309e0 con 0x7f5a68067310 2024-03-18T16:21:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.166+0000 7f5a6e000640 1 -- 172.21.15.102:0/2669310617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a68067310 msgr2=0x7f5a680c6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.166+0000 7f5a6e000640 1 --2- 172.21.15.102:0/2669310617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a68067310 0x7f5a680c6f20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5a540097a0 tx=0x7f5a5402b960 comp rx=0 tx=0).stop 2024-03-18T16:21:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.166+0000 7f5a6e000640 1 -- 172.21.15.102:0/2669310617 shutdown_connections 2024-03-18T16:21:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.166+0000 7f5a6e000640 1 --2- 172.21.15.102:0/2669310617 >> v2:172.21.15.102:3300/0 conn(0x7f5a6815ad90 0x7f5a68164c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.166+0000 7f5a6e000640 1 --2- 172.21.15.102:0/2669310617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a68067310 0x7f5a680c6f20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.166+0000 7f5a6e000640 1 -- 172.21.15.102:0/2669310617 >> 172.21.15.102:0/2669310617 conn(0x7f5a680c8cf0 msgr2=0x7f5a680c9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.167+0000 7f5a6e000640 1 -- 172.21.15.102:0/2669310617 shutdown_connections 2024-03-18T16:21:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.167+0000 7f5a6e000640 1 -- 172.21.15.102:0/2669310617 wait complete. 2024-03-18T16:21:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.169+0000 7f5a6e000640 1 Processor -- start 2024-03-18T16:21:47.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.169+0000 7f5a6e000640 1 -- start start 2024-03-18T16:21:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.170+0000 7f5a6e000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a68067310 0x7f5a680cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:47.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.170+0000 7f5a6e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a681542d0 0x7f5a68158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.171+0000 7f5a6e000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a6815ad90 0x7f5a681590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.171+0000 7f5a6e000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5a681546b0 con 0x7f5a681542d0 2024-03-18T16:21:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.171+0000 7f5a6e000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6806bb40 con 0x7f5a68067310 2024-03-18T16:21:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.171+0000 7f5a6e000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5a68154830 con 0x7f5a6815ad90 2024-03-18T16:21:47.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.171+0000 7f5a6cc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a68067310 0x7f5a680cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.171+0000 7f5a6d600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a6815ad90 0x7f5a681590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.171+0000 7f5a67e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a681542d0 0x7f5a68158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.171+0000 7f5a6cc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a68067310 0x7f5a680cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59778/0 (socket says 172.21.15.102:59778) 2024-03-18T16:21:47.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.171+0000 7f5a6cc00640 1 -- 172.21.15.102:0/759400244 learned_addr learned my addr 172.21.15.102:0/759400244 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.175+0000 7f5a6cc00640 1 -- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a6815ad90 msgr2=0x7f5a681590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.175+0000 7f5a6cc00640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a6815ad90 0x7f5a681590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.175+0000 7f5a6cc00640 1 -- 172.21.15.102:0/759400244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a681542d0 msgr2=0x7f5a68158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.175+0000 7f5a6cc00640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a681542d0 0x7f5a68158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.175+0000 7f5a6cc00640 1 -- 172.21.15.102:0/759400244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a681f3b90 con 0x7f5a68067310 2024-03-18T16:21:47.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.177+0000 7f5a6cc00640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a68067310 0x7f5a680cb3e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00c430 tx=0x7f5a5c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.181+0000 7f5a64c00640 1 -- 172.21.15.102:0/759400244 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c017070 con 0x7f5a68067310 2024-03-18T16:21:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.181+0000 7f5a64c00640 1 -- 172.21.15.102:0/759400244 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a5c005ce0 con 0x7f5a68067310 2024-03-18T16:21:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.181+0000 7f5a64c00640 1 -- 172.21.15.102:0/759400244 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c0072f0 con 0x7f5a68067310 2024-03-18T16:21:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.181+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a681f3e50 con 0x7f5a68067310 2024-03-18T16:21:47.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.181+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a681f4410 con 0x7f5a68067310 2024-03-18T16:21:47.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.182+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a6804f5b0 con 0x7f5a68067310 2024-03-18T16:21:47.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.435+0000 7f5a64c00640 1 -- 172.21.15.102:0/759400244 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5a5c01a070 con 0x7f5a68067310 2024-03-18T16:21:47.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.437+0000 7f5a64c00640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a4806bc20 0x7f5a4806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:47.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.437+0000 7f5a64c00640 1 -- 172.21.15.102:0/759400244 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f5a5c08a570 con 0x7f5a68067310 2024-03-18T16:21:47.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.438+0000 7f5a67e00640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a4806bc20 0x7f5a4806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:47.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.439+0000 7f5a67e00640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a4806bc20 0x7f5a4806e0e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a54036000 tx=0x7f5a5403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:47.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.448+0000 7f5a64c00640 1 -- 172.21.15.102:0/759400244 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5a5c021020 con 0x7f5a68067310 2024-03-18T16:21:47.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.668+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5a681509b0 con 0x7f5a68067310 2024-03-18T16:21:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.673+0000 7f5a64c00640 1 -- 172.21.15.102:0/759400244 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f5a5c058720 con 0x7f5a68067310 2024-03-18T16:21:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-18T16:21:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.675+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a4806bc20 msgr2=0x7f5a4806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.675+0000 7f5a6e000640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a4806bc20 0x7f5a4806e0e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a54036000 tx=0x7f5a5403b040 comp rx=0 tx=0).stop 2024-03-18T16:21:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.675+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a68067310 msgr2=0x7f5a680cb3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.675+0000 7f5a6e000640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a68067310 0x7f5a680cb3e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00c430 tx=0x7f5a5c00c8f0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.676+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 shutdown_connections 2024-03-18T16:21:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.676+0000 7f5a6e000640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a4806bc20 0x7f5a4806e0e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.676+0000 7f5a6e000640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a6815ad90 0x7f5a681590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.676+0000 7f5a6e000640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a681542d0 0x7f5a68158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.676+0000 7f5a6e000640 1 --2- 172.21.15.102:0/759400244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a68067310 0x7f5a680cb3e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.676+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 >> 172.21.15.102:0/759400244 conn(0x7f5a680c8cf0 msgr2=0x7f5a681530d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.676+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 shutdown_connections 2024-03-18T16:21:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.676+0000 7f5a6e000640 1 -- 172.21.15.102:0/759400244 wait complete. 2024-03-18T16:21:47.697 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-18T16:21:47.697 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T16:21:47.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:47.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.993+0000 7f8c3f800640 1 Processor -- start 2024-03-18T16:21:47.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.993+0000 7f8c3f800640 1 -- start start 2024-03-18T16:21:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.994+0000 7f8c3f800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c380c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.995+0000 7f8c3f800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c3815c200 con 0x7f8c38072700 2024-03-18T16:21:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.995+0000 7f8c3f800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c3815c4e0 con 0x7f8c381524c0 2024-03-18T16:21:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.995+0000 7f8c3f800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8c38072b20 con 0x7f8c38158ee0 2024-03-18T16:21:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.995+0000 7f8c3d800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c380c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.995+0000 7f8c37e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8c38072700 0x7f8c3815b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59406/0 (socket says 172.21.15.102:59406) 2024-03-18T16:21:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.996+0000 7f8c37e00640 1 -- 172.21.15.102:0/1098257542 learned_addr learned my addr 172.21.15.102:0/1098257542 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:48.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.998+0000 7f8c3d800640 1 -- 172.21.15.102:0/1098257542 >> v1:172.21.15.154:6789/0 conn(0x7f8c381524c0 legacy=0x7f8c3806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.998+0000 7f8c3d800640 1 -- 172.21.15.102:0/1098257542 >> v1:172.21.15.102:6789/0 conn(0x7f8c38072700 legacy=0x7f8c3815b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:47.998+0000 7f8c3d800640 1 -- 172.21.15.102:0/1098257542 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c3815c7c0 con 0x7f8c38158ee0 2024-03-18T16:21:48.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.000+0000 7f8c3d800640 1 --2- 172.21.15.102:0/1098257542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c380c51e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009dc0 tx=0x7f8c3002bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80dc2c4dea52436 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:48.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.004+0000 7f8c37400640 1 -- 172.21.15.102:0/1098257542 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30039070 con 0x7f8c38158ee0 2024-03-18T16:21:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.004+0000 7f8c37400640 1 -- 172.21.15.102:0/1098257542 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3002cd60 con 0x7f8c38158ee0 2024-03-18T16:21:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.004+0000 7f8c37400640 1 -- 172.21.15.102:0/1098257542 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30034660 con 0x7f8c38158ee0 2024-03-18T16:21:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.004+0000 7f8c3f800640 1 -- 172.21.15.102:0/1098257542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 msgr2=0x7f8c380c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.004+0000 7f8c3f800640 1 --2- 172.21.15.102:0/1098257542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c380c51e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009dc0 tx=0x7f8c3002bfb0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.005+0000 7f8c3f800640 1 -- 172.21.15.102:0/1098257542 shutdown_connections 2024-03-18T16:21:48.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.005+0000 7f8c3f800640 1 --2- 172.21.15.102:0/1098257542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c380c51e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.005+0000 7f8c3f800640 1 -- 172.21.15.102:0/1098257542 >> 172.21.15.102:0/1098257542 conn(0x7f8c380700d0 msgr2=0x7f8c3806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:48.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.005+0000 7f8c3f800640 1 -- 172.21.15.102:0/1098257542 shutdown_connections 2024-03-18T16:21:48.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.005+0000 7f8c3f800640 1 -- 172.21.15.102:0/1098257542 wait complete. 2024-03-18T16:21:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.006+0000 7f8c3f800640 1 Processor -- start 2024-03-18T16:21:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.006+0000 7f8c3f800640 1 -- start start 2024-03-18T16:21:48.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.007+0000 7f8c3f800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c38072700 0x7f8c380c0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:48.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.007+0000 7f8c3f800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c381524c0 0x7f8c381575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:48.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.007+0000 7f8c3f800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c381efec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:48.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.007+0000 7f8c3f800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8c381589c0 con 0x7f8c381524c0 2024-03-18T16:21:48.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.007+0000 7f8c3f800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c381586e0 con 0x7f8c38072700 2024-03-18T16:21:48.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.007+0000 7f8c3f800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8c381528a0 con 0x7f8c38158ee0 2024-03-18T16:21:48.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.008+0000 7f8c3d800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c381efec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:48.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.008+0000 7f8c37e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c381524c0 0x7f8c381575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:48.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.008+0000 7f8c3ce00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c38072700 0x7f8c380c0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:48.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.008+0000 7f8c3d800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c381efec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52896/0 (socket says 172.21.15.102:52896) 2024-03-18T16:21:48.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.008+0000 7f8c3d800640 1 -- 172.21.15.102:0/2490389726 learned_addr learned my addr 172.21.15.102:0/2490389726 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:48.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.012+0000 7f8c3d800640 1 -- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c38072700 msgr2=0x7f8c380c0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.012+0000 7f8c3d800640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c38072700 0x7f8c380c0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.012+0000 7f8c3d800640 1 -- 172.21.15.102:0/2490389726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c381524c0 msgr2=0x7f8c381575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.012+0000 7f8c3d800640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c381524c0 0x7f8c381575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.012+0000 7f8c3d800640 1 -- 172.21.15.102:0/2490389726 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c381f3420 con 0x7f8c38158ee0 2024-03-18T16:21:48.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.014+0000 7f8c3d800640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c381efec0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8c3002c5c0 tx=0x7f8c30004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:48.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.019+0000 7f8c34c00640 1 -- 172.21.15.102:0/2490389726 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c30039050 con 0x7f8c38158ee0 2024-03-18T16:21:48.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.020+0000 7f8c34c00640 1 -- 172.21.15.102:0/2490389726 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c30045070 con 0x7f8c38158ee0 2024-03-18T16:21:48.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.020+0000 7f8c34c00640 1 -- 172.21.15.102:0/2490389726 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3003e850 con 0x7f8c38158ee0 2024-03-18T16:21:48.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.020+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c381f3700 con 0x7f8c38158ee0 2024-03-18T16:21:48.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.020+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c381f3cc0 con 0x7f8c38158ee0 2024-03-18T16:21:48.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.021+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c381f40a0 con 0x7f8c38158ee0 2024-03-18T16:21:48.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.281+0000 7f8c34c00640 1 -- 172.21.15.102:0/2490389726 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8c3003e9f0 con 0x7f8c38158ee0 2024-03-18T16:21:48.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.283+0000 7f8c34c00640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8c1806bc50 0x7f8c1806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.284+0000 7f8c3ce00640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8c1806bc50 0x7f8c1806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:48.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.284+0000 7f8c34c00640 1 -- 172.21.15.102:0/2490389726 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f8c3007a6a0 con 0x7f8c38158ee0 2024-03-18T16:21:48.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.285+0000 7f8c3ce00640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8c1806bc50 0x7f8c1806e110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c000f60 tx=0x7f8c2c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:48.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.295+0000 7f8c34c00640 1 -- 172.21.15.102:0/2490389726 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8c381f40a0 con 0x7f8c38158ee0 2024-03-18T16:21:48.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.511+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c38074fd0 con 0x7f8c38158ee0 2024-03-18T16:21:48.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.562+0000 7f8c34c00640 1 -- 172.21.15.102:0/2490389726 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18574 (secure 0 0 0) 0x7f8c3007c600 con 0x7f8c38158ee0 2024-03-18T16:21:48.565 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:48.565 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":27,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:30.100713+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:48.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.567+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8c1806bc50 msgr2=0x7f8c1806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.567+0000 7f8c3f800640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8c1806bc50 0x7f8c1806e110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c000f60 tx=0x7f8c2c009040 comp rx=0 tx=0).stop 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.567+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 msgr2=0x7f8c381efec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.567+0000 7f8c3f800640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c381efec0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8c3002c5c0 tx=0x7f8c30004a40 comp rx=0 tx=0).stop 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.567+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 shutdown_connections 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.567+0000 7f8c3f800640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8c1806bc50 0x7f8c1806e110 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.567+0000 7f8c3f800640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8c38158ee0 0x7f8c381efec0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.568+0000 7f8c3f800640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c381524c0 0x7f8c381575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.568+0000 7f8c3f800640 1 --2- 172.21.15.102:0/2490389726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c38072700 0x7f8c380c0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.568+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 >> 172.21.15.102:0/2490389726 conn(0x7f8c380700d0 msgr2=0x7f8c380c6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.568+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 shutdown_connections 2024-03-18T16:21:48.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.568+0000 7f8c3f800640 1 -- 172.21.15.102:0/2490389726 wait complete. 2024-03-18T16:21:48.592 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:21:05.204661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:21:19.524753+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:21:21.031843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:21:26.818690+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:21:48.593 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:21:48.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.803+0000 7fa56a600640 1 Processor -- start 2024-03-18T16:21:48.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.803+0000 7fa56a600640 1 -- start start 2024-03-18T16:21:48.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.804+0000 7fa56a600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa5641629f0 0x7fa5640c7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:48.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.804+0000 7fa56a600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa564072070 0x7fa56406f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:48.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.804+0000 7fa56a600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa56415f250 con 0x7fa564158c40 2024-03-18T16:21:48.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.804+0000 7fa56a600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa56415f530 con 0x7fa5641629f0 2024-03-18T16:21:48.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.804+0000 7fa56a600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa564161f10 con 0x7fa564072070 2024-03-18T16:21:48.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.805+0000 7fa563e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa564072070 0x7fa56406f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:48.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.806+0000 7fa563e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa564072070 0x7fa56406f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59796/0 (socket says 172.21.15.102:59796) 2024-03-18T16:21:48.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.806+0000 7fa563e00640 1 -- 172.21.15.102:0/1454696846 learned_addr learned my addr 172.21.15.102:0/1454696846 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:48.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.806+0000 7fa55aa00640 1 --2- 172.21.15.102:0/1454696846 >> v2:172.21.15.102:3300/0 conn(0x7fa5641629f0 0x7fa5640c7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:48.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.809+0000 7fa563e00640 1 -- 172.21.15.102:0/1454696846 >> v2:172.21.15.102:3300/0 conn(0x7fa5641629f0 msgr2=0x7fa5640c7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:48.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.809+0000 7fa563e00640 1 --2- 172.21.15.102:0/1454696846 >> v2:172.21.15.102:3300/0 conn(0x7fa5641629f0 0x7fa5640c7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.809+0000 7fa563e00640 1 -- 172.21.15.102:0/1454696846 >> v1:172.21.15.102:6789/0 conn(0x7fa564158c40 legacy=0x7fa56415e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.809+0000 7fa563e00640 1 -- 172.21.15.102:0/1454696846 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5641621f0 con 0x7fa564072070 2024-03-18T16:21:48.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.812+0000 7fa563e00640 1 --2- 172.21.15.102:0/1454696846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa564072070 0x7fa56406f250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa56406d680 tx=0x7fa55c02c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0d92b19ffd82a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:48.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.816+0000 7fa562a00640 1 -- 172.21.15.102:0/1454696846 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c02cbb0 con 0x7fa564072070 2024-03-18T16:21:48.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.816+0000 7fa562a00640 1 -- 172.21.15.102:0/1454696846 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55c02cd50 con 0x7fa564072070 2024-03-18T16:21:48.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.816+0000 7fa562a00640 1 -- 172.21.15.102:0/1454696846 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c034610 con 0x7fa564072070 2024-03-18T16:21:48.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.816+0000 7fa56a600640 1 -- 172.21.15.102:0/1454696846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa564072070 msgr2=0x7fa56406f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.816+0000 7fa56a600640 1 --2- 172.21.15.102:0/1454696846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa564072070 0x7fa56406f250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa56406d680 tx=0x7fa55c02c010 comp rx=0 tx=0).stop 2024-03-18T16:21:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.817+0000 7fa56a600640 1 -- 172.21.15.102:0/1454696846 shutdown_connections 2024-03-18T16:21:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.817+0000 7fa56a600640 1 --2- 172.21.15.102:0/1454696846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa564072070 0x7fa56406f250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.817+0000 7fa56a600640 1 --2- 172.21.15.102:0/1454696846 >> v2:172.21.15.102:3300/0 conn(0x7fa5641629f0 0x7fa5640c7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.817+0000 7fa56a600640 1 -- 172.21.15.102:0/1454696846 >> 172.21.15.102:0/1454696846 conn(0x7fa5640c83a0 msgr2=0x7fa5640c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.817+0000 7fa56a600640 1 -- 172.21.15.102:0/1454696846 shutdown_connections 2024-03-18T16:21:48.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.817+0000 7fa56a600640 1 -- 172.21.15.102:0/1454696846 wait complete. 2024-03-18T16:21:48.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.818+0000 7fa56a600640 1 Processor -- start 2024-03-18T16:21:48.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.819+0000 7fa56a600640 1 -- start start 2024-03-18T16:21:48.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.819+0000 7fa56a600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa564072070 0x7fa56415c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:48.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.820+0000 7fa56a600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa564158c40 0x7fa564160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:48.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.820+0000 7fa56a600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5641629f0 0x7fa5641565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:48.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.820+0000 7fa56a600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa56415eed0 con 0x7fa5641629f0 2024-03-18T16:21:48.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.820+0000 7fa56a600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa564072480 con 0x7fa564158c40 2024-03-18T16:21:48.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.820+0000 7fa56a600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa56406f790 con 0x7fa564072070 2024-03-18T16:21:48.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.820+0000 7fa563e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5641629f0 0x7fa5641565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:48.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.821+0000 7fa563e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5641629f0 0x7fa5641565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34756/0 (socket says 172.21.15.102:34756) 2024-03-18T16:21:48.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.821+0000 7fa563e00640 1 -- 172.21.15.102:0/3187502893 learned_addr learned my addr 172.21.15.102:0/3187502893 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:48.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.822+0000 7fa563400640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa564072070 0x7fa56415c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:48.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.823+0000 7fa55aa00640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa564158c40 0x7fa564160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:48.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.825+0000 7fa563e00640 1 -- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa564072070 msgr2=0x7fa56415c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.825+0000 7fa563e00640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa564072070 0x7fa56415c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.825+0000 7fa563e00640 1 -- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa564158c40 msgr2=0x7fa564160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:48.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.825+0000 7fa563e00640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa564158c40 0x7fa564160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:48.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.825+0000 7fa563e00640 1 -- 172.21.15.102:0/3187502893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56415c110 con 0x7fa5641629f0 2024-03-18T16:21:48.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.829+0000 7fa563e00640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5641629f0 0x7fa5641565f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa55c02c620 tx=0x7fa55c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:48.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.835+0000 7fa55be00640 1 -- 172.21.15.102:0/3187502893 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c043020 con 0x7fa5641629f0 2024-03-18T16:21:48.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.835+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa56415ced0 con 0x7fa5641629f0 2024-03-18T16:21:48.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.835+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa56415d490 con 0x7fa5641629f0 2024-03-18T16:21:48.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.836+0000 7fa55be00640 1 -- 172.21.15.102:0/3187502893 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55c02cbc0 con 0x7fa5641629f0 2024-03-18T16:21:48.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.836+0000 7fa55be00640 1 -- 172.21.15.102:0/3187502893 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c009510 con 0x7fa5641629f0 2024-03-18T16:21:48.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:48.837+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa53c005290 con 0x7fa5641629f0 2024-03-18T16:21:49.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.251+0000 7fa55be00640 1 -- 172.21.15.102:0/3187502893 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa55c046020 con 0x7fa5641629f0 2024-03-18T16:21:49.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.254+0000 7fa55be00640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc00 0x7fa54006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:49.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.254+0000 7fa563400640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc00 0x7fa54006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:49.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.255+0000 7fa563400640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc00 0x7fa54006e0c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa554001780 tx=0x7fa554009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:49.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.290+0000 7fa55be00640 1 -- 172.21.15.102:0/3187502893 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7fa55c079a30 con 0x7fa5641629f0 2024-03-18T16:21:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.291+0000 7fa55be00640 1 -- 172.21.15.102:0/3187502893 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa55c04d030 con 0x7fa5641629f0 2024-03-18T16:21:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.508+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa53c005850 con 0x7fa5641629f0 2024-03-18T16:21:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.523+0000 7fa55be00640 1 -- 172.21.15.102:0/3187502893 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fa55c07b990 con 0x7fa5641629f0 2024-03-18T16:21:49.526 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-18T16:21:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc00 msgr2=0x7fa54006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc00 0x7fa54006e0c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa554001780 tx=0x7fa554009040 comp rx=0 tx=0).stop 2024-03-18T16:21:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5641629f0 msgr2=0x7fa5641565f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5641629f0 0x7fa5641565f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa55c02c620 tx=0x7fa55c004a40 comp rx=0 tx=0).stop 2024-03-18T16:21:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 shutdown_connections 2024-03-18T16:21:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc00 0x7fa54006e0c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5641629f0 0x7fa5641565f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa564158c40 0x7fa564160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 --2- 172.21.15.102:0/3187502893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa564072070 0x7fa56415c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 >> 172.21.15.102:0/3187502893 conn(0x7fa5640c83a0 msgr2=0x7fa564072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:49.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 shutdown_connections 2024-03-18T16:21:49.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.527+0000 7fa56a600640 1 -- 172.21.15.102:0/3187502893 wait complete. 2024-03-18T16:21:49.554 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:21:49.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.764+0000 7f39e8200640 1 Processor -- start 2024-03-18T16:21:49.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.764+0000 7f39e8200640 1 -- start start 2024-03-18T16:21:49.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.766+0000 7f39e8200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f39e00c97e0 0x7f39e00ca320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:49.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.766+0000 7f39e8200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 0x7f39e00c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:49.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.766+0000 7f39e8200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39e015e960 con 0x7f39e0154690 2024-03-18T16:21:49.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.766+0000 7f39e8200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f39e00ca080 con 0x7f39e00c97e0 2024-03-18T16:21:49.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.766+0000 7f39e8200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f39e00c4410 con 0x7f39e00ca860 2024-03-18T16:21:49.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.766+0000 7f39e4e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f39e00c97e0 0x7f39e00ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:49.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.766+0000 7f39e6200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 0x7f39e00c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:49.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.766+0000 7f39e5800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f39e0154690 0x7f39e00c3d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59426/0 (socket says 172.21.15.102:59426) 2024-03-18T16:21:49.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.766+0000 7f39e5800640 1 -- 172.21.15.102:0/1054562588 learned_addr learned my addr 172.21.15.102:0/1054562588 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.769+0000 7f39e4e00640 1 -- 172.21.15.102:0/1054562588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 msgr2=0x7f39e00c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.769+0000 7f39e4e00640 1 --2- 172.21.15.102:0/1054562588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 0x7f39e00c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.769+0000 7f39e4e00640 1 -- 172.21.15.102:0/1054562588 >> v1:172.21.15.102:6789/0 conn(0x7f39e0154690 legacy=0x7f39e00c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.769+0000 7f39e4e00640 1 -- 172.21.15.102:0/1054562588 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e00c46d0 con 0x7f39e00c97e0 2024-03-18T16:21:49.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.771+0000 7f39e4e00640 1 --2- 172.21.15.102:0/1054562588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 0x7f39e00ca320 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0097d0 tx=0x7f39cc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a556e5d242bb9b7d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:49.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39dfe00640 1 -- 172.21.15.102:0/1054562588 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39cc035020 con 0x7f39e00c97e0 2024-03-18T16:21:49.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39dfe00640 1 -- 172.21.15.102:0/1054562588 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39cc031810 con 0x7f39e00c97e0 2024-03-18T16:21:49.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39dfe00640 1 -- 172.21.15.102:0/1054562588 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39cc031be0 con 0x7f39e00c97e0 2024-03-18T16:21:49.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39e8200640 1 -- 172.21.15.102:0/1054562588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 msgr2=0x7f39e00ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:49.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39e8200640 1 --2- 172.21.15.102:0/1054562588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 0x7f39e00ca320 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0097d0 tx=0x7f39cc02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39e8200640 1 -- 172.21.15.102:0/1054562588 shutdown_connections 2024-03-18T16:21:49.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39e8200640 1 --2- 172.21.15.102:0/1054562588 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 0x7f39e00c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39e8200640 1 --2- 172.21.15.102:0/1054562588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 0x7f39e00ca320 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39e8200640 1 -- 172.21.15.102:0/1054562588 >> 172.21.15.102:0/1054562588 conn(0x7f39e0071680 msgr2=0x7f39e0071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:49.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.775+0000 7f39e8200640 1 -- 172.21.15.102:0/1054562588 shutdown_connections 2024-03-18T16:21:49.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.776+0000 7f39e8200640 1 -- 172.21.15.102:0/1054562588 wait complete. 2024-03-18T16:21:49.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.777+0000 7f39e8200640 1 Processor -- start 2024-03-18T16:21:49.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.777+0000 7f39e8200640 1 -- start start 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.778+0000 7f39e8200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 0x7f39e0153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.778+0000 7f39e8200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 0x7f39e0152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.778+0000 7f39e8200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39e0154690 0x7f39e00c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.778+0000 7f39e8200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f39e015e160 con 0x7f39e0154690 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.778+0000 7f39e8200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39e015de80 con 0x7f39e00c97e0 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.778+0000 7f39e8200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f39e015e440 con 0x7f39e00ca860 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.779+0000 7f39e5800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 0x7f39e0153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.779+0000 7f39e4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 0x7f39e0152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.779+0000 7f39e6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39e0154690 0x7f39e00c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:49.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.779+0000 7f39e5800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 0x7f39e0153590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59806/0 (socket says 172.21.15.102:59806) 2024-03-18T16:21:49.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.779+0000 7f39e5800640 1 -- 172.21.15.102:0/553404171 learned_addr learned my addr 172.21.15.102:0/553404171 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:49.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.782+0000 7f39e5800640 1 -- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 msgr2=0x7f39e0152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:49.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.782+0000 7f39e5800640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 0x7f39e0152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.782+0000 7f39e5800640 1 -- 172.21.15.102:0/553404171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39e0154690 msgr2=0x7f39e00c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:49.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.782+0000 7f39e5800640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39e0154690 0x7f39e00c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:49.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.782+0000 7f39e5800640 1 -- 172.21.15.102:0/553404171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e01f0ef0 con 0x7f39e00c97e0 2024-03-18T16:21:49.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.785+0000 7f39e5800640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 0x7f39e0153590 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f39d400c430 tx=0x7f39d400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:49.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.788+0000 7f39dd600640 1 -- 172.21.15.102:0/553404171 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d4017070 con 0x7f39e00c97e0 2024-03-18T16:21:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.788+0000 7f39dd600640 1 -- 172.21.15.102:0/553404171 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d4005ce0 con 0x7f39e00c97e0 2024-03-18T16:21:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.788+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e00b3f30 con 0x7f39e00c97e0 2024-03-18T16:21:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.789+0000 7f39dd600640 1 -- 172.21.15.102:0/553404171 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d40072f0 con 0x7f39e00c97e0 2024-03-18T16:21:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.789+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e01f0410 con 0x7f39e00c97e0 2024-03-18T16:21:49.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.790+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39e0074fd0 con 0x7f39e00c97e0 2024-03-18T16:21:49.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.809+0000 7f39dd600640 1 -- 172.21.15.102:0/553404171 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f39d401a070 con 0x7f39e00c97e0 2024-03-18T16:21:49.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.810+0000 7f39dd600640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39c006bc20 0x7f39c006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:49.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.811+0000 7f39dd600640 1 -- 172.21.15.102:0/553404171 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f39d40567c0 con 0x7f39e00c97e0 2024-03-18T16:21:49.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.811+0000 7f39e4e00640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39c006bc20 0x7f39c006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:49.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:49.812+0000 7f39e4e00640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39c006bc20 0x7f39c006e0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39cc036000 tx=0x7f39cc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:50.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.054+0000 7f39dd600640 1 -- 172.21.15.102:0/553404171 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f39d4021020 con 0x7f39e00c97e0 2024-03-18T16:21:50.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:21:50.264 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:21:50.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.260+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f39e00c95c0 con 0x7f39e00c97e0 2024-03-18T16:21:50.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.380+0000 7f39dd600640 1 -- 172.21.15.102:0/553404171 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f39d4058720 con 0x7f39e00c97e0 2024-03-18T16:21:50.383 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39c006bc20 msgr2=0x7f39c006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39c006bc20 0x7f39c006e0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39cc036000 tx=0x7f39cc03c040 comp rx=0 tx=0).stop 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 msgr2=0x7f39e0153590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 0x7f39e0153590 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f39d400c430 tx=0x7f39d400c8f0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 shutdown_connections 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39c006bc20 0x7f39c006e0e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39e0154690 0x7f39e00c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39e00ca860 0x7f39e0152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 --2- 172.21.15.102:0/553404171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00c97e0 0x7f39e0153590 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 >> 172.21.15.102:0/553404171 conn(0x7f39e0071680 msgr2=0x7f39e015d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:50.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 shutdown_connections 2024-03-18T16:21:50.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.383+0000 7f39e8200640 1 -- 172.21.15.102:0/553404171 wait complete. 2024-03-18T16:21:50.405 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:21:50.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.658+0000 7f58c6a00640 1 Processor -- start 2024-03-18T16:21:50.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.658+0000 7f58c6a00640 1 -- start start 2024-03-18T16:21:50.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.661+0000 7f58c6a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f58c0154690 0x7f58c00c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:50.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.661+0000 7f58c6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c00c97e0 0x7f58c00c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:50.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.661+0000 7f58c6a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c015e960 con 0x7f58c00c4240 2024-03-18T16:21:50.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.661+0000 7f58c6a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f58c00ca080 con 0x7f58c0154690 2024-03-18T16:21:50.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.661+0000 7f58c6a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f58c00c46d0 con 0x7f58c00c97e0 2024-03-18T16:21:50.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.662+0000 7f58bfe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f58c0154690 0x7f58c00c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:50.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.662+0000 7f58c4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c00c97e0 0x7f58c00c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:50.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.662+0000 7f58bf400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f58c00c4240 0x7f58c00ca320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33258/0 (socket says 172.21.15.102:33258) 2024-03-18T16:21:50.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.662+0000 7f58bf400640 1 -- 172.21.15.102:0/1636809111 learned_addr learned my addr 172.21.15.102:0/1636809111 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:50.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.665+0000 7f58c4a00640 1 -- 172.21.15.102:0/1636809111 >> v2:172.21.15.154:3300/0 conn(0x7f58c0154690 msgr2=0x7f58c00c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:50.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.665+0000 7f58c4a00640 1 --2- 172.21.15.102:0/1636809111 >> v2:172.21.15.154:3300/0 conn(0x7f58c0154690 0x7f58c00c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.665+0000 7f58c4a00640 1 -- 172.21.15.102:0/1636809111 >> v1:172.21.15.154:6789/0 conn(0x7f58c00c4240 legacy=0x7f58c00ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:50.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.665+0000 7f58c4a00640 1 -- 172.21.15.102:0/1636809111 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c015dba0 con 0x7f58c00c97e0 2024-03-18T16:21:50.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.667+0000 7f58c4a00640 1 --2- 172.21.15.102:0/1636809111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c00c97e0 0x7f58c00c8a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009d90 tx=0x7f58b802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4bf722876c523ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:50.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.671+0000 7f58bea00640 1 -- 172.21.15.102:0/1636809111 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8039070 con 0x7f58c00c97e0 2024-03-18T16:21:50.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.671+0000 7f58bea00640 1 -- 172.21.15.102:0/1636809111 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b802cc50 con 0x7f58c00c97e0 2024-03-18T16:21:50.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.671+0000 7f58bea00640 1 -- 172.21.15.102:0/1636809111 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8034440 con 0x7f58c00c97e0 2024-03-18T16:21:50.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.671+0000 7f58c6a00640 1 -- 172.21.15.102:0/1636809111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c00c97e0 msgr2=0x7f58c00c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:50.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.671+0000 7f58c6a00640 1 --2- 172.21.15.102:0/1636809111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c00c97e0 0x7f58c00c8a10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58b8009d90 tx=0x7f58b802bea0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.672+0000 7f58c6a00640 1 -- 172.21.15.102:0/1636809111 shutdown_connections 2024-03-18T16:21:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.672+0000 7f58c6a00640 1 --2- 172.21.15.102:0/1636809111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c00c97e0 0x7f58c00c8a10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.672+0000 7f58c6a00640 1 --2- 172.21.15.102:0/1636809111 >> v2:172.21.15.154:3300/0 conn(0x7f58c0154690 0x7f58c00c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.672+0000 7f58c6a00640 1 -- 172.21.15.102:0/1636809111 >> 172.21.15.102:0/1636809111 conn(0x7f58c0071680 msgr2=0x7f58c0071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.672+0000 7f58c6a00640 1 -- 172.21.15.102:0/1636809111 shutdown_connections 2024-03-18T16:21:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.672+0000 7f58c6a00640 1 -- 172.21.15.102:0/1636809111 wait complete. 2024-03-18T16:21:50.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.674+0000 7f58c6a00640 1 Processor -- start 2024-03-18T16:21:50.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.675+0000 7f58c6a00640 1 -- start start 2024-03-18T16:21:50.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.676+0000 7f58c6a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c00c97e0 0x7f58c00b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:50.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.676+0000 7f58c6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c0154690 0x7f58c006f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:50.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.676+0000 7f58c6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c0151db0 0x7f58c0152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:50.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.676+0000 7f58c6a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f58c015e440 con 0x7f58c0154690 2024-03-18T16:21:50.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.676+0000 7f58c6a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58c015e160 con 0x7f58c00c97e0 2024-03-18T16:21:50.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.676+0000 7f58c6a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f58c00caa30 con 0x7f58c0151db0 2024-03-18T16:21:50.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.677+0000 7f58c4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c0151db0 0x7f58c0152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:50.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.677+0000 7f58bfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c00c97e0 0x7f58c00b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:50.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.677+0000 7f58bfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c00c97e0 0x7f58c00b3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59830/0 (socket says 172.21.15.102:59830) 2024-03-18T16:21:50.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.677+0000 7f58bfe00640 1 -- 172.21.15.102:0/2432814826 learned_addr learned my addr 172.21.15.102:0/2432814826 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:50.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.677+0000 7f58c4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c0151db0 0x7f58c0152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52944/0 (socket says 172.21.15.102:52944) 2024-03-18T16:21:50.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.677+0000 7f58bf400640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c0154690 0x7f58c006f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:50.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.681+0000 7f58c4a00640 1 -- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c00c97e0 msgr2=0x7f58c00b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.681+0000 7f58c4a00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c00c97e0 0x7f58c00b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.681+0000 7f58c4a00640 1 -- 172.21.15.102:0/2432814826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c0154690 msgr2=0x7f58c006f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.681+0000 7f58c4a00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c0154690 0x7f58c006f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.681+0000 7f58c4a00640 1 -- 172.21.15.102:0/2432814826 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c014d5c0 con 0x7f58c0151db0 2024-03-18T16:21:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.683+0000 7f58c4a00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c0151db0 0x7f58c0152240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f58b8033000 tx=0x7f58b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:50.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.686+0000 7f58b7e00640 1 -- 172.21.15.102:0/2432814826 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8039050 con 0x7f58c0151db0 2024-03-18T16:21:50.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.687+0000 7f58b7e00640 1 -- 172.21.15.102:0/2432814826 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b8045070 con 0x7f58c0151db0 2024-03-18T16:21:50.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.687+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c0152860 con 0x7f58c0151db0 2024-03-18T16:21:50.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.687+0000 7f58b7e00640 1 -- 172.21.15.102:0/2432814826 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b8009450 con 0x7f58c0151db0 2024-03-18T16:21:50.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.687+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c0158670 con 0x7f58c0151db0 2024-03-18T16:21:50.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.688+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c0074fd0 con 0x7f58c0151db0 2024-03-18T16:21:50.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.719+0000 7f58b7e00640 1 -- 172.21.15.102:0/2432814826 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f58b803e610 con 0x7f58c0151db0 2024-03-18T16:21:50.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.720+0000 7f58b7e00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f58a006bc20 0x7f58a006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.720+0000 7f58bfe00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f58a006bc20 0x7f58a006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:50.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.721+0000 7f58bfe00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f58a006bc20 0x7f58a006e0e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009b60 tx=0x7f58b0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:50.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.722+0000 7f58b7e00640 1 -- 172.21.15.102:0/2432814826 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f58b807a330 con 0x7f58c0151db0 2024-03-18T16:21:50.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:50.977+0000 7f58b7e00640 1 -- 172.21.15.102:0/2432814826 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f58b807c290 con 0x7f58c0151db0 2024-03-18T16:21:51.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.184+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f58c00c95c0 con 0x7f58c0151db0 2024-03-18T16:21:51.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.190+0000 7f58b7e00640 1 -- 172.21.15.102:0/2432814826 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f58b8080330 con 0x7f58c0151db0 2024-03-18T16:21:51.193 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-18T16:21:51.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f58a006bc20 msgr2=0x7f58a006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:51.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f58a006bc20 0x7f58a006e0e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f58b0009b60 tx=0x7f58b0009190 comp rx=0 tx=0).stop 2024-03-18T16:21:51.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c0151db0 msgr2=0x7f58c0152240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c0151db0 0x7f58c0152240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f58b8033000 tx=0x7f58b8004a40 comp rx=0 tx=0).stop 2024-03-18T16:21:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 shutdown_connections 2024-03-18T16:21:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f58a006bc20 0x7f58a006e0e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58c0151db0 0x7f58c0152240 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58c0154690 0x7f58c006f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 --2- 172.21.15.102:0/2432814826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c00c97e0 0x7f58c00b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 >> 172.21.15.102:0/2432814826 conn(0x7f58c0071680 msgr2=0x7f58c006c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.192+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 shutdown_connections 2024-03-18T16:21:51.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.193+0000 7f58c6a00640 1 -- 172.21.15.102:0/2432814826 wait complete. 2024-03-18T16:21:51.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:21:51.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.389+0000 7fc760a00640 1 Processor -- start 2024-03-18T16:21:51.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.389+0000 7fc760a00640 1 -- start start 2024-03-18T16:21:51.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.390+0000 7fc760a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc75c13fc30 0x7fc75c140010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:51.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.390+0000 7fc760a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc75c071cc0 0x7fc75c072120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.390+0000 7fc760a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc75c0c5160 con 0x7fc75c06f250 2024-03-18T16:21:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.390+0000 7fc760a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc75c1409b0 con 0x7fc75c13fc30 2024-03-18T16:21:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.390+0000 7fc760a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc75c0c2da0 con 0x7fc75c071cc0 2024-03-18T16:21:51.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.391+0000 7fc759e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc75c13fc30 0x7fc75c140010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:51.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.394+0000 7fc759e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc75c13fc30 0x7fc75c140010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55116/0 (socket says 172.21.15.102:55116) 2024-03-18T16:21:51.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.394+0000 7fc759e00640 1 -- 172.21.15.102:0/3175334555 learned_addr learned my addr 172.21.15.102:0/3175334555 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:51.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.397+0000 7fc759e00640 1 -- 172.21.15.102:0/3175334555 >> v2:172.21.15.154:3300/0 conn(0x7fc75c071cc0 msgr2=0x7fc75c072120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:51.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.397+0000 7fc759e00640 1 --2- 172.21.15.102:0/3175334555 >> v2:172.21.15.154:3300/0 conn(0x7fc75c071cc0 0x7fc75c072120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.397+0000 7fc759e00640 1 -- 172.21.15.102:0/3175334555 >> v1:172.21.15.154:6789/0 conn(0x7fc75c06f250 legacy=0x7fc75c0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:51.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.397+0000 7fc759e00640 1 -- 172.21.15.102:0/3175334555 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75c0c53d0 con 0x7fc75c13fc30 2024-03-18T16:21:51.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.399+0000 7fc759e00640 1 --2- 172.21.15.102:0/3175334555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 0x7fc75c140010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0c92d0 tx=0x7fc74c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7760b178fcf9861 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.404+0000 7fc758a00640 1 -- 172.21.15.102:0/3175334555 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c02e070 con 0x7fc75c13fc30 2024-03-18T16:21:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.404+0000 7fc758a00640 1 -- 172.21.15.102:0/3175334555 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc74c037070 con 0x7fc75c13fc30 2024-03-18T16:21:51.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.404+0000 7fc758a00640 1 -- 172.21.15.102:0/3175334555 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c031a00 con 0x7fc75c13fc30 2024-03-18T16:21:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.405+0000 7fc760a00640 1 -- 172.21.15.102:0/3175334555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 msgr2=0x7fc75c140010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.405+0000 7fc760a00640 1 --2- 172.21.15.102:0/3175334555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 0x7fc75c140010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0c92d0 tx=0x7fc74c02b880 comp rx=0 tx=0).stop 2024-03-18T16:21:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.405+0000 7fc760a00640 1 -- 172.21.15.102:0/3175334555 shutdown_connections 2024-03-18T16:21:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.405+0000 7fc760a00640 1 --2- 172.21.15.102:0/3175334555 >> v2:172.21.15.154:3300/0 conn(0x7fc75c071cc0 0x7fc75c072120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.405+0000 7fc760a00640 1 --2- 172.21.15.102:0/3175334555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 0x7fc75c140010 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.405+0000 7fc760a00640 1 -- 172.21.15.102:0/3175334555 >> 172.21.15.102:0/3175334555 conn(0x7fc75c0c83a0 msgr2=0x7fc75c0c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.405+0000 7fc760a00640 1 -- 172.21.15.102:0/3175334555 shutdown_connections 2024-03-18T16:21:51.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.405+0000 7fc760a00640 1 -- 172.21.15.102:0/3175334555 wait complete. 2024-03-18T16:21:51.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.407+0000 7fc760a00640 1 Processor -- start 2024-03-18T16:21:51.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.407+0000 7fc760a00640 1 -- start start 2024-03-18T16:21:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.408+0000 7fc760a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc75c06f250 0x7fc75c157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.408+0000 7fc760a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc75c071cc0 0x7fc75c0ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.408+0000 7fc760a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 0x7fc75c1ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.408+0000 7fc760a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c0c5c70 con 0x7fc75c13fc30 2024-03-18T16:21:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.408+0000 7fc760a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c0c5990 con 0x7fc75c06f250 2024-03-18T16:21:51.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.408+0000 7fc760a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc75c0c3990 con 0x7fc75c071cc0 2024-03-18T16:21:51.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.409+0000 7fc75a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 0x7fc75c1ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:51.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.409+0000 7fc75a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 0x7fc75c1ebe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55128/0 (socket says 172.21.15.102:55128) 2024-03-18T16:21:51.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.409+0000 7fc75a800640 1 -- 172.21.15.102:0/2568688766 learned_addr learned my addr 172.21.15.102:0/2568688766 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:51.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.413+0000 7fc75a800640 1 -- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc75c071cc0 msgr2=0x7fc75c0ca980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:51.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.413+0000 7fc75a800640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc75c071cc0 0x7fc75c0ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.413+0000 7fc75a800640 1 -- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc75c06f250 msgr2=0x7fc75c157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:51.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.413+0000 7fc75a800640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc75c06f250 0x7fc75c157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.413+0000 7fc75a800640 1 -- 172.21.15.102:0/2568688766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75c0cb480 con 0x7fc75c13fc30 2024-03-18T16:21:51.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.415+0000 7fc75a800640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 0x7fc75c1ebe70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc75400c7b0 tx=0x7fc75400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:51.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.419+0000 7fc752000640 1 -- 172.21.15.102:0/2568688766 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc754017070 con 0x7fc75c13fc30 2024-03-18T16:21:51.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.420+0000 7fc752000640 1 -- 172.21.15.102:0/2568688766 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc754003da0 con 0x7fc75c13fc30 2024-03-18T16:21:51.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.420+0000 7fc752000640 1 -- 172.21.15.102:0/2568688766 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7540052b0 con 0x7fc75c13fc30 2024-03-18T16:21:51.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.420+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75c158000 con 0x7fc75c13fc30 2024-03-18T16:21:51.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.420+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75c13e760 con 0x7fc75c13fc30 2024-03-18T16:21:51.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.421+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc75c06fec0 con 0x7fc75c13fc30 2024-03-18T16:21:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.443+0000 7fc752000640 1 -- 172.21.15.102:0/2568688766 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc75401a070 con 0x7fc75c13fc30 2024-03-18T16:21:51.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.444+0000 7fc752000640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc73c06bc20 0x7fc73c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.444+0000 7fc759e00640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc73c06bc20 0x7fc73c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.445+0000 7fc759e00640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc73c06bc20 0x7fc73c06e0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc74c036000 tx=0x7fc74c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.448+0000 7fc752000640 1 -- 172.21.15.102:0/2568688766 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7fc754056a10 con 0x7fc75c13fc30 2024-03-18T16:21:51.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.734+0000 7fc752000640 1 -- 172.21.15.102:0/2568688766 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc75c06fec0 con 0x7fc75c13fc30 2024-03-18T16:21:51.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.929+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc75c074fd0 con 0x7fc75c13fc30 2024-03-18T16:21:51.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.936+0000 7fc752000640 1 -- 172.21.15.102:0/2568688766 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fc754021020 con 0x7fc75c13fc30 2024-03-18T16:21:51.938 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-18T16:21:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc73c06bc20 msgr2=0x7fc73c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc73c06bc20 0x7fc73c06e0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc74c036000 tx=0x7fc74c030000 comp rx=0 tx=0).stop 2024-03-18T16:21:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 msgr2=0x7fc75c1ebe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 0x7fc75c1ebe70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc75400c7b0 tx=0x7fc75400cc70 comp rx=0 tx=0).stop 2024-03-18T16:21:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 shutdown_connections 2024-03-18T16:21:51.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc73c06bc20 0x7fc73c06e0e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc75c13fc30 0x7fc75c1ebe70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc75c071cc0 0x7fc75c0ca980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 --2- 172.21.15.102:0/2568688766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc75c06f250 0x7fc75c157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.938+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 >> 172.21.15.102:0/2568688766 conn(0x7fc75c0c83a0 msgr2=0x7fc75c138c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.939+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 shutdown_connections 2024-03-18T16:21:51.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:51.939+0000 7fc760a00640 1 -- 172.21.15.102:0/2568688766 wait complete. 2024-03-18T16:21:51.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T16:21:52.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.143+0000 7f6d47800640 1 Processor -- start 2024-03-18T16:21:52.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.143+0000 7f6d47800640 1 -- start start 2024-03-18T16:21:52.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.144+0000 7f6d47800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6d40154990 0x7f6d4015af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:52.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.144+0000 7f6d47800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d400701f0 0x7f6d40070630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:52.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.144+0000 7f6d47800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6d400c97d0 0x7f6d400c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:52.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.144+0000 7f6d47800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6d4015e810 con 0x7f6d400c97d0 2024-03-18T16:21:52.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.144+0000 7f6d47800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6d4015b470 con 0x7f6d40154990 2024-03-18T16:21:52.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.144+0000 7f6d47800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d40070c10 con 0x7f6d400701f0 2024-03-18T16:21:52.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.145+0000 7f6d44e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6d40154990 0x7f6d4015af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:52.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.145+0000 7f6d3fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d400701f0 0x7f6d40070630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:52.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.145+0000 7f6d45800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6d400c97d0 0x7f6d400c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:52.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.145+0000 7f6d44e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6d40154990 0x7f6d4015af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42924/0 (socket says 172.21.15.102:42924) 2024-03-18T16:21:52.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.145+0000 7f6d44e00640 1 -- 172.21.15.102:0/1234659617 learned_addr learned my addr 172.21.15.102:0/1234659617 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:52.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.147+0000 7f6d3fe00640 1 -- 172.21.15.102:0/1234659617 >> v2:172.21.15.154:3300/0 conn(0x7f6d40154990 msgr2=0x7f6d4015af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:52.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.147+0000 7f6d3fe00640 1 --2- 172.21.15.102:0/1234659617 >> v2:172.21.15.154:3300/0 conn(0x7f6d40154990 0x7f6d4015af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.147+0000 7f6d3fe00640 1 -- 172.21.15.102:0/1234659617 >> v2:172.21.15.102:3300/0 conn(0x7f6d400c97d0 msgr2=0x7f6d400c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:52.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.147+0000 7f6d3fe00640 1 --2- 172.21.15.102:0/1234659617 >> v2:172.21.15.102:3300/0 conn(0x7f6d400c97d0 0x7f6d400c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.148+0000 7f6d3fe00640 1 -- 172.21.15.102:0/1234659617 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4015dc00 con 0x7f6d400701f0 2024-03-18T16:21:52.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.148+0000 7f6d44e00640 1 --2- 172.21.15.102:0/1234659617 >> v2:172.21.15.154:3300/0 conn(0x7f6d40154990 0x7f6d4015af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:52.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.150+0000 7f6d3fe00640 1 --2- 172.21.15.102:0/1234659617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d400701f0 0x7f6d40070630 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6d40072420 tx=0x7f6d2802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d9de4499b792e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:52.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.154+0000 7f6d3f400640 1 -- 172.21.15.102:0/1234659617 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d28039070 con 0x7f6d400701f0 2024-03-18T16:21:52.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.154+0000 7f6d3f400640 1 -- 172.21.15.102:0/1234659617 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d2802cc60 con 0x7f6d400701f0 2024-03-18T16:21:52.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.154+0000 7f6d3f400640 1 -- 172.21.15.102:0/1234659617 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d28034660 con 0x7f6d400701f0 2024-03-18T16:21:52.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.154+0000 7f6d47800640 1 -- 172.21.15.102:0/1234659617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d400701f0 msgr2=0x7f6d40070630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:52.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.154+0000 7f6d47800640 1 --2- 172.21.15.102:0/1234659617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d400701f0 0x7f6d40070630 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6d40072420 tx=0x7f6d2802beb0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.155+0000 7f6d47800640 1 -- 172.21.15.102:0/1234659617 shutdown_connections 2024-03-18T16:21:52.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.155+0000 7f6d47800640 1 --2- 172.21.15.102:0/1234659617 >> v2:172.21.15.102:3300/0 conn(0x7f6d400c97d0 0x7f6d400c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.155+0000 7f6d47800640 1 --2- 172.21.15.102:0/1234659617 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d400701f0 0x7f6d40070630 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.155+0000 7f6d47800640 1 --2- 172.21.15.102:0/1234659617 >> v2:172.21.15.154:3300/0 conn(0x7f6d40154990 0x7f6d4015af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.155+0000 7f6d47800640 1 -- 172.21.15.102:0/1234659617 >> 172.21.15.102:0/1234659617 conn(0x7f6d40073040 msgr2=0x7f6d4006c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:52.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.155+0000 7f6d47800640 1 -- 172.21.15.102:0/1234659617 shutdown_connections 2024-03-18T16:21:52.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.155+0000 7f6d47800640 1 -- 172.21.15.102:0/1234659617 wait complete. 2024-03-18T16:21:52.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.157+0000 7f6d47800640 1 Processor -- start 2024-03-18T16:21:52.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.157+0000 7f6d47800640 1 -- start start 2024-03-18T16:21:52.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.158+0000 7f6d47800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d400701f0 0x7f6d400cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:52.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.158+0000 7f6d47800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d400c97d0 0x7f6d40152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:52.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.158+0000 7f6d47800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d40154990 0x7f6d400cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:52.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.158+0000 7f6d47800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4015e4a0 con 0x7f6d400c97d0 2024-03-18T16:21:52.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.158+0000 7f6d47800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4015e1c0 con 0x7f6d400701f0 2024-03-18T16:21:52.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.158+0000 7f6d47800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d400c7900 con 0x7f6d40154990 2024-03-18T16:21:52.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.159+0000 7f6d44e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d400701f0 0x7f6d400cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:52.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.159+0000 7f6d45800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d40154990 0x7f6d400cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:52.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.159+0000 7f6d3fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d400c97d0 0x7f6d40152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:52.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.159+0000 7f6d44e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d400701f0 0x7f6d400cacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42938/0 (socket says 172.21.15.102:42938) 2024-03-18T16:21:52.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.159+0000 7f6d44e00640 1 -- 172.21.15.102:0/2514379926 learned_addr learned my addr 172.21.15.102:0/2514379926 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:52.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.162+0000 7f6d45800640 1 -- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d400701f0 msgr2=0x7f6d400cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:52.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.162+0000 7f6d45800640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d400701f0 0x7f6d400cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.162+0000 7f6d45800640 1 -- 172.21.15.102:0/2514379926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d400c97d0 msgr2=0x7f6d40152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:52.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.162+0000 7f6d45800640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d400c97d0 0x7f6d40152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.162+0000 7f6d45800640 1 -- 172.21.15.102:0/2514379926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d401f4030 con 0x7f6d40154990 2024-03-18T16:21:52.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.165+0000 7f6d45800640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d40154990 0x7f6d400cb8c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6d3800e820 tx=0x7f6d3800ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:52.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.169+0000 7f6d3cc00640 1 -- 172.21.15.102:0/2514379926 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d38017070 con 0x7f6d40154990 2024-03-18T16:21:52.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.169+0000 7f6d3cc00640 1 -- 172.21.15.102:0/2514379926 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d3800ee70 con 0x7f6d40154990 2024-03-18T16:21:52.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.169+0000 7f6d3cc00640 1 -- 172.21.15.102:0/2514379926 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3800cb40 con 0x7f6d40154990 2024-03-18T16:21:52.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.169+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d401f02c0 con 0x7f6d40154990 2024-03-18T16:21:52.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.169+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d401f0560 con 0x7f6d40154990 2024-03-18T16:21:52.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.170+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d40074fd0 con 0x7f6d40154990 2024-03-18T16:21:52.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.188+0000 7f6d3cc00640 1 -- 172.21.15.102:0/2514379926 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6d3801a070 con 0x7f6d40154990 2024-03-18T16:21:52.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.190+0000 7f6d3cc00640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d1c06bc20 0x7f6d1c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:52.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.190+0000 7f6d44e00640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d1c06bc20 0x7f6d1c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:52.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.191+0000 7f6d44e00640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d1c06bc20 0x7f6d1c06e0e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009630 tx=0x7f6d30009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:52.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.191+0000 7f6d3cc00640 1 -- 172.21.15.102:0/2514379926 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6532+0+0 (secure 0 0 0) 0x7f6d38056d50 con 0x7f6d40154990 2024-03-18T16:21:52.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.438+0000 7f6d3cc00640 1 -- 172.21.15.102:0/2514379926 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6d38058cb0 con 0x7f6d40154990 2024-03-18T16:21:52.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.657+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6d4006f8e0 con 0x7f6d40154990 2024-03-18T16:21:52.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.663+0000 7f6d3cc00640 1 -- 172.21.15.102:0/2514379926 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f6d3805cd50 con 0x7f6d40154990 2024-03-18T16:21:52.664 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-18T16:21:52.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.665+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d1c06bc20 msgr2=0x7f6d1c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:52.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.665+0000 7f6d47800640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d1c06bc20 0x7f6d1c06e0e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009630 tx=0x7f6d30009240 comp rx=0 tx=0).stop 2024-03-18T16:21:52.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.665+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d40154990 msgr2=0x7f6d400cb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:52.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.665+0000 7f6d47800640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d40154990 0x7f6d400cb8c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6d3800e820 tx=0x7f6d3800ec20 comp rx=0 tx=0).stop 2024-03-18T16:21:52.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.666+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 shutdown_connections 2024-03-18T16:21:52.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.666+0000 7f6d47800640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d1c06bc20 0x7f6d1c06e0e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.666+0000 7f6d47800640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d40154990 0x7f6d400cb8c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.666+0000 7f6d47800640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d400c97d0 0x7f6d40152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.666+0000 7f6d47800640 1 --2- 172.21.15.102:0/2514379926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d400701f0 0x7f6d400cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.666+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 >> 172.21.15.102:0/2514379926 conn(0x7f6d40073040 msgr2=0x7f6d4015bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.666+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 shutdown_connections 2024-03-18T16:21:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:52.666+0000 7f6d47800640 1 -- 172.21.15.102:0/2514379926 wait complete. 2024-03-18T16:21:52.685 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi102.front.sepia.ceph.com... 2024-03-18T16:21:52.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:52.686 DEBUG:teuthology.orchestra.run.smithi102:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:21:52.766 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:ceph-fuse[42266]: fuse finished with error 0 and tester_r 0 2024-03-18T16:21:52.771 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:21:58.873 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:21:58.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:58.873 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-18T16:21:58.938 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:59.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.140+0000 7f24e5c00640 1 Processor -- start 2024-03-18T16:21:59.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.140+0000 7f24e5c00640 1 -- start start 2024-03-18T16:21:59.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.142+0000 7f24e5c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24e01546e0 0x7f24e00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.142+0000 7f24e5c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24e0070e00 0x7f24e00c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.142+0000 7f24e5c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24e00c7c30 con 0x7f24e00c9af0 2024-03-18T16:21:59.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.142+0000 7f24e5c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f24e00731b0 con 0x7f24e0070e00 2024-03-18T16:21:59.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.142+0000 7f24e5c00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f24e0073770 con 0x7f24e01546e0 2024-03-18T16:21:59.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.165+0000 7f24dfe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24e01546e0 0x7f24e00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:59.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.165+0000 7f24dfe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24e01546e0 0x7f24e00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42942/0 (socket says 172.21.15.102:42942) 2024-03-18T16:21:59.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.165+0000 7f24dfe00640 1 -- 172.21.15.102:0/578938746 learned_addr learned my addr 172.21.15.102:0/578938746 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:59.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.167+0000 7f24dfe00640 1 -- 172.21.15.102:0/578938746 >> v2:172.21.15.102:3300/0 conn(0x7f24e0070e00 msgr2=0x7f24e00c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:59.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.167+0000 7f24dfe00640 1 --2- 172.21.15.102:0/578938746 >> v2:172.21.15.102:3300/0 conn(0x7f24e0070e00 0x7f24e00c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.168+0000 7f24dfe00640 1 -- 172.21.15.102:0/578938746 >> v1:172.21.15.102:6789/0 conn(0x7f24e00c9af0 legacy=0x7f24e015a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:21:59.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.168+0000 7f24dfe00640 1 -- 172.21.15.102:0/578938746 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e00c38d0 con 0x7f24e01546e0 2024-03-18T16:21:59.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.170+0000 7f24dfe00640 1 --2- 172.21.15.102:0/578938746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e01546e0 0x7f24e00702c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f24e006c270 tx=0x7f24d002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0a1bc74f2336d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:59.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.172+0000 7f24dea00640 1 -- 172.21.15.102:0/578938746 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d002e070 con 0x7f24e01546e0 2024-03-18T16:21:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.173+0000 7f24dea00640 1 -- 172.21.15.102:0/578938746 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24d0037070 con 0x7f24e01546e0 2024-03-18T16:21:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.174+0000 7f24dea00640 1 -- 172.21.15.102:0/578938746 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d0031a00 con 0x7f24e01546e0 2024-03-18T16:21:59.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.175+0000 7f24e5c00640 1 -- 172.21.15.102:0/578938746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e01546e0 msgr2=0x7f24e00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.175+0000 7f24e5c00640 1 --2- 172.21.15.102:0/578938746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e01546e0 0x7f24e00702c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f24e006c270 tx=0x7f24d002b880 comp rx=0 tx=0).stop 2024-03-18T16:21:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.175+0000 7f24e5c00640 1 -- 172.21.15.102:0/578938746 shutdown_connections 2024-03-18T16:21:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.175+0000 7f24e5c00640 1 --2- 172.21.15.102:0/578938746 >> v2:172.21.15.102:3300/0 conn(0x7f24e0070e00 0x7f24e00c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.175+0000 7f24e5c00640 1 --2- 172.21.15.102:0/578938746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e01546e0 0x7f24e00702c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.175+0000 7f24e5c00640 1 -- 172.21.15.102:0/578938746 >> 172.21.15.102:0/578938746 conn(0x7f24e00ccb50 msgr2=0x7f24e00ccf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.175+0000 7f24e5c00640 1 -- 172.21.15.102:0/578938746 shutdown_connections 2024-03-18T16:21:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.175+0000 7f24e5c00640 1 -- 172.21.15.102:0/578938746 wait complete. 2024-03-18T16:21:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.179+0000 7f24e5c00640 1 Processor -- start 2024-03-18T16:21:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.179+0000 7f24e5c00640 1 -- start start 2024-03-18T16:21:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.181+0000 7f24e5c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e0070e00 0x7f24e014f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.181+0000 7f24e5c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24e00c9af0 0x7f24e014fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.181+0000 7f24e5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e01546e0 0x7f24e0141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.181+0000 7f24e5c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24e0070ac0 con 0x7f24e01546e0 2024-03-18T16:21:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.181+0000 7f24e5c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24e0073400 con 0x7f24e0070e00 2024-03-18T16:21:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.181+0000 7f24e5c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24e00c78f0 con 0x7f24e00c9af0 2024-03-18T16:21:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.182+0000 7f24dfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e0070e00 0x7f24e014f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.182+0000 7f24df400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24e00c9af0 0x7f24e014fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.182+0000 7f24dfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e0070e00 0x7f24e014f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42954/0 (socket says 172.21.15.102:42954) 2024-03-18T16:21:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.182+0000 7f24dfe00640 1 -- 172.21.15.102:0/314531799 learned_addr learned my addr 172.21.15.102:0/314531799 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.185+0000 7f24dfe00640 1 -- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24e00c9af0 msgr2=0x7f24e014fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.185+0000 7f24dfe00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24e00c9af0 0x7f24e014fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.185+0000 7f24dfe00640 1 -- 172.21.15.102:0/314531799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e01546e0 msgr2=0x7f24e0141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.185+0000 7f24dfe00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e01546e0 0x7f24e0141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.185+0000 7f24dfe00640 1 -- 172.21.15.102:0/314531799 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e014f830 con 0x7f24e0070e00 2024-03-18T16:21:59.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.188+0000 7f24dfe00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e0070e00 0x7f24e014f0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f24d002b8c0 tx=0x7f24d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.192+0000 7f24d7e00640 1 -- 172.21.15.102:0/314531799 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d0032880 con 0x7f24e0070e00 2024-03-18T16:21:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.192+0000 7f24d7e00640 1 -- 172.21.15.102:0/314531799 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24d0032a20 con 0x7f24e0070e00 2024-03-18T16:21:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.193+0000 7f24d7e00640 1 -- 172.21.15.102:0/314531799 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d0032e10 con 0x7f24e0070e00 2024-03-18T16:21:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.193+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e0150ba0 con 0x7f24e0070e00 2024-03-18T16:21:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.193+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e0159c10 con 0x7f24e0070e00 2024-03-18T16:21:59.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.194+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e0074fd0 con 0x7f24e0070e00 2024-03-18T16:21:59.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.215+0000 7f24d7e00640 1 -- 172.21.15.102:0/314531799 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f24d00453f0 con 0x7f24e0070e00 2024-03-18T16:21:59.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.217+0000 7f24d7e00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24c006bc20 0x7f24c006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.217+0000 7f24d7e00640 1 -- 172.21.15.102:0/314531799 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f24d00793f0 con 0x7f24e0070e00 2024-03-18T16:21:59.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.217+0000 7f24df400640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24c006bc20 0x7f24c006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:59.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.218+0000 7f24df400640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24c006bc20 0x7f24c006e0e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f24e01f4250 tx=0x7f24cc009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:59.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.469+0000 7f24d7e00640 1 -- 172.21.15.102:0/314531799 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f24d0053080 con 0x7f24e0070e00 2024-03-18T16:21:59.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.680+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24e01453a0 con 0x7f24e0070e00 2024-03-18T16:21:59.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.727+0000 7f24d7e00640 1 -- 172.21.15.102:0/314531799 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18907 (secure 0 0 0) 0x7f24e014f830 con 0x7f24e0070e00 2024-03-18T16:21:59.732 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:21:59.732 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":29,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:54.749120+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:59.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.730+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24c006bc20 msgr2=0x7f24c006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.730+0000 7f24e5c00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24c006bc20 0x7f24c006e0e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f24e01f4250 tx=0x7f24cc009060 comp rx=0 tx=0).stop 2024-03-18T16:21:59.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.730+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e0070e00 msgr2=0x7f24e014f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.730+0000 7f24e5c00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e0070e00 0x7f24e014f0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f24d002b8c0 tx=0x7f24d00046c0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.730+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 shutdown_connections 2024-03-18T16:21:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.730+0000 7f24e5c00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24c006bc20 0x7f24c006e0e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.730+0000 7f24e5c00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e01546e0 0x7f24e0141700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.730+0000 7f24e5c00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24e00c9af0 0x7f24e014fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.730+0000 7f24e5c00640 1 --2- 172.21.15.102:0/314531799 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24e0070e00 0x7f24e014f0e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.731+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 >> 172.21.15.102:0/314531799 conn(0x7f24e00ccb50 msgr2=0x7f24e0153300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.731+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 shutdown_connections 2024-03-18T16:21:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.731+0000 7f24e5c00640 1 -- 172.21.15.102:0/314531799 wait complete. 2024-03-18T16:21:59.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:59.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.960+0000 7f3939600640 1 Processor -- start 2024-03-18T16:21:59.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.961+0000 7f3939600640 1 -- start start 2024-03-18T16:21:59.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.962+0000 7f3939600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3934158420 0x7f393415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.962+0000 7f3939600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39340c72d0 con 0x7f3934141650 2024-03-18T16:21:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.962+0000 7f3939600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3934158840 con 0x7f39340c4fa0 2024-03-18T16:21:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.962+0000 7f3939600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f393415c8e0 con 0x7f3934158420 2024-03-18T16:21:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.963+0000 7f3933e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3934141650 0x7f39340c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34700/0 (socket says 172.21.15.102:34700) 2024-03-18T16:21:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.963+0000 7f3933e00640 1 -- 172.21.15.102:0/2492302381 learned_addr learned my addr 172.21.15.102:0/2492302381 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:21:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.963+0000 7f392b400640 1 --2- 172.21.15.102:0/2492302381 >> v2:172.21.15.154:3300/0 conn(0x7f3934158420 0x7f393415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:59.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.967+0000 7f3933400640 1 -- 172.21.15.102:0/2492302381 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1332329242 0 0) 0x7f39340c72d0 con 0x7f3934141650 2024-03-18T16:21:59.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.967+0000 7f392b400640 1 -- 172.21.15.102:0/2492302381 >> v1:172.21.15.154:6789/0 conn(0x7f39340c4fa0 legacy=0x7f3934162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.967+0000 7f392b400640 1 -- 172.21.15.102:0/2492302381 >> v1:172.21.15.102:6789/0 conn(0x7f3934141650 legacy=0x7f39340c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.967+0000 7f392b400640 1 -- 172.21.15.102:0/2492302381 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3934158ad0 con 0x7f3934158420 2024-03-18T16:21:59.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.972+0000 7f392b400640 1 --2- 172.21.15.102:0/2492302381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934158420 0x7f393415c3a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f391c0097a0 tx=0x7f391c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e49cd90a58816a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:59.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.976+0000 7f3933400640 1 -- 172.21.15.102:0/2492302381 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c034070 con 0x7f3934158420 2024-03-18T16:21:59.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.976+0000 7f3933400640 1 -- 172.21.15.102:0/2492302381 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f391c030610 con 0x7f3934158420 2024-03-18T16:21:59.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.976+0000 7f3933400640 1 -- 172.21.15.102:0/2492302381 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c0309e0 con 0x7f3934158420 2024-03-18T16:21:59.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.976+0000 7f3939600640 1 -- 172.21.15.102:0/2492302381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934158420 msgr2=0x7f393415c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.976+0000 7f3939600640 1 --2- 172.21.15.102:0/2492302381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934158420 0x7f393415c3a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f391c0097a0 tx=0x7f391c02b960 comp rx=0 tx=0).stop 2024-03-18T16:21:59.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.976+0000 7f3939600640 1 -- 172.21.15.102:0/2492302381 shutdown_connections 2024-03-18T16:21:59.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.976+0000 7f3939600640 1 --2- 172.21.15.102:0/2492302381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934158420 0x7f393415c3a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.976+0000 7f3939600640 1 -- 172.21.15.102:0/2492302381 >> 172.21.15.102:0/2492302381 conn(0x7f39340cb890 msgr2=0x7f39340cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:59.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.977+0000 7f3939600640 1 -- 172.21.15.102:0/2492302381 shutdown_connections 2024-03-18T16:21:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.977+0000 7f3939600640 1 -- 172.21.15.102:0/2492302381 wait complete. 2024-03-18T16:21:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.978+0000 7f3939600640 1 Processor -- start 2024-03-18T16:21:59.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.978+0000 7f3939600640 1 -- start start 2024-03-18T16:21:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.979+0000 7f3939600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39340c4fa0 0x7f39341ec240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.979+0000 7f3939600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934141650 0x7f39341ec780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.979+0000 7f3939600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3934158420 0x7f3934138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.979+0000 7f3939600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3934159090 con 0x7f39340c4fa0 2024-03-18T16:21:59.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.979+0000 7f3939600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3934159370 con 0x7f3934141650 2024-03-18T16:21:59.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.979+0000 7f3939600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f39340c63c0 con 0x7f3934158420 2024-03-18T16:21:59.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.980+0000 7f392b400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934141650 0x7f39341ec780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:59.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.980+0000 7f3933e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39340c4fa0 0x7f39341ec240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:59.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.980+0000 7f392b400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934141650 0x7f39341ec780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42962/0 (socket says 172.21.15.102:42962) 2024-03-18T16:21:59.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.980+0000 7f392b400640 1 -- 172.21.15.102:0/707881619 learned_addr learned my addr 172.21.15.102:0/707881619 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:21:59.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.980+0000 7f3938c00640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3934158420 0x7f3934138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.983+0000 7f3933e00640 1 -- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3934158420 msgr2=0x7f3934138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.983+0000 7f3933e00640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3934158420 0x7f3934138000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.983+0000 7f3933e00640 1 -- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934141650 msgr2=0x7f39341ec780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.983+0000 7f3933e00640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934141650 0x7f39341ec780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.983+0000 7f3933e00640 1 -- 172.21.15.102:0/707881619 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393413d580 con 0x7f39340c4fa0 2024-03-18T16:21:59.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.986+0000 7f3933e00640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39340c4fa0 0x7f39341ec240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f392400c410 tx=0x7f392400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:59.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.989+0000 7f3930c00640 1 -- 172.21.15.102:0/707881619 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39240074e0 con 0x7f39340c4fa0 2024-03-18T16:21:59.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.989+0000 7f3930c00640 1 -- 172.21.15.102:0/707881619 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39240047e0 con 0x7f39340c4fa0 2024-03-18T16:21:59.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.989+0000 7f3930c00640 1 -- 172.21.15.102:0/707881619 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3924004bd0 con 0x7f39340c4fa0 2024-03-18T16:21:59.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.989+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3934157ae0 con 0x7f39340c4fa0 2024-03-18T16:21:59.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.989+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3934159720 con 0x7f39340c4fa0 2024-03-18T16:21:59.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:21:59.991+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f393406e3d0 con 0x7f39340c4fa0 2024-03-18T16:22:00.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.013+0000 7f3930c00640 1 -- 172.21.15.102:0/707881619 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3924011070 con 0x7f39340c4fa0 2024-03-18T16:22:00.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.014+0000 7f3930c00640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f391006bc20 0x7f391006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:00.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.015+0000 7f392b400640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f391006bc20 0x7f391006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:00.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.016+0000 7f392b400640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f391006bc20 0x7f391006e0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f391c02b9a0 tx=0x7f391c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:00.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.016+0000 7f3930c00640 1 -- 172.21.15.102:0/707881619 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f3924057130 con 0x7f39340c4fa0 2024-03-18T16:22:00.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.322+0000 7f3930c00640 1 -- 172.21.15.102:0/707881619 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f393406e3d0 con 0x7f39340c4fa0 2024-03-18T16:22:00.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.534+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3934074fd0 con 0x7f39340c4fa0 2024-03-18T16:22:00.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.732+0000 7f3930c00640 1 -- 172.21.15.102:0/707881619 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18907 (secure 0 0 0) 0x7f3924059090 con 0x7f39340c4fa0 2024-03-18T16:22:00.736 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:00.736 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":29,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:54.749120+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:22:00.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f391006bc20 msgr2=0x7f391006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:00.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f391006bc20 0x7f391006e0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f391c02b9a0 tx=0x7f391c03b040 comp rx=0 tx=0).stop 2024-03-18T16:22:00.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39340c4fa0 msgr2=0x7f39341ec240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:00.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39340c4fa0 0x7f39341ec240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f392400c410 tx=0x7f392400c8d0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 shutdown_connections 2024-03-18T16:22:00.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f391006bc20 0x7f391006e0e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3934158420 0x7f3934138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3934141650 0x7f39341ec780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 --2- 172.21.15.102:0/707881619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39340c4fa0 0x7f39341ec240 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 >> 172.21.15.102:0/707881619 conn(0x7f39340cb890 msgr2=0x7f3934071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:00.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 shutdown_connections 2024-03-18T16:22:00.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.736+0000 7f3939600640 1 -- 172.21.15.102:0/707881619 wait complete. 2024-03-18T16:22:00.758 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T16:22:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.964+0000 7fd46a000640 1 Processor -- start 2024-03-18T16:22:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.965+0000 7fd46a000640 1 -- start start 2024-03-18T16:22:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.966+0000 7fd46a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 0x7fd4640723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.966+0000 7fd46a000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd46406ba70 0x7fd46406be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.966+0000 7fd46a000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd464158390 con 0x7fd4640c4fb0 2024-03-18T16:22:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.966+0000 7fd46a000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd464141ae0 con 0x7fd46406ba70 2024-03-18T16:22:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.966+0000 7fd46a000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd464158670 con 0x7fd464141700 2024-03-18T16:22:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.967+0000 7fd463e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd4640c4fb0 0x7fd46415bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56388/0 (socket says 172.21.15.102:56388) 2024-03-18T16:22:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.967+0000 7fd463e00640 1 -- 172.21.15.102:0/507135443 learned_addr learned my addr 172.21.15.102:0/507135443 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.967+0000 7fd463400640 1 --2- 172.21.15.102:0/507135443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 0x7fd4640723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:00.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.970+0000 7fd463400640 1 -- 172.21.15.102:0/507135443 >> v2:172.21.15.102:3300/0 conn(0x7fd46406ba70 msgr2=0x7fd46406be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:00.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.970+0000 7fd463400640 1 --2- 172.21.15.102:0/507135443 >> v2:172.21.15.102:3300/0 conn(0x7fd46406ba70 0x7fd46406be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.970+0000 7fd463400640 1 -- 172.21.15.102:0/507135443 >> v1:172.21.15.102:6789/0 conn(0x7fd4640c4fb0 legacy=0x7fd46415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:00.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.970+0000 7fd463400640 1 -- 172.21.15.102:0/507135443 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd464158950 con 0x7fd464141700 2024-03-18T16:22:00.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.972+0000 7fd463400640 1 --2- 172.21.15.102:0/507135443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 0x7fd4640723b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd4540097a0 tx=0x7fd45402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f83dfaab793408d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:00.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.976+0000 7fd462000640 1 -- 172.21.15.102:0/507135443 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454034070 con 0x7fd464141700 2024-03-18T16:22:00.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.976+0000 7fd462000640 1 -- 172.21.15.102:0/507135443 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd454030800 con 0x7fd464141700 2024-03-18T16:22:00.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.976+0000 7fd462000640 1 -- 172.21.15.102:0/507135443 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454030bd0 con 0x7fd464141700 2024-03-18T16:22:00.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.976+0000 7fd46a000640 1 -- 172.21.15.102:0/507135443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 msgr2=0x7fd4640723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:00.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.976+0000 7fd46a000640 1 --2- 172.21.15.102:0/507135443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 0x7fd4640723b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd4540097a0 tx=0x7fd45402b940 comp rx=0 tx=0).stop 2024-03-18T16:22:00.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.976+0000 7fd46a000640 1 -- 172.21.15.102:0/507135443 shutdown_connections 2024-03-18T16:22:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.976+0000 7fd46a000640 1 --2- 172.21.15.102:0/507135443 >> v2:172.21.15.102:3300/0 conn(0x7fd46406ba70 0x7fd46406be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.976+0000 7fd46a000640 1 --2- 172.21.15.102:0/507135443 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 0x7fd4640723b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.976+0000 7fd46a000640 1 -- 172.21.15.102:0/507135443 >> 172.21.15.102:0/507135443 conn(0x7fd4640c83a0 msgr2=0x7fd4640c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.977+0000 7fd46a000640 1 -- 172.21.15.102:0/507135443 shutdown_connections 2024-03-18T16:22:00.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.977+0000 7fd46a000640 1 -- 172.21.15.102:0/507135443 wait complete. 2024-03-18T16:22:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.978+0000 7fd46a000640 1 Processor -- start 2024-03-18T16:22:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.978+0000 7fd46a000640 1 -- start start 2024-03-18T16:22:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.979+0000 7fd46a000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd46406ba70 0x7fd46415b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.979+0000 7fd46a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4640c4fb0 0x7fd4640c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.979+0000 7fd46a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 0x7fd4641371d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.979+0000 7fd46a000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd46415c260 con 0x7fd4640c4fb0 2024-03-18T16:22:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.979+0000 7fd46a000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd464158f10 con 0x7fd46406ba70 2024-03-18T16:22:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.979+0000 7fd46a000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd46415c3e0 con 0x7fd464141700 2024-03-18T16:22:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.980+0000 7fd463e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 0x7fd4641371d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.980+0000 7fd462a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4640c4fb0 0x7fd4640c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.980+0000 7fd463400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd46406ba70 0x7fd46415b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.980+0000 7fd462a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4640c4fb0 0x7fd4640c45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57206/0 (socket says 172.21.15.102:57206) 2024-03-18T16:22:00.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.980+0000 7fd462a00640 1 -- 172.21.15.102:0/56928292 learned_addr learned my addr 172.21.15.102:0/56928292 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:00.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.984+0000 7fd462a00640 1 -- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 msgr2=0x7fd4641371d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:00.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.984+0000 7fd462a00640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 0x7fd4641371d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.984+0000 7fd462a00640 1 -- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd46406ba70 msgr2=0x7fd46415b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:00.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.984+0000 7fd462a00640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd46406ba70 0x7fd46415b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:00.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.984+0000 7fd462a00640 1 -- 172.21.15.102:0/56928292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4641f1130 con 0x7fd4640c4fb0 2024-03-18T16:22:00.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.987+0000 7fd462a00640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4640c4fb0 0x7fd4640c45e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd45000b4d0 tx=0x7fd45000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:00.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.991+0000 7fd45b400640 1 -- 172.21.15.102:0/56928292 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd450014070 con 0x7fd4640c4fb0 2024-03-18T16:22:00.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.991+0000 7fd45b400640 1 -- 172.21.15.102:0/56928292 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd450010870 con 0x7fd4640c4fb0 2024-03-18T16:22:00.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.991+0000 7fd45b400640 1 -- 172.21.15.102:0/56928292 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd450010c60 con 0x7fd4640c4fb0 2024-03-18T16:22:00.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.991+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4641593e0 con 0x7fd4640c4fb0 2024-03-18T16:22:00.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.991+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4641f1410 con 0x7fd4640c4fb0 2024-03-18T16:22:00.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:00.992+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd464074fd0 con 0x7fd4640c4fb0 2024-03-18T16:22:01.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:01.013+0000 7fd45b400640 1 -- 172.21.15.102:0/56928292 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd450017070 con 0x7fd4640c4fb0 2024-03-18T16:22:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:01.014+0000 7fd45b400640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd44406bc20 0x7fd44406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:01.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:01.015+0000 7fd463400640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd44406bc20 0x7fd44406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:01.016+0000 7fd463400640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd44406bc20 0x7fd44406e0e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd454036000 tx=0x7fd45403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:01.017+0000 7fd45b400640 1 -- 172.21.15.102:0/56928292 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fd450057110 con 0x7fd4640c4fb0 2024-03-18T16:22:01.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:01.277+0000 7fd45b400640 1 -- 172.21.15.102:0/56928292 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd450059070 con 0x7fd4640c4fb0 2024-03-18T16:22:01.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:01.530+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd464140f20 con 0x7fd4640c4fb0 2024-03-18T16:22:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.831+0000 7fd45b400640 1 -- 172.21.15.102:0/56928292 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7fd45005d110 con 0x7fd4640c4fb0 2024-03-18T16:22:02.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd44406bc20 msgr2=0x7fd44406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:02.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd44406bc20 0x7fd44406e0e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd454036000 tx=0x7fd45403b040 comp rx=0 tx=0).stop 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4640c4fb0 msgr2=0x7fd4640c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4640c4fb0 0x7fd4640c45e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd45000b4d0 tx=0x7fd45000b990 comp rx=0 tx=0).stop 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 shutdown_connections 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd44406bc20 0x7fd44406e0e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd464141700 0x7fd4641371d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4640c4fb0 0x7fd4640c45e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 --2- 172.21.15.102:0/56928292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd46406ba70 0x7fd46415b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 >> 172.21.15.102:0/56928292 conn(0x7fd4640c83a0 msgr2=0x7fd464072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.835+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 shutdown_connections 2024-03-18T16:22:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:02.836+0000 7fd46a000640 1 -- 172.21.15.102:0/56928292 wait complete. 2024-03-18T16:22:02.859 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2024-03-18T16:22:02.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-18T16:22:03.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.065+0000 7f1298c00640 1 Processor -- start 2024-03-18T16:22:03.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.066+0000 7f1298c00640 1 -- start start 2024-03-18T16:22:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.067+0000 7f1298c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1294141700 0x7f12940723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.067+0000 7f1298c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1294158390 con 0x7f129406ba70 2024-03-18T16:22:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.067+0000 7f1298c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1294158670 con 0x7f129406be90 2024-03-18T16:22:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.067+0000 7f1298c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1294141ae0 con 0x7f1294141700 2024-03-18T16:22:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.068+0000 7f1292800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f129406be90 0x7f129415bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33994/0 (socket says 172.21.15.102:33994) 2024-03-18T16:22:03.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.068+0000 7f1292800640 1 -- 172.21.15.102:0/1076688627 learned_addr learned my addr 172.21.15.102:0/1076688627 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:03.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.068+0000 7f1291e00640 1 --2- 172.21.15.102:0/1076688627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1294141700 0x7f12940723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:03.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.070+0000 7f1291e00640 1 -- 172.21.15.102:0/1076688627 >> v1:172.21.15.154:6789/0 conn(0x7f129406be90 legacy=0x7f129415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:03.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.071+0000 7f1291e00640 1 -- 172.21.15.102:0/1076688627 >> v1:172.21.15.102:6789/0 conn(0x7f129406ba70 legacy=0x7f12940c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:03.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.071+0000 7f1291400640 1 -- 172.21.15.102:0/1076688627 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686303674 0 0) 0x7f1294158670 con 0x7f129406be90 2024-03-18T16:22:03.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.071+0000 7f1291e00640 1 -- 172.21.15.102:0/1076688627 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1294158950 con 0x7f1294141700 2024-03-18T16:22:03.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.073+0000 7f1291e00640 1 --2- 172.21.15.102:0/1076688627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1294141700 0x7f12940723b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f12840097d0 tx=0x7f128402b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e34a81db54005595 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:03.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.077+0000 7f1291400640 1 -- 172.21.15.102:0/1076688627 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284034070 con 0x7f1294141700 2024-03-18T16:22:03.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.077+0000 7f1291400640 1 -- 172.21.15.102:0/1076688627 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1284030890 con 0x7f1294141700 2024-03-18T16:22:03.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.077+0000 7f1291400640 1 -- 172.21.15.102:0/1076688627 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284030c60 con 0x7f1294141700 2024-03-18T16:22:03.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.077+0000 7f1298c00640 1 -- 172.21.15.102:0/1076688627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1294141700 msgr2=0x7f12940723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:03.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.077+0000 7f1298c00640 1 --2- 172.21.15.102:0/1076688627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1294141700 0x7f12940723b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f12840097d0 tx=0x7f128402b9d0 comp rx=0 tx=0).stop 2024-03-18T16:22:03.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.077+0000 7f1298c00640 1 -- 172.21.15.102:0/1076688627 shutdown_connections 2024-03-18T16:22:03.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.077+0000 7f1298c00640 1 --2- 172.21.15.102:0/1076688627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1294141700 0x7f12940723b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:03.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.077+0000 7f1298c00640 1 -- 172.21.15.102:0/1076688627 >> 172.21.15.102:0/1076688627 conn(0x7f12940c83a0 msgr2=0x7f12940c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:03.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.078+0000 7f1298c00640 1 -- 172.21.15.102:0/1076688627 shutdown_connections 2024-03-18T16:22:03.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.078+0000 7f1298c00640 1 -- 172.21.15.102:0/1076688627 wait complete. 2024-03-18T16:22:03.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.079+0000 7f1298c00640 1 Processor -- start 2024-03-18T16:22:03.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.079+0000 7f1298c00640 1 -- start start 2024-03-18T16:22:03.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.080+0000 7f1298c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f129406ba70 0x7f12941e2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:03.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.080+0000 7f1298c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f129406be90 0x7f1294154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:03.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.080+0000 7f1298c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1294141700 0x7f1294153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.080+0000 7f1298c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1294158f10 con 0x7f1294141700 2024-03-18T16:22:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.080+0000 7f1298c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f129406c300 con 0x7f129406be90 2024-03-18T16:22:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.080+0000 7f1298c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f12940c56c0 con 0x7f129406ba70 2024-03-18T16:22:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.081+0000 7f1291e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f129406ba70 0x7f12941e2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.081+0000 7f1292800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1294141700 0x7f1294153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.081+0000 7f1289400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f129406be90 0x7f1294154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.081+0000 7f1291e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f129406ba70 0x7f12941e2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43758/0 (socket says 172.21.15.102:43758) 2024-03-18T16:22:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.081+0000 7f1291e00640 1 -- 172.21.15.102:0/2498142145 learned_addr learned my addr 172.21.15.102:0/2498142145 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:03.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.085+0000 7f1291e00640 1 -- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f129406be90 msgr2=0x7f1294154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:03.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.085+0000 7f1291e00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f129406be90 0x7f1294154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:03.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.085+0000 7f1291e00640 1 -- 172.21.15.102:0/2498142145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1294141700 msgr2=0x7f1294153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:03.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.085+0000 7f1291e00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1294141700 0x7f1294153f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:03.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.085+0000 7f1291e00640 1 -- 172.21.15.102:0/2498142145 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12941553c0 con 0x7f129406ba70 2024-03-18T16:22:03.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.087+0000 7f1291e00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f129406ba70 0x7f12941e2700 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f128402bfe0 tx=0x7f12840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:03.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.091+0000 7f128aa00640 1 -- 172.21.15.102:0/2498142145 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284034040 con 0x7f129406ba70 2024-03-18T16:22:03.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.091+0000 7f128aa00640 1 -- 172.21.15.102:0/2498142145 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12840443f0 con 0x7f129406ba70 2024-03-18T16:22:03.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.091+0000 7f128aa00640 1 -- 172.21.15.102:0/2498142145 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12840447e0 con 0x7f129406ba70 2024-03-18T16:22:03.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.091+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12941f15d0 con 0x7f129406ba70 2024-03-18T16:22:03.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.091+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12941f1b90 con 0x7f129406ba70 2024-03-18T16:22:03.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.092+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1294074fd0 con 0x7f129406ba70 2024-03-18T16:22:03.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.112+0000 7f128aa00640 1 -- 172.21.15.102:0/2498142145 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1284037070 con 0x7f129406ba70 2024-03-18T16:22:03.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.113+0000 7f128aa00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f127006bc50 0x7f127006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:03.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.114+0000 7f128aa00640 1 -- 172.21.15.102:0/2498142145 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f1284079af0 con 0x7f129406ba70 2024-03-18T16:22:03.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.114+0000 7f1289400640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f127006bc50 0x7f127006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:03.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.115+0000 7f1289400640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f127006bc50 0x7f127006e110 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f12940c54f0 tx=0x7f127c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:03.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.388+0000 7f128aa00640 1 -- 172.21.15.102:0/2498142145 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f128403f070 con 0x7f129406ba70 2024-03-18T16:22:03.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:03.640+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f12940c63a0 con 0x7f129406ba70 2024-03-18T16:22:04.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.121+0000 7f128aa00640 1 -- 172.21.15.102:0/2498142145 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v9) ==== 105+0+0 (secure 0 0 0) 0x7f128407ba50 con 0x7f129406ba70 2024-03-18T16:22:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f127006bc50 msgr2=0x7f127006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f127006bc50 0x7f127006e110 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f12940c54f0 tx=0x7f127c009040 comp rx=0 tx=0).stop 2024-03-18T16:22:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f129406ba70 msgr2=0x7f12941e2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f129406ba70 0x7f12941e2700 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f128402bfe0 tx=0x7f12840046c0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 shutdown_connections 2024-03-18T16:22:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f127006bc50 0x7f127006e110 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1294141700 0x7f1294153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f129406be90 0x7f1294154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 --2- 172.21.15.102:0/2498142145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f129406ba70 0x7f12941e2700 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 >> 172.21.15.102:0/2498142145 conn(0x7f12940c83a0 msgr2=0x7f1294140840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 shutdown_connections 2024-03-18T16:22:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.125+0000 7f1298c00640 1 -- 172.21.15.102:0/2498142145 wait complete. 2024-03-18T16:22:04.150 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-18T16:22:04.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.361+0000 7f99bb000640 1 Processor -- start 2024-03-18T16:22:04.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.362+0000 7f99bb000640 1 -- start start 2024-03-18T16:22:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.363+0000 7f99bb000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b4150f40 0x7f99b4071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.363+0000 7f99bb000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f99b40c6740 0x7f99b415dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.363+0000 7f99bb000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99b415e5e0 con 0x7f99b4073350 2024-03-18T16:22:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.363+0000 7f99bb000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f99b40717d0 con 0x7f99b40c6740 2024-03-18T16:22:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.363+0000 7f99bb000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f99b406d9d0 con 0x7f99b4150f40 2024-03-18T16:22:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.364+0000 7f99ba600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f99b40c6740 0x7f99b415dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.364+0000 7f99b9200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f99b4073350 0x7f99b415d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56410/0 (socket says 172.21.15.102:56410) 2024-03-18T16:22:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.364+0000 7f99b9200640 1 -- 172.21.15.102:0/3049115746 learned_addr learned my addr 172.21.15.102:0/3049115746 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.368+0000 7f99ba600640 1 -- 172.21.15.102:0/3049115746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b4150f40 msgr2=0x7f99b4071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.368+0000 7f99ba600640 1 --2- 172.21.15.102:0/3049115746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b4150f40 0x7f99b4071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.368+0000 7f99ba600640 1 -- 172.21.15.102:0/3049115746 >> v1:172.21.15.102:6789/0 conn(0x7f99b4073350 legacy=0x7f99b415d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:04.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.368+0000 7f99ba600640 1 -- 172.21.15.102:0/3049115746 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b406dcb0 con 0x7f99b40c6740 2024-03-18T16:22:04.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.373+0000 7f99ba600640 1 --2- 172.21.15.102:0/3049115746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 0x7f99b415dfa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f99ac009d90 tx=0x7f99ac02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d122891f236ff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.377+0000 7f99b3e00640 1 -- 172.21.15.102:0/3049115746 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac039070 con 0x7f99b40c6740 2024-03-18T16:22:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.377+0000 7f99b3e00640 1 -- 172.21.15.102:0/3049115746 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99ac02cce0 con 0x7f99b40c6740 2024-03-18T16:22:04.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.377+0000 7f99b3e00640 1 -- 172.21.15.102:0/3049115746 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac034660 con 0x7f99b40c6740 2024-03-18T16:22:04.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.378+0000 7f99bb000640 1 -- 172.21.15.102:0/3049115746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 msgr2=0x7f99b415dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:04.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.378+0000 7f99bb000640 1 --2- 172.21.15.102:0/3049115746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 0x7f99b415dfa0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f99ac009d90 tx=0x7f99ac02bf30 comp rx=0 tx=0).stop 2024-03-18T16:22:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.379+0000 7f99bb000640 1 -- 172.21.15.102:0/3049115746 shutdown_connections 2024-03-18T16:22:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.379+0000 7f99bb000640 1 --2- 172.21.15.102:0/3049115746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 0x7f99b415dfa0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.379+0000 7f99bb000640 1 --2- 172.21.15.102:0/3049115746 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b4150f40 0x7f99b4071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.379+0000 7f99bb000640 1 -- 172.21.15.102:0/3049115746 >> 172.21.15.102:0/3049115746 conn(0x7f99b406b200 msgr2=0x7f99b406b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.379+0000 7f99bb000640 1 -- 172.21.15.102:0/3049115746 shutdown_connections 2024-03-18T16:22:04.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.379+0000 7f99bb000640 1 -- 172.21.15.102:0/3049115746 wait complete. 2024-03-18T16:22:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.381+0000 7f99bb000640 1 Processor -- start 2024-03-18T16:22:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.381+0000 7f99bb000640 1 -- start start 2024-03-18T16:22:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.382+0000 7f99bb000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99b4073350 0x7f99b41e42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.382+0000 7f99bb000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 0x7f99b41e47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.382+0000 7f99bb000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b4150f40 0x7f99b415a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.382+0000 7f99bb000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99b41514a0 con 0x7f99b40c6740 2024-03-18T16:22:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.382+0000 7f99bb000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99b4151320 con 0x7f99b4073350 2024-03-18T16:22:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.382+0000 7f99bb000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f99b406e090 con 0x7f99b4150f40 2024-03-18T16:22:04.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.382+0000 7f99b9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 0x7f99b41e47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.383+0000 7f99b9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 0x7f99b41e47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57220/0 (socket says 172.21.15.102:57220) 2024-03-18T16:22:04.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.383+0000 7f99b9200640 1 -- 172.21.15.102:0/236708275 learned_addr learned my addr 172.21.15.102:0/236708275 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.387+0000 7f99b9200640 1 -- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b4150f40 msgr2=0x7f99b415a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.387+0000 7f99b9200640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b4150f40 0x7f99b415a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.387+0000 7f99b9200640 1 -- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99b4073350 msgr2=0x7f99b41e42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:04.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.387+0000 7f99b9200640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99b4073350 0x7f99b41e42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:04.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.387+0000 7f99b9200640 1 -- 172.21.15.102:0/236708275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b4157b80 con 0x7f99b40c6740 2024-03-18T16:22:04.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.389+0000 7f99b9200640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 0x7f99b41e47f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99a400c400 tx=0x7f99a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:04.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.394+0000 7f99b1600640 1 -- 172.21.15.102:0/236708275 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a4017070 con 0x7f99b40c6740 2024-03-18T16:22:04.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.394+0000 7f99b1600640 1 -- 172.21.15.102:0/236708275 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a4005ce0 con 0x7f99b40c6740 2024-03-18T16:22:04.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.394+0000 7f99b1600640 1 -- 172.21.15.102:0/236708275 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a40072f0 con 0x7f99b40c6740 2024-03-18T16:22:04.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.394+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b415d550 con 0x7f99b40c6740 2024-03-18T16:22:04.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.394+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b41e6030 con 0x7f99b40c6740 2024-03-18T16:22:04.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.395+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b4074fd0 con 0x7f99b40c6740 2024-03-18T16:22:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.419+0000 7f99b1600640 1 -- 172.21.15.102:0/236708275 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f99a401a070 con 0x7f99b40c6740 2024-03-18T16:22:04.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.420+0000 7f99b1600640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f999806bc50 0x7f999806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:04.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.420+0000 7f99b9c00640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f999806bc50 0x7f999806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:04.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.421+0000 7f99b9c00640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f999806bc50 0x7f999806e110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009670 tx=0x7f99a8006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:04.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.789+0000 7f99b1600640 1 -- 172.21.15.102:0/236708275 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f99a4056790 con 0x7f99b40c6740 2024-03-18T16:22:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:04.789+0000 7f99b1600640 1 -- 172.21.15.102:0/236708275 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f99a4021020 con 0x7f99b40c6740 2024-03-18T16:22:05.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:05.050+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f99b40ca3b0 con 0x7f99b40c6740 2024-03-18T16:22:06.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.349+0000 7f99b1600640 1 -- 172.21.15.102:0/236708275 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v11) ==== 122+0+0 (secure 0 0 0) 0x7f99a4058df0 con 0x7f99b40c6740 2024-03-18T16:22:06.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.352+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f999806bc50 msgr2=0x7f999806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:06.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.352+0000 7f99bb000640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f999806bc50 0x7f999806e110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009670 tx=0x7f99a8006bf0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.352+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 msgr2=0x7f99b41e47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:06.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.352+0000 7f99bb000640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 0x7f99b41e47f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99a400c400 tx=0x7f99a400c8c0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.352+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 shutdown_connections 2024-03-18T16:22:06.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.352+0000 7f99bb000640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f999806bc50 0x7f999806e110 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.353+0000 7f99bb000640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99b4150f40 0x7f99b415a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.353+0000 7f99bb000640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b40c6740 0x7f99b41e47f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.353+0000 7f99bb000640 1 --2- 172.21.15.102:0/236708275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99b4073350 0x7f99b41e42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.353+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 >> 172.21.15.102:0/236708275 conn(0x7f99b406b200 msgr2=0x7f99b414f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:06.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.353+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 shutdown_connections 2024-03-18T16:22:06.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.353+0000 7f99bb000640 1 -- 172.21.15.102:0/236708275 wait complete. 2024-03-18T16:22:06.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-18T16:22:06.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.586+0000 7f8f68e00640 1 Processor -- start 2024-03-18T16:22:06.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.586+0000 7f8f68e00640 1 -- start start 2024-03-18T16:22:06.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.588+0000 7f8f68e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8f64151290 0x7f8f6406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:06.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.588+0000 7f8f68e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f6406bcf0 0x7f8f64157b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:06.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.588+0000 7f8f68e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f6415adb0 con 0x7f8f6415a590 2024-03-18T16:22:06.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.588+0000 7f8f68e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8f6415b090 con 0x7f8f64151290 2024-03-18T16:22:06.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.588+0000 7f8f68e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8f6415b370 con 0x7f8f6406bcf0 2024-03-18T16:22:06.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.589+0000 7f8f63400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8f64151290 0x7f8f6406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:06.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.589+0000 7f8f5aa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f6406bcf0 0x7f8f64157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:06.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.589+0000 7f8f63e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8f6415a590 0x7f8f640728e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:34006/0 (socket says 172.21.15.102:34006) 2024-03-18T16:22:06.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.589+0000 7f8f63e00640 1 -- 172.21.15.102:0/4042142098 learned_addr learned my addr 172.21.15.102:0/4042142098 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:06.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.592+0000 7f8f5aa00640 1 -- 172.21.15.102:0/4042142098 >> v2:172.21.15.154:3300/0 conn(0x7f8f64151290 msgr2=0x7f8f6406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:06.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.592+0000 7f8f5aa00640 1 --2- 172.21.15.102:0/4042142098 >> v2:172.21.15.154:3300/0 conn(0x7f8f64151290 0x7f8f6406ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.592+0000 7f8f5aa00640 1 -- 172.21.15.102:0/4042142098 >> v1:172.21.15.154:6789/0 conn(0x7f8f6415a590 legacy=0x7f8f640728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:06.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.592+0000 7f8f5aa00640 1 -- 172.21.15.102:0/4042142098 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f64157370 con 0x7f8f6406bcf0 2024-03-18T16:22:06.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.596+0000 7f8f5aa00640 1 --2- 172.21.15.102:0/4042142098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f6406bcf0 0x7f8f64157b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c0097a0 tx=0x7f8f4c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ee76d661be2dd8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f62a00640 1 -- 172.21.15.102:0/4042142098 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c034070 con 0x7f8f6406bcf0 2024-03-18T16:22:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f62a00640 1 -- 172.21.15.102:0/4042142098 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f4c030610 con 0x7f8f6406bcf0 2024-03-18T16:22:06.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f62a00640 1 -- 172.21.15.102:0/4042142098 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c0309e0 con 0x7f8f6406bcf0 2024-03-18T16:22:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f68e00640 1 -- 172.21.15.102:0/4042142098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f6406bcf0 msgr2=0x7f8f64157b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f68e00640 1 --2- 172.21.15.102:0/4042142098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f6406bcf0 0x7f8f64157b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c0097a0 tx=0x7f8f4c02b960 comp rx=0 tx=0).stop 2024-03-18T16:22:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f68e00640 1 -- 172.21.15.102:0/4042142098 shutdown_connections 2024-03-18T16:22:06.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f68e00640 1 --2- 172.21.15.102:0/4042142098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f6406bcf0 0x7f8f64157b70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f68e00640 1 --2- 172.21.15.102:0/4042142098 >> v2:172.21.15.154:3300/0 conn(0x7f8f64151290 0x7f8f6406ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f68e00640 1 -- 172.21.15.102:0/4042142098 >> 172.21.15.102:0/4042142098 conn(0x7f8f6406b200 msgr2=0x7f8f6406b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f68e00640 1 -- 172.21.15.102:0/4042142098 shutdown_connections 2024-03-18T16:22:06.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.600+0000 7f8f68e00640 1 -- 172.21.15.102:0/4042142098 wait complete. 2024-03-18T16:22:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.602+0000 7f8f68e00640 1 Processor -- start 2024-03-18T16:22:06.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.602+0000 7f8f68e00640 1 -- start start 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.603+0000 7f8f68e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f64151290 0x7f8f641f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.603+0000 7f8f68e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f6415a590 0x7f8f640cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.603+0000 7f8f68e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f64155250 0x7f8f640c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.603+0000 7f8f68e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8f64073500 con 0x7f8f6415a590 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.603+0000 7f8f68e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8f64157020 con 0x7f8f64155250 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.603+0000 7f8f68e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8f64151670 con 0x7f8f64151290 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.604+0000 7f8f63400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f64151290 0x7f8f641f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.604+0000 7f8f63e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f64155250 0x7f8f640c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.604+0000 7f8f5aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f6415a590 0x7f8f640cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.604+0000 7f8f63e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f64155250 0x7f8f640c7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47330/0 (socket says 172.21.15.102:47330) 2024-03-18T16:22:06.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.604+0000 7f8f63e00640 1 -- 172.21.15.102:0/108623185 learned_addr learned my addr 172.21.15.102:0/108623185 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:06.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.609+0000 7f8f5aa00640 1 -- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f64151290 msgr2=0x7f8f641f2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:06.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.609+0000 7f8f5aa00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f64151290 0x7f8f641f2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.609+0000 7f8f5aa00640 1 -- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f64155250 msgr2=0x7f8f640c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:06.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.609+0000 7f8f5aa00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f64155250 0x7f8f640c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:06.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.609+0000 7f8f5aa00640 1 -- 172.21.15.102:0/108623185 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f64155820 con 0x7f8f6415a590 2024-03-18T16:22:06.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.612+0000 7f8f5aa00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f6415a590 0x7f8f640cad70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f6406ecb0 tx=0x7f8f4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:06.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.617+0000 7f8f5be00640 1 -- 172.21.15.102:0/108623185 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c046020 con 0x7f8f6415a590 2024-03-18T16:22:06.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.617+0000 7f8f5be00640 1 -- 172.21.15.102:0/108623185 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f4c030410 con 0x7f8f6415a590 2024-03-18T16:22:06.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.617+0000 7f8f5be00640 1 -- 172.21.15.102:0/108623185 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c034040 con 0x7f8f6415a590 2024-03-18T16:22:06.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.617+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f64156bb0 con 0x7f8f6415a590 2024-03-18T16:22:06.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.617+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f640c9750 con 0x7f8f6415a590 2024-03-18T16:22:06.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.618+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f64156db0 con 0x7f8f6415a590 2024-03-18T16:22:06.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.643+0000 7f8f5be00640 1 -- 172.21.15.102:0/108623185 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8f4c037070 con 0x7f8f6415a590 2024-03-18T16:22:06.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.644+0000 7f8f5be00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f4006bc50 0x7f8f4006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:06.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.644+0000 7f8f63400640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f4006bc50 0x7f8f4006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:06.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.645+0000 7f8f63400640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f4006bc50 0x7f8f4006e110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009b40 tx=0x7f8f54009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:06.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:06.646+0000 7f8f5be00640 1 -- 172.21.15.102:0/108623185 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f8f4c0305c0 con 0x7f8f6415a590 2024-03-18T16:22:07.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:07.055+0000 7f8f5be00640 1 -- 172.21.15.102:0/108623185 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8f4c050020 con 0x7f8f6415a590 2024-03-18T16:22:07.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:07.308+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f8f64074fd0 con 0x7f8f6415a590 2024-03-18T16:22:08.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.432+0000 7f8f5be00640 1 -- 172.21.15.102:0/108623185 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v12) ==== 115+0+0 (secure 0 0 0) 0x7f8f4c07b950 con 0x7f8f6415a590 2024-03-18T16:22:08.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f4006bc50 msgr2=0x7f8f4006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f4006bc50 0x7f8f4006e110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009b40 tx=0x7f8f54009190 comp rx=0 tx=0).stop 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f6415a590 msgr2=0x7f8f640cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f6415a590 0x7f8f640cad70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f6406ecb0 tx=0x7f8f4c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 shutdown_connections 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f4006bc50 0x7f8f4006e110 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f64155250 0x7f8f640c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f6415a590 0x7f8f640cad70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 --2- 172.21.15.102:0/108623185 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8f64151290 0x7f8f641f2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 >> 172.21.15.102:0/108623185 conn(0x7f8f6406b200 msgr2=0x7f8f640722b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 shutdown_connections 2024-03-18T16:22:08.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.435+0000 7f8f68e00640 1 -- 172.21.15.102:0/108623185 wait complete. 2024-03-18T16:22:08.460 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-18T16:22:08.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:22:08.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.660+0000 7f391ec00640 1 Processor -- start 2024-03-18T16:22:08.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.660+0000 7f391ec00640 1 -- start start 2024-03-18T16:22:08.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.660+0000 7f391ec00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f39180c7d50 0x7f3918158490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:08.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.661+0000 7f391ec00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c66a0 0x7f39181589d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:08.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.661+0000 7f391ec00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3918070260 con 0x7f39180c62c0 2024-03-18T16:22:08.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.661+0000 7f391ec00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3918070500 con 0x7f39180c7d50 2024-03-18T16:22:08.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.661+0000 7f391ec00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3918157f90 con 0x7f39180c66a0 2024-03-18T16:22:08.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.661+0000 7f391e200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c66a0 0x7f39181589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:08.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.662+0000 7f391e200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c66a0 0x7f39181589d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43802/0 (socket says 172.21.15.102:43802) 2024-03-18T16:22:08.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.662+0000 7f391e200640 1 -- 172.21.15.102:0/4004395926 learned_addr learned my addr 172.21.15.102:0/4004395926 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:08.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.665+0000 7f391e200640 1 -- 172.21.15.102:0/4004395926 >> v2:172.21.15.102:3300/0 conn(0x7f39180c7d50 msgr2=0x7f3918158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:08.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.665+0000 7f391e200640 1 --2- 172.21.15.102:0/4004395926 >> v2:172.21.15.102:3300/0 conn(0x7f39180c7d50 0x7f3918158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.665+0000 7f391e200640 1 -- 172.21.15.102:0/4004395926 >> v1:172.21.15.102:6789/0 conn(0x7f39180c62c0 legacy=0x7f391806bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:08.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.665+0000 7f391e200640 1 -- 172.21.15.102:0/4004395926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39180707c0 con 0x7f39180c66a0 2024-03-18T16:22:08.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.667+0000 7f391e200640 1 --2- 172.21.15.102:0/4004395926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c66a0 0x7f39181589d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f39180ca420 tx=0x7f391002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93c1bc11d93ff64f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:08.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f3917e00640 1 -- 172.21.15.102:0/4004395926 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391002c9c0 con 0x7f39180c66a0 2024-03-18T16:22:08.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f3917e00640 1 -- 172.21.15.102:0/4004395926 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f391002cb60 con 0x7f39180c66a0 2024-03-18T16:22:08.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f3917e00640 1 -- 172.21.15.102:0/4004395926 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39100343f0 con 0x7f39180c66a0 2024-03-18T16:22:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f391ec00640 1 -- 172.21.15.102:0/4004395926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c66a0 msgr2=0x7f39181589d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f391ec00640 1 --2- 172.21.15.102:0/4004395926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c66a0 0x7f39181589d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f39180ca420 tx=0x7f391002be20 comp rx=0 tx=0).stop 2024-03-18T16:22:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f391ec00640 1 -- 172.21.15.102:0/4004395926 shutdown_connections 2024-03-18T16:22:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f391ec00640 1 --2- 172.21.15.102:0/4004395926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c66a0 0x7f39181589d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f391ec00640 1 --2- 172.21.15.102:0/4004395926 >> v2:172.21.15.102:3300/0 conn(0x7f39180c7d50 0x7f3918158490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f391ec00640 1 -- 172.21.15.102:0/4004395926 >> 172.21.15.102:0/4004395926 conn(0x7f391806c310 msgr2=0x7f391806c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:08.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f391ec00640 1 -- 172.21.15.102:0/4004395926 shutdown_connections 2024-03-18T16:22:08.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.671+0000 7f391ec00640 1 -- 172.21.15.102:0/4004395926 wait complete. 2024-03-18T16:22:08.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.673+0000 7f391ec00640 1 Processor -- start 2024-03-18T16:22:08.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.674+0000 7f391ec00640 1 -- start start 2024-03-18T16:22:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.675+0000 7f391ec00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c62c0 0x7f39181f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.675+0000 7f391ec00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39180c66a0 0x7f3918146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.675+0000 7f391ec00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39180c7d50 0x7f3918146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.675+0000 7f391ec00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f39180c3ab0 con 0x7f39180c7d50 2024-03-18T16:22:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.675+0000 7f391ec00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3918070d80 con 0x7f39180c66a0 2024-03-18T16:22:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.675+0000 7f391ec00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f391814e3c0 con 0x7f39180c62c0 2024-03-18T16:22:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.675+0000 7f391d800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c62c0 0x7f39181f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.676+0000 7f391d800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c62c0 0x7f39181f3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43808/0 (socket says 172.21.15.102:43808) 2024-03-18T16:22:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.676+0000 7f391d800640 1 -- 172.21.15.102:0/3328706801 learned_addr learned my addr 172.21.15.102:0/3328706801 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:08.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.676+0000 7f391ce00640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39180c66a0 0x7f3918146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.681+0000 7f391d800640 1 -- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39180c66a0 msgr2=0x7f3918146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.681+0000 7f391d800640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39180c66a0 0x7f3918146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.681+0000 7f391d800640 1 -- 172.21.15.102:0/3328706801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39180c7d50 msgr2=0x7f3918146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.681+0000 7f391d800640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39180c7d50 0x7f3918146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:08.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.681+0000 7f391d800640 1 -- 172.21.15.102:0/3328706801 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39181f4380 con 0x7f39180c62c0 2024-03-18T16:22:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.683+0000 7f391d800640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c62c0 0x7f39181f3930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f390c00c410 tx=0x7f390c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.687+0000 7f3915600640 1 -- 172.21.15.102:0/3328706801 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c017070 con 0x7f39180c62c0 2024-03-18T16:22:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.687+0000 7f3915600640 1 -- 172.21.15.102:0/3328706801 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f390c0035f0 con 0x7f39180c62c0 2024-03-18T16:22:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.687+0000 7f3915600640 1 -- 172.21.15.102:0/3328706801 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c0039e0 con 0x7f39180c62c0 2024-03-18T16:22:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.687+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39180ccca0 con 0x7f39180c62c0 2024-03-18T16:22:08.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.687+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f39180c5cf0 con 0x7f39180c62c0 2024-03-18T16:22:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.689+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39180c6d30 con 0x7f39180c62c0 2024-03-18T16:22:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.708+0000 7f3915600640 1 -- 172.21.15.102:0/3328706801 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f390c01a070 con 0x7f39180c62c0 2024-03-18T16:22:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.710+0000 7f3915600640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38f806bc20 0x7f38f806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.710+0000 7f391ce00640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38f806bc20 0x7f38f806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.710+0000 7f3915600640 1 -- 172.21.15.102:0/3328706801 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f390c0567c0 con 0x7f39180c62c0 2024-03-18T16:22:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.710+0000 7f391ce00640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38f806bc20 0x7f38f806e0e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f39181e4740 tx=0x7f3904009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:08.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:08.963+0000 7f3915600640 1 -- 172.21.15.102:0/3328706801 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f39180c6d30 con 0x7f39180c62c0 2024-03-18T16:22:09.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.217+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39180c8a30 con 0x7f39180c62c0 2024-03-18T16:22:09.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.248+0000 7f3915600640 1 -- 172.21.15.102:0/3328706801 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6857 (secure 0 0 0) 0x7f390c021020 con 0x7f39180c62c0 2024-03-18T16:22:09.251 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:09.251 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4250,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6834/3423054813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3423054813},{"type":"v1","addr":"172.21.15.102:6834","nonce":3423054813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4253,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:22:07.416732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4270,"mds_1":4256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:creating","state_seq":1,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1}]} 2024-03-18T16:22:09.252 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 12 2024-03-18T16:22:09.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38f806bc20 msgr2=0x7f38f806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:09.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38f806bc20 0x7f38f806e0e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f39181e4740 tx=0x7f3904009060 comp rx=0 tx=0).stop 2024-03-18T16:22:09.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c62c0 msgr2=0x7f39181f3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:09.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c62c0 0x7f39181f3930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f390c00c410 tx=0x7f390c00c8d0 comp rx=0 tx=0).stop 2024-03-18T16:22:09.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 shutdown_connections 2024-03-18T16:22:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38f806bc20 0x7f38f806e0e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39180c7d50 0x7f3918146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39180c66a0 0x7f3918146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 --2- 172.21.15.102:0/3328706801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39180c62c0 0x7f39181f3930 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 >> 172.21.15.102:0/3328706801 conn(0x7f391806c310 msgr2=0x7f39181e6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 shutdown_connections 2024-03-18T16:22:09.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.252+0000 7f391ec00640 1 -- 172.21.15.102:0/3328706801 wait complete. 2024-03-18T16:22:09.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:22:09.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.959+0000 7fd012200640 1 Processor -- start 2024-03-18T16:22:10.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.960+0000 7fd012200640 1 -- start start 2024-03-18T16:22:10.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.961+0000 7fd012200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c141650 0x7fd00c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.961+0000 7fd012200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd00c15c3a0 0x7fd00c0c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.961+0000 7fd012200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd00c158ad0 con 0x7fd00c0c4fa0 2024-03-18T16:22:10.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.961+0000 7fd012200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd00c159370 con 0x7fd00c15c3a0 2024-03-18T16:22:10.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.961+0000 7fd012200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd00c158db0 con 0x7fd00c141650 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.962+0000 7fd010e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c141650 0x7fd00c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.962+0000 7fd010e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c141650 0x7fd00c158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43822/0 (socket says 172.21.15.102:43822) 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.962+0000 7fd010e00640 1 -- 172.21.15.102:0/1445792575 learned_addr learned my addr 172.21.15.102:0/1445792575 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.962+0000 7fd003e00640 1 --2- 172.21.15.102:0/1445792575 >> v2:172.21.15.102:3300/0 conn(0x7fd00c15c3a0 0x7fd00c0c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.964+0000 7fd010e00640 1 -- 172.21.15.102:0/1445792575 >> v2:172.21.15.102:3300/0 conn(0x7fd00c15c3a0 msgr2=0x7fd00c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.964+0000 7fd010e00640 1 --2- 172.21.15.102:0/1445792575 >> v2:172.21.15.102:3300/0 conn(0x7fd00c15c3a0 0x7fd00c0c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.964+0000 7fd010e00640 1 -- 172.21.15.102:0/1445792575 >> v1:172.21.15.102:6789/0 conn(0x7fd00c0c4fa0 legacy=0x7fd00c06f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.964+0000 7fd010e00640 1 -- 172.21.15.102:0/1445792575 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00c159090 con 0x7fd00c141650 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.966+0000 7fd010e00640 1 --2- 172.21.15.102:0/1445792575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c141650 0x7fd00c158420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009870 tx=0x7fcffc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=339fa3381cab8e70 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.968+0000 7fd00be00640 1 -- 172.21.15.102:0/1445792575 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc034070 con 0x7fd00c141650 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.970+0000 7fd00be00640 1 -- 172.21.15.102:0/1445792575 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcffc030840 con 0x7fd00c141650 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.970+0000 7fd00be00640 1 -- 172.21.15.102:0/1445792575 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc030c10 con 0x7fd00c141650 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.971+0000 7fd012200640 1 -- 172.21.15.102:0/1445792575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c141650 msgr2=0x7fd00c158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.971+0000 7fd012200640 1 --2- 172.21.15.102:0/1445792575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c141650 0x7fd00c158420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009870 tx=0x7fcffc02b980 comp rx=0 tx=0).stop 2024-03-18T16:22:10.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.971+0000 7fd012200640 1 -- 172.21.15.102:0/1445792575 shutdown_connections 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.971+0000 7fd012200640 1 --2- 172.21.15.102:0/1445792575 >> v2:172.21.15.102:3300/0 conn(0x7fd00c15c3a0 0x7fd00c0c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.971+0000 7fd012200640 1 --2- 172.21.15.102:0/1445792575 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c141650 0x7fd00c158420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.971+0000 7fd012200640 1 -- 172.21.15.102:0/1445792575 >> 172.21.15.102:0/1445792575 conn(0x7fd00c0cb890 msgr2=0x7fd00c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.971+0000 7fd012200640 1 -- 172.21.15.102:0/1445792575 shutdown_connections 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.971+0000 7fd012200640 1 -- 172.21.15.102:0/1445792575 wait complete. 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.973+0000 7fd012200640 1 Processor -- start 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.973+0000 7fd012200640 1 -- start start 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.974+0000 7fd012200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd00c0c4fa0 0x7fd00c159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.974+0000 7fd012200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c141650 0x7fd00c13dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.974+0000 7fd012200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c15c3a0 0x7fd00c157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.974+0000 7fd012200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c15c7c0 con 0x7fd00c141650 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.974+0000 7fd012200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c0c66a0 con 0x7fd00c0c4fa0 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.974+0000 7fd012200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd00c141a30 con 0x7fd00c15c3a0 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.974+0000 7fd003e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c141650 0x7fd00c13dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:10.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.974+0000 7fd011800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c15c3a0 0x7fd00c157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.974+0000 7fd010e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd00c0c4fa0 0x7fd00c159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.975+0000 7fd003e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c141650 0x7fd00c13dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57258/0 (socket says 172.21.15.102:57258) 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.975+0000 7fd003e00640 1 -- 172.21.15.102:0/2418027742 learned_addr learned my addr 172.21.15.102:0/2418027742 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.978+0000 7fd003e00640 1 -- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c15c3a0 msgr2=0x7fd00c157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.978+0000 7fd003e00640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c15c3a0 0x7fd00c157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.978+0000 7fd003e00640 1 -- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd00c0c4fa0 msgr2=0x7fd00c159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.978+0000 7fd003e00640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd00c0c4fa0 0x7fd00c159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.978+0000 7fd003e00640 1 -- 172.21.15.102:0/2418027742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00c1f1500 con 0x7fd00c141650 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.981+0000 7fd003e00640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c141650 0x7fd00c13dba0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e870 tx=0x7fcff400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.985+0000 7fd009600640 1 -- 172.21.15.102:0/2418027742 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4017070 con 0x7fd00c141650 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.985+0000 7fd009600640 1 -- 172.21.15.102:0/2418027742 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff400c940 con 0x7fd00c141650 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.985+0000 7fd009600640 1 -- 172.21.15.102:0/2418027742 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff400cd30 con 0x7fd00c141650 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.985+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00c1f3300 con 0x7fd00c141650 2024-03-18T16:22:10.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.985+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00c15a760 con 0x7fd00c141650 2024-03-18T16:22:10.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:09.986+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd00c074fd0 con 0x7fd00c141650 2024-03-18T16:22:10.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.010+0000 7fd009600640 1 -- 172.21.15.102:0/2418027742 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcff401a070 con 0x7fd00c141650 2024-03-18T16:22:10.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.011+0000 7fd009600640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcfe806bc20 0x7fcfe806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.012+0000 7fd010e00640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcfe806bc20 0x7fcfe806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:10.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.013+0000 7fd010e00640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcfe806bc20 0x7fcfe806e0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcffc036000 tx=0x7fcffc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:10.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.323+0000 7fd009600640 1 -- 172.21.15.102:0/2418027742 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fcff4057590 con 0x7fd00c141650 2024-03-18T16:22:10.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.324+0000 7fd009600640 1 -- 172.21.15.102:0/2418027742 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcff40594f0 con 0x7fd00c141650 2024-03-18T16:22:10.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.534+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd00c0b7f00 con 0x7fd00c141650 2024-03-18T16:22:10.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.608+0000 7fd009600640 1 -- 172.21.15.102:0/2418027742 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18907 (secure 0 0 0) 0x7fcff405d590 con 0x7fd00c141650 2024-03-18T16:22:10.611 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:10.611 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":29,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:54.749120+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcfe806bc20 msgr2=0x7fcfe806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcfe806bc20 0x7fcfe806e0e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcffc036000 tx=0x7fcffc008030 comp rx=0 tx=0).stop 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c141650 msgr2=0x7fd00c13dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c141650 0x7fd00c13dba0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e870 tx=0x7fcff400ed30 comp rx=0 tx=0).stop 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 shutdown_connections 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcfe806bc20 0x7fcfe806e0e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd00c15c3a0 0x7fd00c157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd00c141650 0x7fd00c13dba0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 --2- 172.21.15.102:0/2418027742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd00c0c4fa0 0x7fd00c159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 >> 172.21.15.102:0/2418027742 conn(0x7fd00c0cb890 msgr2=0x7fd00c072b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:10.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 shutdown_connections 2024-03-18T16:22:10.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.611+0000 7fd012200640 1 -- 172.21.15.102:0/2418027742 wait complete. 2024-03-18T16:22:10.634 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T16:22:10.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:22:10.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.852+0000 7f2def600640 1 Processor -- start 2024-03-18T16:22:10.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.852+0000 7f2def600640 1 -- start start 2024-03-18T16:22:10.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.853+0000 7f2def600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de8154670 0x7f2de8070db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.853+0000 7f2def600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2de80c3b00 0x7f2de815e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.853+0000 7f2def600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2de80c77e0 0x7f2de80ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.853+0000 7f2def600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2de8071460 con 0x7f2de80c77e0 2024-03-18T16:22:10.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.853+0000 7f2def600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2de815dd30 con 0x7f2de80c3b00 2024-03-18T16:22:10.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.853+0000 7f2def600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2de815da00 con 0x7f2de8154670 2024-03-18T16:22:10.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.854+0000 7f2ded600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2de80c77e0 0x7f2de80ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:10.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.855+0000 7f2ded600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2de80c77e0 0x7f2de80ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51664/0 (socket says 172.21.15.102:51664) 2024-03-18T16:22:10.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.855+0000 7f2ded600640 1 -- 172.21.15.102:0/2849036678 learned_addr learned my addr 172.21.15.102:0/2849036678 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.858+0000 7f2ded600640 1 -- 172.21.15.102:0/2849036678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de8154670 msgr2=0x7f2de8070db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:10.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.858+0000 7f2ded600640 1 --2- 172.21.15.102:0/2849036678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de8154670 0x7f2de8070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.858+0000 7f2ded600640 1 -- 172.21.15.102:0/2849036678 >> v2:172.21.15.154:3300/0 conn(0x7f2de80c3b00 msgr2=0x7f2de815e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:10.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.858+0000 7f2ded600640 1 --2- 172.21.15.102:0/2849036678 >> v2:172.21.15.154:3300/0 conn(0x7f2de80c3b00 0x7f2de815e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.858+0000 7f2ded600640 1 -- 172.21.15.102:0/2849036678 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de806bac0 con 0x7f2de80c77e0 2024-03-18T16:22:10.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.860+0000 7f2ded600640 1 --2- 172.21.15.102:0/2849036678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c77e0 0x7f2de80ccb10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2de8073920 tx=0x7f2de002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1078c73c9115c6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.864+0000 7f2de7400640 1 -- 172.21.15.102:0/2849036678 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de002ca70 con 0x7f2de80c77e0 2024-03-18T16:22:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.864+0000 7f2de7400640 1 -- 172.21.15.102:0/2849036678 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2de002cc10 con 0x7f2de80c77e0 2024-03-18T16:22:10.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.864+0000 7f2de7400640 1 -- 172.21.15.102:0/2849036678 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de00343f0 con 0x7f2de80c77e0 2024-03-18T16:22:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.865+0000 7f2def600640 1 -- 172.21.15.102:0/2849036678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c77e0 msgr2=0x7f2de80ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.865+0000 7f2def600640 1 --2- 172.21.15.102:0/2849036678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c77e0 0x7f2de80ccb10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2de8073920 tx=0x7f2de002bed0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.865+0000 7f2def600640 1 -- 172.21.15.102:0/2849036678 shutdown_connections 2024-03-18T16:22:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.865+0000 7f2def600640 1 --2- 172.21.15.102:0/2849036678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c77e0 0x7f2de80ccb10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.865+0000 7f2def600640 1 --2- 172.21.15.102:0/2849036678 >> v2:172.21.15.154:3300/0 conn(0x7f2de80c3b00 0x7f2de815e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.865+0000 7f2def600640 1 --2- 172.21.15.102:0/2849036678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de8154670 0x7f2de8070db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.865+0000 7f2def600640 1 -- 172.21.15.102:0/2849036678 >> 172.21.15.102:0/2849036678 conn(0x7f2de806ed50 msgr2=0x7f2de806f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:10.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.865+0000 7f2def600640 1 -- 172.21.15.102:0/2849036678 shutdown_connections 2024-03-18T16:22:10.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.865+0000 7f2def600640 1 -- 172.21.15.102:0/2849036678 wait complete. 2024-03-18T16:22:10.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.867+0000 7f2def600640 1 Processor -- start 2024-03-18T16:22:10.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.867+0000 7f2def600640 1 -- start start 2024-03-18T16:22:10.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.868+0000 7f2def600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c3b00 0x7f2de81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.868+0000 7f2def600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de80c77e0 0x7f2de814d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.868+0000 7f2def600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de8154670 0x7f2de81f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.868+0000 7f2def600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2de8154a50 con 0x7f2de80c3b00 2024-03-18T16:22:10.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.868+0000 7f2def600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2de806c080 con 0x7f2de8154670 2024-03-18T16:22:10.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.868+0000 7f2def600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2de815e700 con 0x7f2de80c77e0 2024-03-18T16:22:10.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.869+0000 7f2decc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c3b00 0x7f2de81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:10.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.869+0000 7f2decc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c3b00 0x7f2de81529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51670/0 (socket says 172.21.15.102:51670) 2024-03-18T16:22:10.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.869+0000 7f2decc00640 1 -- 172.21.15.102:0/1784123120 learned_addr learned my addr 172.21.15.102:0/1784123120 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.873+0000 7f2decc00640 1 -- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de80c77e0 msgr2=0x7f2de814d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.873+0000 7f2decc00640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de80c77e0 0x7f2de814d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.873+0000 7f2decc00640 1 -- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de8154670 msgr2=0x7f2de81f01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.873+0000 7f2decc00640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de8154670 0x7f2de81f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:10.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.873+0000 7f2decc00640 1 -- 172.21.15.102:0/1784123120 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de80cb400 con 0x7f2de80c3b00 2024-03-18T16:22:10.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.875+0000 7f2decc00640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c3b00 0x7f2de81529c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00b4d0 tx=0x7f2ddc00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.879+0000 7f2de4c00640 1 -- 172.21.15.102:0/1784123120 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc015020 con 0x7f2de80c3b00 2024-03-18T16:22:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.879+0000 7f2de4c00640 1 -- 172.21.15.102:0/1784123120 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ddc011890 con 0x7f2de80c3b00 2024-03-18T16:22:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.879+0000 7f2de4c00640 1 -- 172.21.15.102:0/1784123120 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc011c80 con 0x7f2de80c3b00 2024-03-18T16:22:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.879+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de81f0de0 con 0x7f2de80c3b00 2024-03-18T16:22:10.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.879+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de80cb980 con 0x7f2de80c3b00 2024-03-18T16:22:10.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.881+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de8074fd0 con 0x7f2de80c3b00 2024-03-18T16:22:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.903+0000 7f2de4c00640 1 -- 172.21.15.102:0/1784123120 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2ddc018020 con 0x7f2de80c3b00 2024-03-18T16:22:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.904+0000 7f2de4c00640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc806bc20 0x7f2dc806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:10.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.904+0000 7f2de7e00640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc806bc20 0x7f2dc806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:10.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.905+0000 7f2de7e00640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc806bc20 0x7f2dc806e0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2de80b2b80 tx=0x7f2dd4006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:10.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:10.907+0000 7f2de4c00640 1 -- 172.21.15.102:0/1784123120 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f2ddc057690 con 0x7f2de80c3b00 2024-03-18T16:22:11.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.201+0000 7f2de4c00640 1 -- 172.21.15.102:0/1784123120 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2ddc01f020 con 0x7f2de80c3b00 2024-03-18T16:22:11.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.399+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2de8070510 con 0x7f2dc806bc20 2024-03-18T16:22:11.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.408+0000 7f2de4c00640 1 -- 172.21.15.102:0/1784123120 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332709 (secure 0 0 0) 0x7f2de8070510 con 0x7f2dc806bc20 2024-03-18T16:22:11.417 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:11.419 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:22:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc806bc20 msgr2=0x7f2dc806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc806bc20 0x7f2dc806e0e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2de80b2b80 tx=0x7f2dd4006bf0 comp rx=0 tx=0).stop 2024-03-18T16:22:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c3b00 msgr2=0x7f2de81529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c3b00 0x7f2de81529c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc00b4d0 tx=0x7f2ddc00b990 comp rx=0 tx=0).stop 2024-03-18T16:22:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 shutdown_connections 2024-03-18T16:22:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc806bc20 0x7f2dc806e0e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de8154670 0x7f2de81f01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de80c77e0 0x7f2de814d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 --2- 172.21.15.102:0/1784123120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de80c3b00 0x7f2de81529c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 >> 172.21.15.102:0/1784123120 conn(0x7f2de806ed50 msgr2=0x7f2de815b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:11.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 shutdown_connections 2024-03-18T16:22:11.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.415+0000 7f2def600640 1 -- 172.21.15.102:0/1784123120 wait complete. 2024-03-18T16:22:11.437 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-03-18T16:22:10.823759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218768,"kb_used_data":2640,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755952,"statfs":{"total":773094113280,"available":772870094848,"internally_reserved":0,"allocated":2703360,"data_stored":2228844,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002911"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278633+0000","last_change":"2024-03-18T16:21:21.402212+0000","last_active":"2024-03-18T16:21:55.278633+0000","last_peered":"2024-03-18T16:21:55.278633+0000","last_clean":"2024-03-18T16:21:55.278633+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:21:55.278633+0000","last_undegraded":"2024-03-18T16:21:55.278633+0000","last_fullsized":"2024-03-18T16:21:55.278633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:33.215915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841700+0000","last_change":"2024-03-18T16:21:23.045605+0000","last_active":"2024-03-18T16:21:54.841700+0000","last_peered":"2024-03-18T16:21:54.841700+0000","last_clean":"2024-03-18T16:21:54.841700+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:21:54.841700+0000","last_undegraded":"2024-03-18T16:21:54.841700+0000","last_fullsized":"2024-03-18T16:21:54.841700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.839646+0000","last_change":"2024-03-18T16:21:29.146958+0000","last_active":"2024-03-18T16:21:54.839646+0000","last_peered":"2024-03-18T16:21:54.839646+0000","last_clean":"2024-03-18T16:21:54.839646+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:21:54.839646+0000","last_undegraded":"2024-03-18T16:21:54.839646+0000","last_fullsized":"2024-03-18T16:21:54.839646+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:55.871927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842531+0000","last_change":"2024-03-18T16:21:21.413060+0000","last_active":"2024-03-18T16:21:54.842531+0000","last_peered":"2024-03-18T16:21:54.842531+0000","last_clean":"2024-03-18T16:21:54.842531+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:21:54.842531+0000","last_undegraded":"2024-03-18T16:21:54.842531+0000","last_fullsized":"2024-03-18T16:21:54.842531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:40.114937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278167+0000","last_change":"2024-03-18T16:21:23.050849+0000","last_active":"2024-03-18T16:21:55.278167+0000","last_peered":"2024-03-18T16:21:55.278167+0000","last_clean":"2024-03-18T16:21:55.278167+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:21:55.278167+0000","last_undegraded":"2024-03-18T16:21:55.278167+0000","last_fullsized":"2024-03-18T16:21:55.278167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147644+0000","last_change":"2024-03-18T16:21:29.065755+0000","last_active":"2024-03-18T16:21:55.147644+0000","last_peered":"2024-03-18T16:21:55.147644+0000","last_clean":"2024-03-18T16:21:55.147644+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:21:55.147644+0000","last_undegraded":"2024-03-18T16:21:55.147644+0000","last_fullsized":"2024-03-18T16:21:55.147644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:45.934586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266332+0000","last_change":"2024-03-18T16:21:21.398188+0000","last_active":"2024-03-18T16:21:55.266332+0000","last_peered":"2024-03-18T16:21:55.266332+0000","last_clean":"2024-03-18T16:21:55.266332+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:21:55.266332+0000","last_undegraded":"2024-03-18T16:21:55.266332+0000","last_fullsized":"2024-03-18T16:21:55.266332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064521+0000","last_change":"2024-03-18T16:21:22.946288+0000","last_active":"2024-03-18T16:21:55.064521+0000","last_peered":"2024-03-18T16:21:55.064521+0000","last_clean":"2024-03-18T16:21:55.064521+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:21:55.064521+0000","last_undegraded":"2024-03-18T16:21:55.064521+0000","last_fullsized":"2024-03-18T16:21:55.064521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:08.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147915+0000","last_change":"2024-03-18T16:21:29.065891+0000","last_active":"2024-03-18T16:21:55.147915+0000","last_peered":"2024-03-18T16:21:55.147915+0000","last_clean":"2024-03-18T16:21:55.147915+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:21:55.147915+0000","last_undegraded":"2024-03-18T16:21:55.147915+0000","last_fullsized":"2024-03-18T16:21:55.147915+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265954+0000","last_change":"2024-03-18T16:21:21.397071+0000","last_active":"2024-03-18T16:21:55.265954+0000","last_peered":"2024-03-18T16:21:55.265954+0000","last_clean":"2024-03-18T16:21:55.265954+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:21:55.265954+0000","last_undegraded":"2024-03-18T16:21:55.265954+0000","last_fullsized":"2024-03-18T16:21:55.265954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065035+0000","last_change":"2024-03-18T16:21:22.946468+0000","last_active":"2024-03-18T16:21:55.065035+0000","last_peered":"2024-03-18T16:21:55.065035+0000","last_clean":"2024-03-18T16:21:55.065035+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:21:55.065035+0000","last_undegraded":"2024-03-18T16:21:55.065035+0000","last_fullsized":"2024-03-18T16:21:55.065035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147794+0000","last_change":"2024-03-18T16:21:29.073724+0000","last_active":"2024-03-18T16:21:55.147794+0000","last_peered":"2024-03-18T16:21:55.147794+0000","last_clean":"2024-03-18T16:21:55.147794+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:21:55.147794+0000","last_undegraded":"2024-03-18T16:21:55.147794+0000","last_fullsized":"2024-03-18T16:21:55.147794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273576+0000","last_change":"2024-03-18T16:21:23.041926+0000","last_active":"2024-03-18T16:21:55.273576+0000","last_peered":"2024-03-18T16:21:55.273576+0000","last_clean":"2024-03-18T16:21:55.273576+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:21:55.273576+0000","last_undegraded":"2024-03-18T16:21:55.273576+0000","last_fullsized":"2024-03-18T16:21:55.273576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:03.532253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.821045+0000","last_change":"2024-03-18T16:21:21.394801+0000","last_active":"2024-03-18T16:22:05.821045+0000","last_peered":"2024-03-18T16:22:05.821045+0000","last_clean":"2024-03-18T16:22:05.821045+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:22:05.821045+0000","last_undegraded":"2024-03-18T16:22:05.821045+0000","last_fullsized":"2024-03-18T16:22:05.821045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148181+0000","last_change":"2024-03-18T16:21:29.073757+0000","last_active":"2024-03-18T16:21:55.148181+0000","last_peered":"2024-03-18T16:21:55.148181+0000","last_clean":"2024-03-18T16:21:55.148181+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:21:55.148181+0000","last_undegraded":"2024-03-18T16:21:55.148181+0000","last_fullsized":"2024-03-18T16:21:55.148181+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841346+0000","last_change":"2024-03-18T16:21:23.060776+0000","last_active":"2024-03-18T16:21:54.841346+0000","last_peered":"2024-03-18T16:21:54.841346+0000","last_clean":"2024-03-18T16:21:54.841346+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:21:54.841346+0000","last_undegraded":"2024-03-18T16:21:54.841346+0000","last_fullsized":"2024-03-18T16:21:54.841346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831728+0000","last_change":"2024-03-18T16:21:21.412733+0000","last_active":"2024-03-18T16:21:54.831728+0000","last_peered":"2024-03-18T16:21:54.831728+0000","last_clean":"2024-03-18T16:21:54.831728+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:21:54.831728+0000","last_undegraded":"2024-03-18T16:21:54.831728+0000","last_fullsized":"2024-03-18T16:21:54.831728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148007+0000","last_change":"2024-03-18T16:21:29.063980+0000","last_active":"2024-03-18T16:21:55.148007+0000","last_peered":"2024-03-18T16:21:55.148007+0000","last_clean":"2024-03-18T16:21:55.148007+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:21:55.148007+0000","last_undegraded":"2024-03-18T16:21:55.148007+0000","last_fullsized":"2024-03-18T16:21:55.148007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:39.469490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.845055+0000","last_change":"2024-03-18T16:21:54.838108+0000","last_active":"2024-03-18T16:21:54.845055+0000","last_peered":"2024-03-18T16:21:54.845055+0000","last_clean":"2024-03-18T16:21:54.845055+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:21:54.845055+0000","last_undegraded":"2024-03-18T16:21:54.845055+0000","last_fullsized":"2024-03-18T16:21:54.845055+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:40.182971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.826842+0000","last_change":"2024-03-18T16:21:21.407132+0000","last_active":"2024-03-18T16:22:05.826842+0000","last_peered":"2024-03-18T16:22:05.826842+0000","last_clean":"2024-03-18T16:22:05.826842+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:22:05.826842+0000","last_undegraded":"2024-03-18T16:22:05.826842+0000","last_fullsized":"2024-03-18T16:22:05.826842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.272088+0000","last_change":"2024-03-18T16:21:55.269661+0000","last_active":"2024-03-18T16:21:55.272088+0000","last_peered":"2024-03-18T16:21:55.272088+0000","last_clean":"2024-03-18T16:21:55.272088+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:21:55.272088+0000","last_undegraded":"2024-03-18T16:21:55.272088+0000","last_fullsized":"2024-03-18T16:21:55.272088+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:38.290937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148142+0000","last_change":"2024-03-18T16:21:22.840551+0000","last_active":"2024-03-18T16:21:55.148142+0000","last_peered":"2024-03-18T16:21:55.148142+0000","last_clean":"2024-03-18T16:21:55.148142+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:21:55.148142+0000","last_undegraded":"2024-03-18T16:21:55.148142+0000","last_fullsized":"2024-03-18T16:21:55.148142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:17.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.822682+0000","last_change":"2024-03-18T16:21:21.400349+0000","last_active":"2024-03-18T16:22:05.822682+0000","last_peered":"2024-03-18T16:22:05.822682+0000","last_clean":"2024-03-18T16:22:05.822682+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:22:05.822682+0000","last_undegraded":"2024-03-18T16:22:05.822682+0000","last_fullsized":"2024-03-18T16:22:05.822682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148262+0000","last_change":"2024-03-18T16:21:29.064624+0000","last_active":"2024-03-18T16:21:55.148262+0000","last_peered":"2024-03-18T16:21:55.148262+0000","last_clean":"2024-03-18T16:21:55.148262+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:21:55.148262+0000","last_undegraded":"2024-03-18T16:21:55.148262+0000","last_fullsized":"2024-03-18T16:21:55.148262+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278343+0000","last_change":"2024-03-18T16:21:23.055982+0000","last_active":"2024-03-18T16:21:55.278343+0000","last_peered":"2024-03-18T16:21:55.278343+0000","last_clean":"2024-03-18T16:21:55.278343+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:21:55.278343+0000","last_undegraded":"2024-03-18T16:21:55.278343+0000","last_fullsized":"2024-03-18T16:21:55.278343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.038917+0000","last_change":"2024-03-18T16:21:21.400843+0000","last_active":"2024-03-18T16:22:06.038917+0000","last_peered":"2024-03-18T16:22:06.038917+0000","last_clean":"2024-03-18T16:22:06.038917+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:22:06.038917+0000","last_undegraded":"2024-03-18T16:22:06.038917+0000","last_fullsized":"2024-03-18T16:22:06.038917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147884+0000","last_change":"2024-03-18T16:21:29.140888+0000","last_active":"2024-03-18T16:21:55.147884+0000","last_peered":"2024-03-18T16:21:55.147884+0000","last_clean":"2024-03-18T16:21:55.147884+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:21:55.147884+0000","last_undegraded":"2024-03-18T16:21:55.147884+0000","last_fullsized":"2024-03-18T16:21:55.147884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842117+0000","last_change":"2024-03-18T16:21:22.994455+0000","last_active":"2024-03-18T16:21:54.842117+0000","last_peered":"2024-03-18T16:21:54.842117+0000","last_clean":"2024-03-18T16:21:54.842117+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:21:54.842117+0000","last_undegraded":"2024-03-18T16:21:54.842117+0000","last_fullsized":"2024-03-18T16:21:54.842117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834764+0000","last_change":"2024-03-18T16:21:21.402920+0000","last_active":"2024-03-18T16:21:54.834764+0000","last_peered":"2024-03-18T16:21:54.834764+0000","last_clean":"2024-03-18T16:21:54.834764+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:21:54.834764+0000","last_undegraded":"2024-03-18T16:21:54.834764+0000","last_fullsized":"2024-03-18T16:21:54.834764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.838803+0000","last_change":"2024-03-18T16:21:29.145235+0000","last_active":"2024-03-18T16:21:54.838803+0000","last_peered":"2024-03-18T16:21:54.838803+0000","last_clean":"2024-03-18T16:21:54.838803+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:21:54.838803+0000","last_undegraded":"2024-03-18T16:21:54.838803+0000","last_fullsized":"2024-03-18T16:21:54.838803+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:42.379951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.833875+0000","last_change":"2024-03-18T16:21:22.975172+0000","last_active":"2024-03-18T16:21:54.833875+0000","last_peered":"2024-03-18T16:21:54.833875+0000","last_clean":"2024-03-18T16:21:54.833875+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:21:54.833875+0000","last_undegraded":"2024-03-18T16:21:54.833875+0000","last_fullsized":"2024-03-18T16:21:54.833875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266381+0000","last_change":"2024-03-18T16:21:21.399688+0000","last_active":"2024-03-18T16:21:55.266381+0000","last_peered":"2024-03-18T16:21:55.266381+0000","last_clean":"2024-03-18T16:21:55.266381+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:21:55.266381+0000","last_undegraded":"2024-03-18T16:21:55.266381+0000","last_fullsized":"2024-03-18T16:21:55.266381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:11.933810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277363+0000","last_change":"2024-03-18T16:21:29.141555+0000","last_active":"2024-03-18T16:21:55.277363+0000","last_peered":"2024-03-18T16:21:55.277363+0000","last_clean":"2024-03-18T16:21:55.277363+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:21:55.277363+0000","last_undegraded":"2024-03-18T16:21:55.277363+0000","last_fullsized":"2024-03-18T16:21:55.277363+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065093+0000","last_change":"2024-03-18T16:21:22.947980+0000","last_active":"2024-03-18T16:21:55.065093+0000","last_peered":"2024-03-18T16:21:55.065093+0000","last_clean":"2024-03-18T16:21:55.065093+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:21:55.065093+0000","last_undegraded":"2024-03-18T16:21:55.065093+0000","last_fullsized":"2024-03-18T16:21:55.065093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:31.739693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834138+0000","last_change":"2024-03-18T16:21:21.400087+0000","last_active":"2024-03-18T16:21:54.834138+0000","last_peered":"2024-03-18T16:21:54.834138+0000","last_clean":"2024-03-18T16:21:54.834138+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:21:54.834138+0000","last_undegraded":"2024-03-18T16:21:54.834138+0000","last_fullsized":"2024-03-18T16:21:54.834138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.830782+0000","last_change":"2024-03-18T16:21:29.076629+0000","last_active":"2024-03-18T16:21:54.830782+0000","last_peered":"2024-03-18T16:21:54.830782+0000","last_clean":"2024-03-18T16:21:54.830782+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:21:54.830782+0000","last_undegraded":"2024-03-18T16:21:54.830782+0000","last_fullsized":"2024-03-18T16:21:54.830782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148132+0000","last_change":"2024-03-18T16:21:23.055810+0000","last_active":"2024-03-18T16:21:55.148132+0000","last_peered":"2024-03-18T16:21:55.148132+0000","last_clean":"2024-03-18T16:21:55.148132+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:21:55.148132+0000","last_undegraded":"2024-03-18T16:21:55.148132+0000","last_fullsized":"2024-03-18T16:21:55.148132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:40.451255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064938+0000","last_change":"2024-03-18T16:21:21.399134+0000","last_active":"2024-03-18T16:21:55.064938+0000","last_peered":"2024-03-18T16:21:55.064938+0000","last_clean":"2024-03-18T16:21:55.064938+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:21:55.064938+0000","last_undegraded":"2024-03-18T16:21:55.064938+0000","last_fullsized":"2024-03-18T16:21:55.064938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834980+0000","last_change":"2024-03-18T16:21:29.067567+0000","last_active":"2024-03-18T16:21:54.834980+0000","last_peered":"2024-03-18T16:21:54.834980+0000","last_clean":"2024-03-18T16:21:54.834980+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:21:54.834980+0000","last_undegraded":"2024-03-18T16:21:54.834980+0000","last_fullsized":"2024-03-18T16:21:54.834980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.832134+0000","last_change":"2024-03-18T16:21:22.977089+0000","last_active":"2024-03-18T16:21:54.832134+0000","last_peered":"2024-03-18T16:21:54.832134+0000","last_clean":"2024-03-18T16:21:54.832134+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:21:54.832134+0000","last_undegraded":"2024-03-18T16:21:54.832134+0000","last_fullsized":"2024-03-18T16:21:54.832134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827052+0000","last_change":"2024-03-18T16:21:21.408450+0000","last_active":"2024-03-18T16:21:54.827052+0000","last_peered":"2024-03-18T16:21:54.827052+0000","last_clean":"2024-03-18T16:21:54.827052+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:21:54.827052+0000","last_undegraded":"2024-03-18T16:21:54.827052+0000","last_fullsized":"2024-03-18T16:21:54.827052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.835511+0000","last_change":"2024-03-18T16:21:29.076875+0000","last_active":"2024-03-18T16:21:54.835511+0000","last_peered":"2024-03-18T16:21:54.835511+0000","last_clean":"2024-03-18T16:21:54.835511+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:21:54.835511+0000","last_undegraded":"2024-03-18T16:21:54.835511+0000","last_fullsized":"2024-03-18T16:21:54.835511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277013+0000","last_change":"2024-03-18T16:21:23.054497+0000","last_active":"2024-03-18T16:21:55.277013+0000","last_peered":"2024-03-18T16:21:55.277013+0000","last_clean":"2024-03-18T16:21:55.277013+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:21:55.277013+0000","last_undegraded":"2024-03-18T16:21:55.277013+0000","last_fullsized":"2024-03-18T16:21:55.277013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:37.455180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265682+0000","last_change":"2024-03-18T16:21:21.404754+0000","last_active":"2024-03-18T16:21:55.265682+0000","last_peered":"2024-03-18T16:21:55.265682+0000","last_clean":"2024-03-18T16:21:55.265682+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:21:55.265682+0000","last_undegraded":"2024-03-18T16:21:55.265682+0000","last_fullsized":"2024-03-18T16:21:55.265682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278071+0000","last_change":"2024-03-18T16:21:29.147316+0000","last_active":"2024-03-18T16:21:55.278071+0000","last_peered":"2024-03-18T16:21:55.278071+0000","last_clean":"2024-03-18T16:21:55.278071+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:21:55.278071+0000","last_undegraded":"2024-03-18T16:21:55.278071+0000","last_fullsized":"2024-03-18T16:21:55.278071+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842010+0000","last_change":"2024-03-18T16:21:22.947635+0000","last_active":"2024-03-18T16:21:54.842010+0000","last_peered":"2024-03-18T16:21:54.842010+0000","last_clean":"2024-03-18T16:21:54.842010+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:21:54.842010+0000","last_undegraded":"2024-03-18T16:21:54.842010+0000","last_fullsized":"2024-03-18T16:21:54.842010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.058595+0000","last_change":"2024-03-18T16:21:21.381306+0000","last_active":"2024-03-18T16:22:06.058595+0000","last_peered":"2024-03-18T16:22:06.058595+0000","last_clean":"2024-03-18T16:22:06.058595+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:22:06.058595+0000","last_undegraded":"2024-03-18T16:22:06.058595+0000","last_fullsized":"2024-03-18T16:22:06.058595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":186,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265740+0000","last_change":"2024-03-18T16:21:08.662767+0000","last_active":"2024-03-18T16:21:55.265740+0000","last_peered":"2024-03-18T16:21:55.265740+0000","last_clean":"2024-03-18T16:21:55.265740+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:21:55.265740+0000","last_undegraded":"2024-03-18T16:21:55.265740+0000","last_fullsized":"2024-03-18T16:21:55.265740+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_clean_scrub_stamp":"2024-03-18T16:21:05.847994+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:12:10.494707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273700+0000","last_change":"2024-03-18T16:21:29.145697+0000","last_active":"2024-03-18T16:21:55.273700+0000","last_peered":"2024-03-18T16:21:55.273700+0000","last_clean":"2024-03-18T16:21:55.273700+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:21:55.273700+0000","last_undegraded":"2024-03-18T16:21:55.273700+0000","last_fullsized":"2024-03-18T16:21:55.273700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:46.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266203+0000","last_change":"2024-03-18T16:21:22.995291+0000","last_active":"2024-03-18T16:21:55.266203+0000","last_peered":"2024-03-18T16:21:55.266203+0000","last_clean":"2024-03-18T16:21:55.266203+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:21:55.266203+0000","last_undegraded":"2024-03-18T16:21:55.266203+0000","last_fullsized":"2024-03-18T16:21:55.266203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:32.918246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.828217+0000","last_change":"2024-03-18T16:21:21.382137+0000","last_active":"2024-03-18T16:21:54.828217+0000","last_peered":"2024-03-18T16:21:54.828217+0000","last_clean":"2024-03-18T16:21:54.828217+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:21:54.828217+0000","last_undegraded":"2024-03-18T16:21:54.828217+0000","last_fullsized":"2024-03-18T16:21:54.828217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273492+0000","last_change":"2024-03-18T16:21:29.139432+0000","last_active":"2024-03-18T16:21:55.273492+0000","last_peered":"2024-03-18T16:21:55.273492+0000","last_clean":"2024-03-18T16:21:55.273492+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:21:55.273492+0000","last_undegraded":"2024-03-18T16:21:55.273492+0000","last_fullsized":"2024-03-18T16:21:55.273492+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:14.828125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064603+0000","last_change":"2024-03-18T16:21:22.949578+0000","last_active":"2024-03-18T16:21:55.064603+0000","last_peered":"2024-03-18T16:21:55.064603+0000","last_clean":"2024-03-18T16:21:55.064603+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:21:55.064603+0000","last_undegraded":"2024-03-18T16:21:55.064603+0000","last_fullsized":"2024-03-18T16:21:55.064603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:36:20.997720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.835156+0000","last_change":"2024-03-18T16:21:21.400859+0000","last_active":"2024-03-18T16:21:54.835156+0000","last_peered":"2024-03-18T16:21:54.835156+0000","last_clean":"2024-03-18T16:21:54.835156+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:21:54.835156+0000","last_undegraded":"2024-03-18T16:21:54.835156+0000","last_fullsized":"2024-03-18T16:21:54.835156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277608+0000","last_change":"2024-03-18T16:21:29.076813+0000","last_active":"2024-03-18T16:21:55.277608+0000","last_peered":"2024-03-18T16:21:55.277608+0000","last_clean":"2024-03-18T16:21:55.277608+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:21:55.277608+0000","last_undegraded":"2024-03-18T16:21:55.277608+0000","last_fullsized":"2024-03-18T16:21:55.277608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266390+0000","last_change":"2024-03-18T16:21:23.000009+0000","last_active":"2024-03-18T16:21:55.266390+0000","last_peered":"2024-03-18T16:21:55.266390+0000","last_clean":"2024-03-18T16:21:55.266390+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:21:55.266390+0000","last_undegraded":"2024-03-18T16:21:55.266390+0000","last_fullsized":"2024-03-18T16:21:55.266390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:39.383342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.828352+0000","last_change":"2024-03-18T16:21:21.409271+0000","last_active":"2024-03-18T16:21:54.828352+0000","last_peered":"2024-03-18T16:21:54.828352+0000","last_clean":"2024-03-18T16:21:54.828352+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:21:54.828352+0000","last_undegraded":"2024-03-18T16:21:54.828352+0000","last_fullsized":"2024-03-18T16:21:54.828352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064256+0000","last_change":"2024-03-18T16:21:29.139595+0000","last_active":"2024-03-18T16:21:55.064256+0000","last_peered":"2024-03-18T16:21:55.064256+0000","last_clean":"2024-03-18T16:21:55.064256+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:21:55.064256+0000","last_undegraded":"2024-03-18T16:21:55.064256+0000","last_fullsized":"2024-03-18T16:21:55.064256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:49.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266190+0000","last_change":"2024-03-18T16:21:23.000054+0000","last_active":"2024-03-18T16:21:55.266190+0000","last_peered":"2024-03-18T16:21:55.266190+0000","last_clean":"2024-03-18T16:21:55.266190+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:21:55.266190+0000","last_undegraded":"2024-03-18T16:21:55.266190+0000","last_fullsized":"2024-03-18T16:21:55.266190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:34:41.535042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148484+0000","last_change":"2024-03-18T16:21:21.399315+0000","last_active":"2024-03-18T16:21:55.148484+0000","last_peered":"2024-03-18T16:21:55.148484+0000","last_clean":"2024-03-18T16:21:55.148484+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:21:55.148484+0000","last_undegraded":"2024-03-18T16:21:55.148484+0000","last_fullsized":"2024-03-18T16:21:55.148484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:29.543106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147629+0000","last_change":"2024-03-18T16:21:29.065243+0000","last_active":"2024-03-18T16:21:55.147629+0000","last_peered":"2024-03-18T16:21:55.147629+0000","last_clean":"2024-03-18T16:21:55.147629+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:21:55.147629+0000","last_undegraded":"2024-03-18T16:21:55.147629+0000","last_fullsized":"2024-03-18T16:21:55.147629+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265799+0000","last_change":"2024-03-18T16:21:22.996080+0000","last_active":"2024-03-18T16:21:55.265799+0000","last_peered":"2024-03-18T16:21:55.265799+0000","last_clean":"2024-03-18T16:21:55.265799+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:21:55.265799+0000","last_undegraded":"2024-03-18T16:21:55.265799+0000","last_fullsized":"2024-03-18T16:21:55.265799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'10","reported_seq":40,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.832997+0000","last_change":"2024-03-18T16:21:21.404930+0000","last_active":"2024-03-18T16:22:05.832997+0000","last_peered":"2024-03-18T16:22:05.832997+0000","last_clean":"2024-03-18T16:22:05.832997+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:22:05.832997+0000","last_undegraded":"2024-03-18T16:22:05.832997+0000","last_fullsized":"2024-03-18T16:22:05.832997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:02.571905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278055+0000","last_change":"2024-03-18T16:21:29.079031+0000","last_active":"2024-03-18T16:21:55.278055+0000","last_peered":"2024-03-18T16:21:55.278055+0000","last_clean":"2024-03-18T16:21:55.278055+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:21:55.278055+0000","last_undegraded":"2024-03-18T16:21:55.278055+0000","last_fullsized":"2024-03-18T16:21:55.278055+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273519+0000","last_change":"2024-03-18T16:21:23.056832+0000","last_active":"2024-03-18T16:21:55.273519+0000","last_peered":"2024-03-18T16:21:55.273519+0000","last_clean":"2024-03-18T16:21:55.273519+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:21:55.273519+0000","last_undegraded":"2024-03-18T16:21:55.273519+0000","last_fullsized":"2024-03-18T16:21:55.273519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:56.124747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273585+0000","last_change":"2024-03-18T16:21:21.404392+0000","last_active":"2024-03-18T16:21:55.273585+0000","last_peered":"2024-03-18T16:21:55.273585+0000","last_clean":"2024-03-18T16:21:55.273585+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:21:55.273585+0000","last_undegraded":"2024-03-18T16:21:55.273585+0000","last_fullsized":"2024-03-18T16:21:55.273585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841565+0000","last_change":"2024-03-18T16:21:29.139956+0000","last_active":"2024-03-18T16:21:54.841565+0000","last_peered":"2024-03-18T16:21:54.841565+0000","last_clean":"2024-03-18T16:21:54.841565+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:21:54.841565+0000","last_undegraded":"2024-03-18T16:21:54.841565+0000","last_fullsized":"2024-03-18T16:21:54.841565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:52.356673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265953+0000","last_change":"2024-03-18T16:21:22.995275+0000","last_active":"2024-03-18T16:21:55.265953+0000","last_peered":"2024-03-18T16:21:55.265953+0000","last_clean":"2024-03-18T16:21:55.265953+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:21:55.265953+0000","last_undegraded":"2024-03-18T16:21:55.265953+0000","last_fullsized":"2024-03-18T16:21:55.265953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:48.024985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.057948+0000","last_change":"2024-03-18T16:21:21.412044+0000","last_active":"2024-03-18T16:22:06.057948+0000","last_peered":"2024-03-18T16:22:06.057948+0000","last_clean":"2024-03-18T16:22:06.057948+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:22:06.057948+0000","last_undegraded":"2024-03-18T16:22:06.057948+0000","last_fullsized":"2024-03-18T16:22:06.057948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.270056+0000","last_change":"2024-03-18T16:21:55.270056+0000","last_active":"2024-03-18T16:21:55.270056+0000","last_peered":"2024-03-18T16:21:55.270056+0000","last_clean":"2024-03-18T16:21:55.270056+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:21:55.270056+0000","last_undegraded":"2024-03-18T16:21:55.270056+0000","last_fullsized":"2024-03-18T16:21:55.270056+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:50.466826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277605+0000","last_change":"2024-03-18T16:21:23.056116+0000","last_active":"2024-03-18T16:21:55.277605+0000","last_peered":"2024-03-18T16:21:55.277605+0000","last_clean":"2024-03-18T16:21:55.277605+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:21:55.277605+0000","last_undegraded":"2024-03-18T16:21:55.277605+0000","last_fullsized":"2024-03-18T16:21:55.277605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:18.455315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.885447+0000","last_change":"2024-03-18T16:21:21.400378+0000","last_active":"2024-03-18T16:22:05.885447+0000","last_peered":"2024-03-18T16:22:05.885447+0000","last_clean":"2024-03-18T16:22:05.885447+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:22:05.885447+0000","last_undegraded":"2024-03-18T16:22:05.885447+0000","last_fullsized":"2024-03-18T16:22:05.885447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064943+0000","last_change":"2024-03-18T16:21:29.141554+0000","last_active":"2024-03-18T16:21:55.064943+0000","last_peered":"2024-03-18T16:21:55.064943+0000","last_clean":"2024-03-18T16:21:55.064943+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:21:55.064943+0000","last_undegraded":"2024-03-18T16:21:55.064943+0000","last_fullsized":"2024-03-18T16:21:55.064943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:25.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266115+0000","last_change":"2024-03-18T16:21:22.996694+0000","last_active":"2024-03-18T16:21:55.266115+0000","last_peered":"2024-03-18T16:21:55.266115+0000","last_clean":"2024-03-18T16:21:55.266115+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:21:55.266115+0000","last_undegraded":"2024-03-18T16:21:55.266115+0000","last_fullsized":"2024-03-18T16:21:55.266115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:03.704090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'12","reported_seq":42,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.041654+0000","last_change":"2024-03-18T16:21:21.412149+0000","last_active":"2024-03-18T16:22:06.041654+0000","last_peered":"2024-03-18T16:22:06.041654+0000","last_clean":"2024-03-18T16:22:06.041654+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:22:06.041654+0000","last_undegraded":"2024-03-18T16:22:06.041654+0000","last_fullsized":"2024-03-18T16:22:06.041654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831282+0000","last_change":"2024-03-18T16:21:29.077350+0000","last_active":"2024-03-18T16:21:54.831282+0000","last_peered":"2024-03-18T16:21:54.831282+0000","last_clean":"2024-03-18T16:21:54.831282+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:21:54.831282+0000","last_undegraded":"2024-03-18T16:21:54.831282+0000","last_fullsized":"2024-03-18T16:21:54.831282+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148199+0000","last_change":"2024-03-18T16:21:22.992928+0000","last_active":"2024-03-18T16:21:55.148199+0000","last_peered":"2024-03-18T16:21:55.148199+0000","last_clean":"2024-03-18T16:21:55.148199+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:21:55.148199+0000","last_undegraded":"2024-03-18T16:21:55.148199+0000","last_fullsized":"2024-03-18T16:21:55.148199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:50.978267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834994+0000","last_change":"2024-03-18T16:21:21.403749+0000","last_active":"2024-03-18T16:21:54.834994+0000","last_peered":"2024-03-18T16:21:54.834994+0000","last_clean":"2024-03-18T16:21:54.834994+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:21:54.834994+0000","last_undegraded":"2024-03-18T16:21:54.834994+0000","last_fullsized":"2024-03-18T16:21:54.834994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:29:30.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147816+0000","last_change":"2024-03-18T16:21:29.065378+0000","last_active":"2024-03-18T16:21:55.147816+0000","last_peered":"2024-03-18T16:21:55.147816+0000","last_clean":"2024-03-18T16:21:55.147816+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:21:55.147816+0000","last_undegraded":"2024-03-18T16:21:55.147816+0000","last_fullsized":"2024-03-18T16:21:55.147816+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:22:12.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278432+0000","last_change":"2024-03-18T16:21:23.054168+0000","last_active":"2024-03-18T16:21:55.278432+0000","last_peered":"2024-03-18T16:21:55.278432+0000","last_clean":"2024-03-18T16:21:55.278432+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:21:55.278432+0000","last_undegraded":"2024-03-18T16:21:55.278432+0000","last_fullsized":"2024-03-18T16:21:55.278432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.054769+0000","last_change":"2024-03-18T16:21:21.398307+0000","last_active":"2024-03-18T16:22:06.054769+0000","last_peered":"2024-03-18T16:22:06.054769+0000","last_clean":"2024-03-18T16:22:06.054769+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:22:06.054769+0000","last_undegraded":"2024-03-18T16:22:06.054769+0000","last_fullsized":"2024-03-18T16:22:06.054769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840228+0000","last_change":"2024-03-18T16:21:29.065855+0000","last_active":"2024-03-18T16:21:54.840228+0000","last_peered":"2024-03-18T16:21:54.840228+0000","last_clean":"2024-03-18T16:21:54.840228+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:21:54.840228+0000","last_undegraded":"2024-03-18T16:21:54.840228+0000","last_fullsized":"2024-03-18T16:21:54.840228+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:43.207302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064726+0000","last_change":"2024-03-18T16:21:22.993606+0000","last_active":"2024-03-18T16:21:55.064726+0000","last_peered":"2024-03-18T16:21:55.064726+0000","last_clean":"2024-03-18T16:21:55.064726+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:21:55.064726+0000","last_undegraded":"2024-03-18T16:21:55.064726+0000","last_fullsized":"2024-03-18T16:21:55.064726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:38:23.121609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'3","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.040632+0000","last_change":"2024-03-18T16:21:21.399588+0000","last_active":"2024-03-18T16:22:06.040632+0000","last_peered":"2024-03-18T16:22:06.040632+0000","last_clean":"2024-03-18T16:22:06.040632+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:22:06.040632+0000","last_undegraded":"2024-03-18T16:22:06.040632+0000","last_fullsized":"2024-03-18T16:22:06.040632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.843888+0000","last_change":"2024-03-18T16:21:29.144832+0000","last_active":"2024-03-18T16:21:54.843888+0000","last_peered":"2024-03-18T16:21:54.843888+0000","last_clean":"2024-03-18T16:21:54.843888+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:21:54.843888+0000","last_undegraded":"2024-03-18T16:21:54.843888+0000","last_fullsized":"2024-03-18T16:21:54.843888+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:10.103662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277724+0000","last_change":"2024-03-18T16:21:29.148215+0000","last_active":"2024-03-18T16:21:55.277724+0000","last_peered":"2024-03-18T16:21:55.277724+0000","last_clean":"2024-03-18T16:21:55.277724+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:21:55.277724+0000","last_undegraded":"2024-03-18T16:21:55.277724+0000","last_fullsized":"2024-03-18T16:21:55.277724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.829181+0000","last_change":"2024-03-18T16:21:22.980300+0000","last_active":"2024-03-18T16:21:54.829181+0000","last_peered":"2024-03-18T16:21:54.829181+0000","last_clean":"2024-03-18T16:21:54.829181+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:21:54.829181+0000","last_undegraded":"2024-03-18T16:21:54.829181+0000","last_fullsized":"2024-03-18T16:21:54.829181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064626+0000","last_change":"2024-03-18T16:21:21.329294+0000","last_active":"2024-03-18T16:21:55.064626+0000","last_peered":"2024-03-18T16:21:55.064626+0000","last_clean":"2024-03-18T16:21:55.064626+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:21:55.064626+0000","last_undegraded":"2024-03-18T16:21:55.064626+0000","last_fullsized":"2024-03-18T16:21:55.064626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273909+0000","last_change":"2024-03-18T16:21:29.076706+0000","last_active":"2024-03-18T16:21:55.273909+0000","last_peered":"2024-03-18T16:21:55.273909+0000","last_clean":"2024-03-18T16:21:55.273909+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:21:55.273909+0000","last_undegraded":"2024-03-18T16:21:55.273909+0000","last_fullsized":"2024-03-18T16:21:55.273909+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:57.880669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840470+0000","last_change":"2024-03-18T16:21:23.052989+0000","last_active":"2024-03-18T16:21:54.840470+0000","last_peered":"2024-03-18T16:21:54.840470+0000","last_clean":"2024-03-18T16:21:54.840470+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:21:54.840470+0000","last_undegraded":"2024-03-18T16:21:54.840470+0000","last_fullsized":"2024-03-18T16:21:54.840470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840520+0000","last_change":"2024-03-18T16:21:21.413020+0000","last_active":"2024-03-18T16:21:54.840520+0000","last_peered":"2024-03-18T16:21:54.840520+0000","last_clean":"2024-03-18T16:21:54.840520+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:21:54.840520+0000","last_undegraded":"2024-03-18T16:21:54.840520+0000","last_fullsized":"2024-03-18T16:21:54.840520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:49.237529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273671+0000","last_change":"2024-03-18T16:21:21.397354+0000","last_active":"2024-03-18T16:21:55.273671+0000","last_peered":"2024-03-18T16:21:55.273671+0000","last_clean":"2024-03-18T16:21:55.273671+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:21:55.273671+0000","last_undegraded":"2024-03-18T16:21:55.273671+0000","last_fullsized":"2024-03-18T16:21:55.273671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278026+0000","last_change":"2024-03-18T16:21:23.059441+0000","last_active":"2024-03-18T16:21:55.278026+0000","last_peered":"2024-03-18T16:21:55.278026+0000","last_clean":"2024-03-18T16:21:55.278026+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:21:55.278026+0000","last_undegraded":"2024-03-18T16:21:55.278026+0000","last_fullsized":"2024-03-18T16:21:55.278026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827518+0000","last_change":"2024-03-18T16:21:29.058800+0000","last_active":"2024-03-18T16:21:54.827518+0000","last_peered":"2024-03-18T16:21:54.827518+0000","last_clean":"2024-03-18T16:21:54.827518+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:21:54.827518+0000","last_undegraded":"2024-03-18T16:21:54.827518+0000","last_fullsized":"2024-03-18T16:21:54.827518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:20.279579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.074918+0000","last_change":"2024-03-18T16:21:21.412536+0000","last_active":"2024-03-18T16:22:06.074918+0000","last_peered":"2024-03-18T16:22:06.074918+0000","last_clean":"2024-03-18T16:22:06.074918+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:22:06.074918+0000","last_undegraded":"2024-03-18T16:22:06.074918+0000","last_fullsized":"2024-03-18T16:22:06.074918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148039+0000","last_change":"2024-03-18T16:21:22.974963+0000","last_active":"2024-03-18T16:21:55.148039+0000","last_peered":"2024-03-18T16:21:55.148039+0000","last_clean":"2024-03-18T16:21:55.148039+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:21:55.148039+0000","last_undegraded":"2024-03-18T16:21:55.148039+0000","last_fullsized":"2024-03-18T16:21:55.148039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:51.820583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.151933+0000","last_change":"2024-03-18T16:21:55.151933+0000","last_active":"2024-03-18T16:21:55.151933+0000","last_peered":"2024-03-18T16:21:55.151933+0000","last_clean":"2024-03-18T16:21:55.151933+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:21:55.151933+0000","last_undegraded":"2024-03-18T16:21:55.151933+0000","last_fullsized":"2024-03-18T16:21:55.151933+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:38.438575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273462+0000","last_change":"2024-03-18T16:21:23.056857+0000","last_active":"2024-03-18T16:21:55.273462+0000","last_peered":"2024-03-18T16:21:55.273462+0000","last_clean":"2024-03-18T16:21:55.273462+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:21:55.273462+0000","last_undegraded":"2024-03-18T16:21:55.273462+0000","last_fullsized":"2024-03-18T16:21:55.273462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:13.728335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827238+0000","last_change":"2024-03-18T16:21:22.949289+0000","last_active":"2024-03-18T16:21:54.827238+0000","last_peered":"2024-03-18T16:21:54.827238+0000","last_clean":"2024-03-18T16:21:54.827238+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:21:54.827238+0000","last_undegraded":"2024-03-18T16:21:54.827238+0000","last_fullsized":"2024-03-18T16:21:54.827238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:51:51.357680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.848377+0000","last_change":"2024-03-18T16:21:54.845858+0000","last_active":"2024-03-18T16:21:54.848377+0000","last_peered":"2024-03-18T16:21:54.848377+0000","last_clean":"2024-03-18T16:21:54.848377+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:21:54.848377+0000","last_undegraded":"2024-03-18T16:21:54.848377+0000","last_fullsized":"2024-03-18T16:21:54.848377+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277673+0000","last_change":"2024-03-18T16:21:23.059438+0000","last_active":"2024-03-18T16:21:55.277673+0000","last_peered":"2024-03-18T16:21:55.277673+0000","last_clean":"2024-03-18T16:21:55.277673+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:21:55.277673+0000","last_undegraded":"2024-03-18T16:21:55.277673+0000","last_fullsized":"2024-03-18T16:21:55.277673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841882+0000","last_change":"2024-03-18T16:21:22.849115+0000","last_active":"2024-03-18T16:21:54.841882+0000","last_peered":"2024-03-18T16:21:54.841882+0000","last_clean":"2024-03-18T16:21:54.841882+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:21:54.841882+0000","last_undegraded":"2024-03-18T16:21:54.841882+0000","last_fullsized":"2024-03-18T16:21:54.841882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:57.469261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834154+0000","last_change":"2024-03-18T16:21:22.980253+0000","last_active":"2024-03-18T16:21:54.834154+0000","last_peered":"2024-03-18T16:21:54.834154+0000","last_clean":"2024-03-18T16:21:54.834154+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:21:54.834154+0000","last_undegraded":"2024-03-18T16:21:54.834154+0000","last_fullsized":"2024-03-18T16:21:54.834154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273836+0000","last_change":"2024-03-18T16:21:23.042188+0000","last_active":"2024-03-18T16:21:55.273836+0000","last_peered":"2024-03-18T16:21:55.273836+0000","last_clean":"2024-03-18T16:21:55.273836+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:21:55.273836+0000","last_undegraded":"2024-03-18T16:21:55.273836+0000","last_fullsized":"2024-03-18T16:21:55.273836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841847+0000","last_change":"2024-03-18T16:21:22.851677+0000","last_active":"2024-03-18T16:21:54.841847+0000","last_peered":"2024-03-18T16:21:54.841847+0000","last_clean":"2024-03-18T16:21:54.841847+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:21:54.841847+0000","last_undegraded":"2024-03-18T16:21:54.841847+0000","last_fullsized":"2024-03-18T16:21:54.841847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064494+0000","last_change":"2024-03-18T16:21:22.947869+0000","last_active":"2024-03-18T16:21:55.064494+0000","last_peered":"2024-03-18T16:21:55.064494+0000","last_clean":"2024-03-18T16:21:55.064494+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:21:55.064494+0000","last_undegraded":"2024-03-18T16:21:55.064494+0000","last_fullsized":"2024-03-18T16:21:55.064494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840151+0000","last_change":"2024-03-18T16:21:22.851663+0000","last_active":"2024-03-18T16:21:54.840151+0000","last_peered":"2024-03-18T16:21:54.840151+0000","last_clean":"2024-03-18T16:21:54.840151+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:21:54.840151+0000","last_undegraded":"2024-03-18T16:21:54.840151+0000","last_fullsized":"2024-03-18T16:21:54.840151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.271992+0000","last_change":"2024-03-18T16:21:55.268883+0000","last_active":"2024-03-18T16:21:55.271992+0000","last_peered":"2024-03-18T16:21:55.271992+0000","last_clean":"2024-03-18T16:21:55.271992+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:21:55.271992+0000","last_undegraded":"2024-03-18T16:21:55.271992+0000","last_fullsized":"2024-03-18T16:21:55.271992+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277397+0000","last_change":"2024-03-18T16:21:23.059733+0000","last_active":"2024-03-18T16:21:55.277397+0000","last_peered":"2024-03-18T16:21:55.277397+0000","last_clean":"2024-03-18T16:21:55.277397+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:21:55.277397+0000","last_undegraded":"2024-03-18T16:21:55.277397+0000","last_fullsized":"2024-03-18T16:21:55.277397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273660+0000","last_change":"2024-03-18T16:21:23.042457+0000","last_active":"2024-03-18T16:21:55.273660+0000","last_peered":"2024-03-18T16:21:55.273660+0000","last_clean":"2024-03-18T16:21:55.273660+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:21:55.273660+0000","last_undegraded":"2024-03-18T16:21:55.273660+0000","last_fullsized":"2024-03-18T16:21:55.273660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:09.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.845542+0000","last_change":"2024-03-18T16:21:54.842423+0000","last_active":"2024-03-18T16:21:54.845542+0000","last_peered":"2024-03-18T16:21:54.845542+0000","last_clean":"2024-03-18T16:21:54.845542+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:21:54.845542+0000","last_undegraded":"2024-03-18T16:21:54.845542+0000","last_fullsized":"2024-03-18T16:21:54.845542+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:36.656058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.828079+0000","last_change":"2024-03-18T16:21:23.045061+0000","last_active":"2024-03-18T16:21:54.828079+0000","last_peered":"2024-03-18T16:21:54.828079+0000","last_clean":"2024-03-18T16:21:54.828079+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:21:54.828079+0000","last_undegraded":"2024-03-18T16:21:54.828079+0000","last_fullsized":"2024-03-18T16:21:54.828079+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:54.119675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065127+0000","last_change":"2024-03-18T16:21:22.993632+0000","last_active":"2024-03-18T16:21:55.065127+0000","last_peered":"2024-03-18T16:21:55.065127+0000","last_clean":"2024-03-18T16:21:55.065127+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:21:55.065127+0000","last_undegraded":"2024-03-18T16:21:55.065127+0000","last_fullsized":"2024-03-18T16:21:55.065127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:35:42.249037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064410+0000","last_change":"2024-03-18T16:21:23.056575+0000","last_active":"2024-03-18T16:21:55.064410+0000","last_peered":"2024-03-18T16:21:55.064410+0000","last_clean":"2024-03-18T16:21:55.064410+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:21:55.064410+0000","last_undegraded":"2024-03-18T16:21:55.064410+0000","last_fullsized":"2024-03-18T16:21:55.064410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831640+0000","last_change":"2024-03-18T16:21:22.973283+0000","last_active":"2024-03-18T16:21:54.831640+0000","last_peered":"2024-03-18T16:21:54.831640+0000","last_clean":"2024-03-18T16:21:54.831640+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:21:54.831640+0000","last_undegraded":"2024-03-18T16:21:54.831640+0000","last_fullsized":"2024-03-18T16:21:54.831640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277550+0000","last_change":"2024-03-18T16:21:23.057150+0000","last_active":"2024-03-18T16:21:55.277550+0000","last_peered":"2024-03-18T16:21:55.277550+0000","last_clean":"2024-03-18T16:21:55.277550+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:21:55.277550+0000","last_undegraded":"2024-03-18T16:21:55.277550+0000","last_fullsized":"2024-03-18T16:21:55.277550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841077+0000","last_change":"2024-03-18T16:21:23.047968+0000","last_active":"2024-03-18T16:21:54.841077+0000","last_peered":"2024-03-18T16:21:54.841077+0000","last_clean":"2024-03-18T16:21:54.841077+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:21:54.841077+0000","last_undegraded":"2024-03-18T16:21:54.841077+0000","last_fullsized":"2024-03-18T16:21:54.841077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148114+0000","last_change":"2024-03-18T16:21:23.041794+0000","last_active":"2024-03-18T16:21:55.148114+0000","last_peered":"2024-03-18T16:21:55.148114+0000","last_clean":"2024-03-18T16:21:55.148114+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:21:55.148114+0000","last_undegraded":"2024-03-18T16:21:55.148114+0000","last_fullsized":"2024-03-18T16:21:55.148114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:13.065494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273737+0000","last_change":"2024-03-18T16:21:23.036940+0000","last_active":"2024-03-18T16:21:55.273737+0000","last_peered":"2024-03-18T16:21:55.273737+0000","last_clean":"2024-03-18T16:21:55.273737+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:21:55.273737+0000","last_undegraded":"2024-03-18T16:21:55.273737+0000","last_fullsized":"2024-03-18T16:21:55.273737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:45:38.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273810+0000","last_change":"2024-03-18T16:21:23.042140+0000","last_active":"2024-03-18T16:21:55.273810+0000","last_peered":"2024-03-18T16:21:55.273810+0000","last_clean":"2024-03-18T16:21:55.273810+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:21:55.273810+0000","last_undegraded":"2024-03-18T16:21:55.273810+0000","last_fullsized":"2024-03-18T16:21:55.273810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266289+0000","last_change":"2024-03-18T16:21:22.995117+0000","last_active":"2024-03-18T16:21:55.266289+0000","last_peered":"2024-03-18T16:21:55.266289+0000","last_clean":"2024-03-18T16:21:55.266289+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:21:55.266289+0000","last_undegraded":"2024-03-18T16:21:55.266289+0000","last_fullsized":"2024-03-18T16:21:55.266289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:57.544738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.839997+0000","last_change":"2024-03-18T16:21:22.948123+0000","last_active":"2024-03-18T16:21:54.839997+0000","last_peered":"2024-03-18T16:21:54.839997+0000","last_clean":"2024-03-18T16:21:54.839997+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:21:54.839997+0000","last_undegraded":"2024-03-18T16:21:54.839997+0000","last_fullsized":"2024-03-18T16:21:54.839997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265696+0000","last_change":"2024-03-18T16:21:23.049877+0000","last_active":"2024-03-18T16:21:55.265696+0000","last_peered":"2024-03-18T16:21:55.265696+0000","last_clean":"2024-03-18T16:21:55.265696+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:21:55.265696+0000","last_undegraded":"2024-03-18T16:21:55.265696+0000","last_fullsized":"2024-03-18T16:21:55.265696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064503+0000","last_change":"2024-03-18T16:21:23.049483+0000","last_active":"2024-03-18T16:21:55.064503+0000","last_peered":"2024-03-18T16:21:55.064503+0000","last_clean":"2024-03-18T16:21:55.064503+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:21:55.064503+0000","last_undegraded":"2024-03-18T16:21:55.064503+0000","last_fullsized":"2024-03-18T16:21:55.064503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265858+0000","last_change":"2024-03-18T16:21:22.995726+0000","last_active":"2024-03-18T16:21:55.265858+0000","last_peered":"2024-03-18T16:21:55.265858+0000","last_clean":"2024-03-18T16:21:55.265858+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:21:55.265858+0000","last_undegraded":"2024-03-18T16:21:55.265858+0000","last_fullsized":"2024-03-18T16:21:55.265858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064410+0000","last_change":"2024-03-18T16:21:22.949640+0000","last_active":"2024-03-18T16:21:55.064410+0000","last_peered":"2024-03-18T16:21:55.064410+0000","last_clean":"2024-03-18T16:21:55.064410+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:21:55.064410+0000","last_undegraded":"2024-03-18T16:21:55.064410+0000","last_fullsized":"2024-03-18T16:21:55.064410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:21.734293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277697+0000","last_change":"2024-03-18T16:21:23.050203+0000","last_active":"2024-03-18T16:21:55.277697+0000","last_peered":"2024-03-18T16:21:55.277697+0000","last_clean":"2024-03-18T16:21:55.277697+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:21:55.277697+0000","last_undegraded":"2024-03-18T16:21:55.277697+0000","last_fullsized":"2024-03-18T16:21:55.277697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:21.982408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834586+0000","last_change":"2024-03-18T16:21:23.053003+0000","last_active":"2024-03-18T16:21:54.834586+0000","last_peered":"2024-03-18T16:21:54.834586+0000","last_clean":"2024-03-18T16:21:54.834586+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:21:54.834586+0000","last_undegraded":"2024-03-18T16:21:54.834586+0000","last_fullsized":"2024-03-18T16:21:54.834586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266295+0000","last_change":"2024-03-18T16:21:23.041863+0000","last_active":"2024-03-18T16:21:55.266295+0000","last_peered":"2024-03-18T16:21:55.266295+0000","last_clean":"2024-03-18T16:21:55.266295+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:21:55.266295+0000","last_undegraded":"2024-03-18T16:21:55.266295+0000","last_fullsized":"2024-03-18T16:21:55.266295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:26.190759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509456,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":393216,"data_stored":721186,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509457,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":129628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":133966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":130334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":389120,"data_stored":720530,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542160,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":129502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":133434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":130264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:22:11.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:22:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.655+0000 7f8b8ea00640 1 Processor -- start 2024-03-18T16:22:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.655+0000 7f8b8ea00640 1 -- start start 2024-03-18T16:22:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.656+0000 7f8b8ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8b88072700 0x7f8b88158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.656+0000 7f8b8ea00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b8815c200 con 0x7f8b8815b8e0 2024-03-18T16:22:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.656+0000 7f8b8ea00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b8815c4e0 con 0x7f8b881524c0 2024-03-18T16:22:11.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.656+0000 7f8b8ea00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8b88072b20 con 0x7f8b88072700 2024-03-18T16:22:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.657+0000 7f8b87e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8b881524c0 0x7f8b8806e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58464/0 (socket says 172.21.15.102:58464) 2024-03-18T16:22:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.657+0000 7f8b87e00640 1 -- 172.21.15.102:0/78946282 learned_addr learned my addr 172.21.15.102:0/78946282 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:11.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.658+0000 7f8b87400640 1 --2- 172.21.15.102:0/78946282 >> v2:172.21.15.102:3300/0 conn(0x7f8b88072700 0x7f8b88158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:11.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.660+0000 7f8b86a00640 1 -- 172.21.15.102:0/78946282 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475985711 0 0) 0x7f8b8815c4e0 con 0x7f8b881524c0 2024-03-18T16:22:11.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.660+0000 7f8b86a00640 1 -- 172.21.15.102:0/78946282 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b6c003520 con 0x7f8b881524c0 2024-03-18T16:22:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.663+0000 7f8b87400640 1 -- 172.21.15.102:0/78946282 >> v1:172.21.15.154:6789/0 conn(0x7f8b881524c0 legacy=0x7f8b8806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.663+0000 7f8b87400640 1 -- 172.21.15.102:0/78946282 >> v1:172.21.15.102:6789/0 conn(0x7f8b8815b8e0 legacy=0x7f8b880c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:11.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.663+0000 7f8b87400640 1 -- 172.21.15.102:0/78946282 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8815c7c0 con 0x7f8b88072700 2024-03-18T16:22:11.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.668+0000 7f8b87400640 1 --2- 172.21.15.102:0/78946282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b88072700 0x7f8b88158ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8b740097a0 tx=0x7f8b7402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8761e44d3a21bbab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.673+0000 7f8b86a00640 1 -- 172.21.15.102:0/78946282 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74034070 con 0x7f8b88072700 2024-03-18T16:22:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.673+0000 7f8b86a00640 1 -- 172.21.15.102:0/78946282 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b74030610 con 0x7f8b88072700 2024-03-18T16:22:11.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.673+0000 7f8b86a00640 1 -- 172.21.15.102:0/78946282 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b740309e0 con 0x7f8b88072700 2024-03-18T16:22:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.674+0000 7f8b8ea00640 1 -- 172.21.15.102:0/78946282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b88072700 msgr2=0x7f8b88158ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.674+0000 7f8b8ea00640 1 --2- 172.21.15.102:0/78946282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b88072700 0x7f8b88158ee0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8b740097a0 tx=0x7f8b7402b960 comp rx=0 tx=0).stop 2024-03-18T16:22:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.674+0000 7f8b8ea00640 1 -- 172.21.15.102:0/78946282 shutdown_connections 2024-03-18T16:22:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.674+0000 7f8b8ea00640 1 --2- 172.21.15.102:0/78946282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b88072700 0x7f8b88158ee0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.674+0000 7f8b8ea00640 1 -- 172.21.15.102:0/78946282 >> 172.21.15.102:0/78946282 conn(0x7f8b880700d0 msgr2=0x7f8b8806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.674+0000 7f8b8ea00640 1 -- 172.21.15.102:0/78946282 shutdown_connections 2024-03-18T16:22:11.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.674+0000 7f8b8ea00640 1 -- 172.21.15.102:0/78946282 wait complete. 2024-03-18T16:22:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.677+0000 7f8b8ea00640 1 Processor -- start 2024-03-18T16:22:11.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.677+0000 7f8b8ea00640 1 -- start start 2024-03-18T16:22:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.678+0000 7f8b8ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b88072700 0x7f8b881f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.678+0000 7f8b8ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b881524c0 0x7f8b881f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.678+0000 7f8b8ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8815b8e0 0x7f8b8814fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.678+0000 7f8b8ea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881589c0 con 0x7f8b8815b8e0 2024-03-18T16:22:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.678+0000 7f8b8ea00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881586e0 con 0x7f8b88072700 2024-03-18T16:22:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.678+0000 7f8b8ea00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8b881528a0 con 0x7f8b881524c0 2024-03-18T16:22:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.679+0000 7f8b87e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b88072700 0x7f8b881f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.679+0000 7f8b8ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8815b8e0 0x7f8b8814fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.679+0000 7f8b87400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b881524c0 0x7f8b881f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.679+0000 7f8b87e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b88072700 0x7f8b881f34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47316/0 (socket says 172.21.15.102:47316) 2024-03-18T16:22:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.679+0000 7f8b87e00640 1 -- 172.21.15.102:0/2867962938 learned_addr learned my addr 172.21.15.102:0/2867962938 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.683+0000 7f8b8ca00640 1 -- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b881524c0 msgr2=0x7f8b881f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.683+0000 7f8b8ca00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b881524c0 0x7f8b881f3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.683+0000 7f8b8ca00640 1 -- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b88072700 msgr2=0x7f8b881f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.683+0000 7f8b8ca00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b88072700 0x7f8b881f34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:11.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.683+0000 7f8b8ca00640 1 -- 172.21.15.102:0/2867962938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8814f710 con 0x7f8b8815b8e0 2024-03-18T16:22:11.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.683+0000 7f8b87400640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b881524c0 0x7f8b881f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:22:11.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.686+0000 7f8b8ca00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8815b8e0 0x7f8b8814fef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000c700 tx=0x7f8b8000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:11.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.689+0000 7f8b7fe00640 1 -- 172.21.15.102:0/2867962938 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b80003b70 con 0x7f8b8815b8e0 2024-03-18T16:22:11.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.689+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b881f4050 con 0x7f8b8815b8e0 2024-03-18T16:22:11.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.689+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b880adc20 con 0x7f8b8815b8e0 2024-03-18T16:22:11.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.690+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b880c8a00 con 0x7f8b8815b8e0 2024-03-18T16:22:11.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.691+0000 7f8b7fe00640 1 -- 172.21.15.102:0/2867962938 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b80003d10 con 0x7f8b8815b8e0 2024-03-18T16:22:11.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.691+0000 7f8b7fe00640 1 -- 172.21.15.102:0/2867962938 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b800052f0 con 0x7f8b8815b8e0 2024-03-18T16:22:11.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.996+0000 7f8b7fe00640 1 -- 172.21.15.102:0/2867962938 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8b80005490 con 0x7f8b8815b8e0 2024-03-18T16:22:11.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.997+0000 7f8b7fe00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b6806bbd0 0x7f8b6806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:11.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.998+0000 7f8b87e00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b6806bbd0 0x7f8b6806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:12.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:11.999+0000 7f8b87e00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b6806bbd0 0x7f8b6806e090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8b780011e0 tx=0x7f8b78009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.034+0000 7f8b7fe00640 1 -- 172.21.15.102:0/2867962938 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f8b80057700 con 0x7f8b8815b8e0 2024-03-18T16:22:12.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.034+0000 7f8b7fe00640 1 -- 172.21.15.102:0/2867962938 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8b880c8a00 con 0x7f8b8815b8e0 2024-03-18T16:22:12.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.248+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8b88074fd0 con 0x7f8b6806bbd0 2024-03-18T16:22:12.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.259+0000 7f8b7fe00640 1 -- 172.21.15.102:0/2867962938 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332709 (secure 0 0 0) 0x7f8b88074fd0 con 0x7f8b6806bbd0 2024-03-18T16:22:12.263 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:12.270 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:22:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b6806bbd0 msgr2=0x7f8b6806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b6806bbd0 0x7f8b6806e090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8b780011e0 tx=0x7f8b78009040 comp rx=0 tx=0).stop 2024-03-18T16:22:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8815b8e0 msgr2=0x7f8b8814fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8815b8e0 0x7f8b8814fef0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b8000c700 tx=0x7f8b8000cbc0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 shutdown_connections 2024-03-18T16:22:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b6806bbd0 0x7f8b6806e090 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8815b8e0 0x7f8b8814fef0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b881524c0 0x7f8b881f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 --2- 172.21.15.102:0/2867962938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b88072700 0x7f8b881f34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 >> 172.21.15.102:0/2867962938 conn(0x7f8b880700d0 msgr2=0x7f8b88157700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.269+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 shutdown_connections 2024-03-18T16:22:12.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.270+0000 7f8b8ea00640 1 -- 172.21.15.102:0/2867962938 wait complete. 2024-03-18T16:22:12.291 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-03-18T16:22:10.823759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":247,"num_write_kb":2447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218768,"kb_used_data":2640,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755952,"statfs":{"total":773094113280,"available":772870094848,"internally_reserved":0,"allocated":2703360,"data_stored":2228844,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002911"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278633+0000","last_change":"2024-03-18T16:21:21.402212+0000","last_active":"2024-03-18T16:21:55.278633+0000","last_peered":"2024-03-18T16:21:55.278633+0000","last_clean":"2024-03-18T16:21:55.278633+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:21:55.278633+0000","last_undegraded":"2024-03-18T16:21:55.278633+0000","last_fullsized":"2024-03-18T16:21:55.278633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:33.215915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841700+0000","last_change":"2024-03-18T16:21:23.045605+0000","last_active":"2024-03-18T16:21:54.841700+0000","last_peered":"2024-03-18T16:21:54.841700+0000","last_clean":"2024-03-18T16:21:54.841700+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:21:54.841700+0000","last_undegraded":"2024-03-18T16:21:54.841700+0000","last_fullsized":"2024-03-18T16:21:54.841700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.839646+0000","last_change":"2024-03-18T16:21:29.146958+0000","last_active":"2024-03-18T16:21:54.839646+0000","last_peered":"2024-03-18T16:21:54.839646+0000","last_clean":"2024-03-18T16:21:54.839646+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:21:54.839646+0000","last_undegraded":"2024-03-18T16:21:54.839646+0000","last_fullsized":"2024-03-18T16:21:54.839646+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:55.871927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842531+0000","last_change":"2024-03-18T16:21:21.413060+0000","last_active":"2024-03-18T16:21:54.842531+0000","last_peered":"2024-03-18T16:21:54.842531+0000","last_clean":"2024-03-18T16:21:54.842531+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:21:54.842531+0000","last_undegraded":"2024-03-18T16:21:54.842531+0000","last_fullsized":"2024-03-18T16:21:54.842531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:40.114937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278167+0000","last_change":"2024-03-18T16:21:23.050849+0000","last_active":"2024-03-18T16:21:55.278167+0000","last_peered":"2024-03-18T16:21:55.278167+0000","last_clean":"2024-03-18T16:21:55.278167+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:21:55.278167+0000","last_undegraded":"2024-03-18T16:21:55.278167+0000","last_fullsized":"2024-03-18T16:21:55.278167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147644+0000","last_change":"2024-03-18T16:21:29.065755+0000","last_active":"2024-03-18T16:21:55.147644+0000","last_peered":"2024-03-18T16:21:55.147644+0000","last_clean":"2024-03-18T16:21:55.147644+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:21:55.147644+0000","last_undegraded":"2024-03-18T16:21:55.147644+0000","last_fullsized":"2024-03-18T16:21:55.147644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:45.934586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266332+0000","last_change":"2024-03-18T16:21:21.398188+0000","last_active":"2024-03-18T16:21:55.266332+0000","last_peered":"2024-03-18T16:21:55.266332+0000","last_clean":"2024-03-18T16:21:55.266332+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:21:55.266332+0000","last_undegraded":"2024-03-18T16:21:55.266332+0000","last_fullsized":"2024-03-18T16:21:55.266332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064521+0000","last_change":"2024-03-18T16:21:22.946288+0000","last_active":"2024-03-18T16:21:55.064521+0000","last_peered":"2024-03-18T16:21:55.064521+0000","last_clean":"2024-03-18T16:21:55.064521+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:21:55.064521+0000","last_undegraded":"2024-03-18T16:21:55.064521+0000","last_fullsized":"2024-03-18T16:21:55.064521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:08.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147915+0000","last_change":"2024-03-18T16:21:29.065891+0000","last_active":"2024-03-18T16:21:55.147915+0000","last_peered":"2024-03-18T16:21:55.147915+0000","last_clean":"2024-03-18T16:21:55.147915+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:21:55.147915+0000","last_undegraded":"2024-03-18T16:21:55.147915+0000","last_fullsized":"2024-03-18T16:21:55.147915+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265954+0000","last_change":"2024-03-18T16:21:21.397071+0000","last_active":"2024-03-18T16:21:55.265954+0000","last_peered":"2024-03-18T16:21:55.265954+0000","last_clean":"2024-03-18T16:21:55.265954+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:21:55.265954+0000","last_undegraded":"2024-03-18T16:21:55.265954+0000","last_fullsized":"2024-03-18T16:21:55.265954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065035+0000","last_change":"2024-03-18T16:21:22.946468+0000","last_active":"2024-03-18T16:21:55.065035+0000","last_peered":"2024-03-18T16:21:55.065035+0000","last_clean":"2024-03-18T16:21:55.065035+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:21:55.065035+0000","last_undegraded":"2024-03-18T16:21:55.065035+0000","last_fullsized":"2024-03-18T16:21:55.065035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147794+0000","last_change":"2024-03-18T16:21:29.073724+0000","last_active":"2024-03-18T16:21:55.147794+0000","last_peered":"2024-03-18T16:21:55.147794+0000","last_clean":"2024-03-18T16:21:55.147794+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:21:55.147794+0000","last_undegraded":"2024-03-18T16:21:55.147794+0000","last_fullsized":"2024-03-18T16:21:55.147794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273576+0000","last_change":"2024-03-18T16:21:23.041926+0000","last_active":"2024-03-18T16:21:55.273576+0000","last_peered":"2024-03-18T16:21:55.273576+0000","last_clean":"2024-03-18T16:21:55.273576+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:21:55.273576+0000","last_undegraded":"2024-03-18T16:21:55.273576+0000","last_fullsized":"2024-03-18T16:21:55.273576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:03.532253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.821045+0000","last_change":"2024-03-18T16:21:21.394801+0000","last_active":"2024-03-18T16:22:05.821045+0000","last_peered":"2024-03-18T16:22:05.821045+0000","last_clean":"2024-03-18T16:22:05.821045+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:22:05.821045+0000","last_undegraded":"2024-03-18T16:22:05.821045+0000","last_fullsized":"2024-03-18T16:22:05.821045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148181+0000","last_change":"2024-03-18T16:21:29.073757+0000","last_active":"2024-03-18T16:21:55.148181+0000","last_peered":"2024-03-18T16:21:55.148181+0000","last_clean":"2024-03-18T16:21:55.148181+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:21:55.148181+0000","last_undegraded":"2024-03-18T16:21:55.148181+0000","last_fullsized":"2024-03-18T16:21:55.148181+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841346+0000","last_change":"2024-03-18T16:21:23.060776+0000","last_active":"2024-03-18T16:21:54.841346+0000","last_peered":"2024-03-18T16:21:54.841346+0000","last_clean":"2024-03-18T16:21:54.841346+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:21:54.841346+0000","last_undegraded":"2024-03-18T16:21:54.841346+0000","last_fullsized":"2024-03-18T16:21:54.841346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831728+0000","last_change":"2024-03-18T16:21:21.412733+0000","last_active":"2024-03-18T16:21:54.831728+0000","last_peered":"2024-03-18T16:21:54.831728+0000","last_clean":"2024-03-18T16:21:54.831728+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:21:54.831728+0000","last_undegraded":"2024-03-18T16:21:54.831728+0000","last_fullsized":"2024-03-18T16:21:54.831728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148007+0000","last_change":"2024-03-18T16:21:29.063980+0000","last_active":"2024-03-18T16:21:55.148007+0000","last_peered":"2024-03-18T16:21:55.148007+0000","last_clean":"2024-03-18T16:21:55.148007+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:21:55.148007+0000","last_undegraded":"2024-03-18T16:21:55.148007+0000","last_fullsized":"2024-03-18T16:21:55.148007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:39.469490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.845055+0000","last_change":"2024-03-18T16:21:54.838108+0000","last_active":"2024-03-18T16:21:54.845055+0000","last_peered":"2024-03-18T16:21:54.845055+0000","last_clean":"2024-03-18T16:21:54.845055+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:21:54.845055+0000","last_undegraded":"2024-03-18T16:21:54.845055+0000","last_fullsized":"2024-03-18T16:21:54.845055+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:40.182971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.826842+0000","last_change":"2024-03-18T16:21:21.407132+0000","last_active":"2024-03-18T16:22:05.826842+0000","last_peered":"2024-03-18T16:22:05.826842+0000","last_clean":"2024-03-18T16:22:05.826842+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:22:05.826842+0000","last_undegraded":"2024-03-18T16:22:05.826842+0000","last_fullsized":"2024-03-18T16:22:05.826842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.272088+0000","last_change":"2024-03-18T16:21:55.269661+0000","last_active":"2024-03-18T16:21:55.272088+0000","last_peered":"2024-03-18T16:21:55.272088+0000","last_clean":"2024-03-18T16:21:55.272088+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:21:55.272088+0000","last_undegraded":"2024-03-18T16:21:55.272088+0000","last_fullsized":"2024-03-18T16:21:55.272088+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:38.290937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148142+0000","last_change":"2024-03-18T16:21:22.840551+0000","last_active":"2024-03-18T16:21:55.148142+0000","last_peered":"2024-03-18T16:21:55.148142+0000","last_clean":"2024-03-18T16:21:55.148142+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:21:55.148142+0000","last_undegraded":"2024-03-18T16:21:55.148142+0000","last_fullsized":"2024-03-18T16:21:55.148142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:17.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.822682+0000","last_change":"2024-03-18T16:21:21.400349+0000","last_active":"2024-03-18T16:22:05.822682+0000","last_peered":"2024-03-18T16:22:05.822682+0000","last_clean":"2024-03-18T16:22:05.822682+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:22:05.822682+0000","last_undegraded":"2024-03-18T16:22:05.822682+0000","last_fullsized":"2024-03-18T16:22:05.822682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148262+0000","last_change":"2024-03-18T16:21:29.064624+0000","last_active":"2024-03-18T16:21:55.148262+0000","last_peered":"2024-03-18T16:21:55.148262+0000","last_clean":"2024-03-18T16:21:55.148262+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:21:55.148262+0000","last_undegraded":"2024-03-18T16:21:55.148262+0000","last_fullsized":"2024-03-18T16:21:55.148262+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278343+0000","last_change":"2024-03-18T16:21:23.055982+0000","last_active":"2024-03-18T16:21:55.278343+0000","last_peered":"2024-03-18T16:21:55.278343+0000","last_clean":"2024-03-18T16:21:55.278343+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:21:55.278343+0000","last_undegraded":"2024-03-18T16:21:55.278343+0000","last_fullsized":"2024-03-18T16:21:55.278343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.038917+0000","last_change":"2024-03-18T16:21:21.400843+0000","last_active":"2024-03-18T16:22:06.038917+0000","last_peered":"2024-03-18T16:22:06.038917+0000","last_clean":"2024-03-18T16:22:06.038917+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:22:06.038917+0000","last_undegraded":"2024-03-18T16:22:06.038917+0000","last_fullsized":"2024-03-18T16:22:06.038917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147884+0000","last_change":"2024-03-18T16:21:29.140888+0000","last_active":"2024-03-18T16:21:55.147884+0000","last_peered":"2024-03-18T16:21:55.147884+0000","last_clean":"2024-03-18T16:21:55.147884+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:21:55.147884+0000","last_undegraded":"2024-03-18T16:21:55.147884+0000","last_fullsized":"2024-03-18T16:21:55.147884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842117+0000","last_change":"2024-03-18T16:21:22.994455+0000","last_active":"2024-03-18T16:21:54.842117+0000","last_peered":"2024-03-18T16:21:54.842117+0000","last_clean":"2024-03-18T16:21:54.842117+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:21:54.842117+0000","last_undegraded":"2024-03-18T16:21:54.842117+0000","last_fullsized":"2024-03-18T16:21:54.842117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834764+0000","last_change":"2024-03-18T16:21:21.402920+0000","last_active":"2024-03-18T16:21:54.834764+0000","last_peered":"2024-03-18T16:21:54.834764+0000","last_clean":"2024-03-18T16:21:54.834764+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:21:54.834764+0000","last_undegraded":"2024-03-18T16:21:54.834764+0000","last_fullsized":"2024-03-18T16:21:54.834764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.838803+0000","last_change":"2024-03-18T16:21:29.145235+0000","last_active":"2024-03-18T16:21:54.838803+0000","last_peered":"2024-03-18T16:21:54.838803+0000","last_clean":"2024-03-18T16:21:54.838803+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:21:54.838803+0000","last_undegraded":"2024-03-18T16:21:54.838803+0000","last_fullsized":"2024-03-18T16:21:54.838803+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:42.379951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.833875+0000","last_change":"2024-03-18T16:21:22.975172+0000","last_active":"2024-03-18T16:21:54.833875+0000","last_peered":"2024-03-18T16:21:54.833875+0000","last_clean":"2024-03-18T16:21:54.833875+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:21:54.833875+0000","last_undegraded":"2024-03-18T16:21:54.833875+0000","last_fullsized":"2024-03-18T16:21:54.833875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266381+0000","last_change":"2024-03-18T16:21:21.399688+0000","last_active":"2024-03-18T16:21:55.266381+0000","last_peered":"2024-03-18T16:21:55.266381+0000","last_clean":"2024-03-18T16:21:55.266381+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:21:55.266381+0000","last_undegraded":"2024-03-18T16:21:55.266381+0000","last_fullsized":"2024-03-18T16:21:55.266381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:11.933810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277363+0000","last_change":"2024-03-18T16:21:29.141555+0000","last_active":"2024-03-18T16:21:55.277363+0000","last_peered":"2024-03-18T16:21:55.277363+0000","last_clean":"2024-03-18T16:21:55.277363+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:21:55.277363+0000","last_undegraded":"2024-03-18T16:21:55.277363+0000","last_fullsized":"2024-03-18T16:21:55.277363+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065093+0000","last_change":"2024-03-18T16:21:22.947980+0000","last_active":"2024-03-18T16:21:55.065093+0000","last_peered":"2024-03-18T16:21:55.065093+0000","last_clean":"2024-03-18T16:21:55.065093+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:21:55.065093+0000","last_undegraded":"2024-03-18T16:21:55.065093+0000","last_fullsized":"2024-03-18T16:21:55.065093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:31.739693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834138+0000","last_change":"2024-03-18T16:21:21.400087+0000","last_active":"2024-03-18T16:21:54.834138+0000","last_peered":"2024-03-18T16:21:54.834138+0000","last_clean":"2024-03-18T16:21:54.834138+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:21:54.834138+0000","last_undegraded":"2024-03-18T16:21:54.834138+0000","last_fullsized":"2024-03-18T16:21:54.834138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.830782+0000","last_change":"2024-03-18T16:21:29.076629+0000","last_active":"2024-03-18T16:21:54.830782+0000","last_peered":"2024-03-18T16:21:54.830782+0000","last_clean":"2024-03-18T16:21:54.830782+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:21:54.830782+0000","last_undegraded":"2024-03-18T16:21:54.830782+0000","last_fullsized":"2024-03-18T16:21:54.830782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148132+0000","last_change":"2024-03-18T16:21:23.055810+0000","last_active":"2024-03-18T16:21:55.148132+0000","last_peered":"2024-03-18T16:21:55.148132+0000","last_clean":"2024-03-18T16:21:55.148132+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:21:55.148132+0000","last_undegraded":"2024-03-18T16:21:55.148132+0000","last_fullsized":"2024-03-18T16:21:55.148132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:40.451255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064938+0000","last_change":"2024-03-18T16:21:21.399134+0000","last_active":"2024-03-18T16:21:55.064938+0000","last_peered":"2024-03-18T16:21:55.064938+0000","last_clean":"2024-03-18T16:21:55.064938+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:21:55.064938+0000","last_undegraded":"2024-03-18T16:21:55.064938+0000","last_fullsized":"2024-03-18T16:21:55.064938+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834980+0000","last_change":"2024-03-18T16:21:29.067567+0000","last_active":"2024-03-18T16:21:54.834980+0000","last_peered":"2024-03-18T16:21:54.834980+0000","last_clean":"2024-03-18T16:21:54.834980+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:21:54.834980+0000","last_undegraded":"2024-03-18T16:21:54.834980+0000","last_fullsized":"2024-03-18T16:21:54.834980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.832134+0000","last_change":"2024-03-18T16:21:22.977089+0000","last_active":"2024-03-18T16:21:54.832134+0000","last_peered":"2024-03-18T16:21:54.832134+0000","last_clean":"2024-03-18T16:21:54.832134+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:21:54.832134+0000","last_undegraded":"2024-03-18T16:21:54.832134+0000","last_fullsized":"2024-03-18T16:21:54.832134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827052+0000","last_change":"2024-03-18T16:21:21.408450+0000","last_active":"2024-03-18T16:21:54.827052+0000","last_peered":"2024-03-18T16:21:54.827052+0000","last_clean":"2024-03-18T16:21:54.827052+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:21:54.827052+0000","last_undegraded":"2024-03-18T16:21:54.827052+0000","last_fullsized":"2024-03-18T16:21:54.827052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.835511+0000","last_change":"2024-03-18T16:21:29.076875+0000","last_active":"2024-03-18T16:21:54.835511+0000","last_peered":"2024-03-18T16:21:54.835511+0000","last_clean":"2024-03-18T16:21:54.835511+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:21:54.835511+0000","last_undegraded":"2024-03-18T16:21:54.835511+0000","last_fullsized":"2024-03-18T16:21:54.835511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277013+0000","last_change":"2024-03-18T16:21:23.054497+0000","last_active":"2024-03-18T16:21:55.277013+0000","last_peered":"2024-03-18T16:21:55.277013+0000","last_clean":"2024-03-18T16:21:55.277013+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:21:55.277013+0000","last_undegraded":"2024-03-18T16:21:55.277013+0000","last_fullsized":"2024-03-18T16:21:55.277013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:37.455180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265682+0000","last_change":"2024-03-18T16:21:21.404754+0000","last_active":"2024-03-18T16:21:55.265682+0000","last_peered":"2024-03-18T16:21:55.265682+0000","last_clean":"2024-03-18T16:21:55.265682+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:21:55.265682+0000","last_undegraded":"2024-03-18T16:21:55.265682+0000","last_fullsized":"2024-03-18T16:21:55.265682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278071+0000","last_change":"2024-03-18T16:21:29.147316+0000","last_active":"2024-03-18T16:21:55.278071+0000","last_peered":"2024-03-18T16:21:55.278071+0000","last_clean":"2024-03-18T16:21:55.278071+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:21:55.278071+0000","last_undegraded":"2024-03-18T16:21:55.278071+0000","last_fullsized":"2024-03-18T16:21:55.278071+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842010+0000","last_change":"2024-03-18T16:21:22.947635+0000","last_active":"2024-03-18T16:21:54.842010+0000","last_peered":"2024-03-18T16:21:54.842010+0000","last_clean":"2024-03-18T16:21:54.842010+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:21:54.842010+0000","last_undegraded":"2024-03-18T16:21:54.842010+0000","last_fullsized":"2024-03-18T16:21:54.842010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.058595+0000","last_change":"2024-03-18T16:21:21.381306+0000","last_active":"2024-03-18T16:22:06.058595+0000","last_peered":"2024-03-18T16:22:06.058595+0000","last_clean":"2024-03-18T16:22:06.058595+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:22:06.058595+0000","last_undegraded":"2024-03-18T16:22:06.058595+0000","last_fullsized":"2024-03-18T16:22:06.058595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":186,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265740+0000","last_change":"2024-03-18T16:21:08.662767+0000","last_active":"2024-03-18T16:21:55.265740+0000","last_peered":"2024-03-18T16:21:55.265740+0000","last_clean":"2024-03-18T16:21:55.265740+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:21:55.265740+0000","last_undegraded":"2024-03-18T16:21:55.265740+0000","last_fullsized":"2024-03-18T16:21:55.265740+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_clean_scrub_stamp":"2024-03-18T16:21:05.847994+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:12:10.494707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273700+0000","last_change":"2024-03-18T16:21:29.145697+0000","last_active":"2024-03-18T16:21:55.273700+0000","last_peered":"2024-03-18T16:21:55.273700+0000","last_clean":"2024-03-18T16:21:55.273700+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:21:55.273700+0000","last_undegraded":"2024-03-18T16:21:55.273700+0000","last_fullsized":"2024-03-18T16:21:55.273700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:46.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266203+0000","last_change":"2024-03-18T16:21:22.995291+0000","last_active":"2024-03-18T16:21:55.266203+0000","last_peered":"2024-03-18T16:21:55.266203+0000","last_clean":"2024-03-18T16:21:55.266203+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:21:55.266203+0000","last_undegraded":"2024-03-18T16:21:55.266203+0000","last_fullsized":"2024-03-18T16:21:55.266203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:32.918246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.828217+0000","last_change":"2024-03-18T16:21:21.382137+0000","last_active":"2024-03-18T16:21:54.828217+0000","last_peered":"2024-03-18T16:21:54.828217+0000","last_clean":"2024-03-18T16:21:54.828217+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:21:54.828217+0000","last_undegraded":"2024-03-18T16:21:54.828217+0000","last_fullsized":"2024-03-18T16:21:54.828217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273492+0000","last_change":"2024-03-18T16:21:29.139432+0000","last_active":"2024-03-18T16:21:55.273492+0000","last_peered":"2024-03-18T16:21:55.273492+0000","last_clean":"2024-03-18T16:21:55.273492+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:21:55.273492+0000","last_undegraded":"2024-03-18T16:21:55.273492+0000","last_fullsized":"2024-03-18T16:21:55.273492+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:14.828125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064603+0000","last_change":"2024-03-18T16:21:22.949578+0000","last_active":"2024-03-18T16:21:55.064603+0000","last_peered":"2024-03-18T16:21:55.064603+0000","last_clean":"2024-03-18T16:21:55.064603+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:21:55.064603+0000","last_undegraded":"2024-03-18T16:21:55.064603+0000","last_fullsized":"2024-03-18T16:21:55.064603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:36:20.997720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.835156+0000","last_change":"2024-03-18T16:21:21.400859+0000","last_active":"2024-03-18T16:21:54.835156+0000","last_peered":"2024-03-18T16:21:54.835156+0000","last_clean":"2024-03-18T16:21:54.835156+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:21:54.835156+0000","last_undegraded":"2024-03-18T16:21:54.835156+0000","last_fullsized":"2024-03-18T16:21:54.835156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277608+0000","last_change":"2024-03-18T16:21:29.076813+0000","last_active":"2024-03-18T16:21:55.277608+0000","last_peered":"2024-03-18T16:21:55.277608+0000","last_clean":"2024-03-18T16:21:55.277608+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:21:55.277608+0000","last_undegraded":"2024-03-18T16:21:55.277608+0000","last_fullsized":"2024-03-18T16:21:55.277608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266390+0000","last_change":"2024-03-18T16:21:23.000009+0000","last_active":"2024-03-18T16:21:55.266390+0000","last_peered":"2024-03-18T16:21:55.266390+0000","last_clean":"2024-03-18T16:21:55.266390+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:21:55.266390+0000","last_undegraded":"2024-03-18T16:21:55.266390+0000","last_fullsized":"2024-03-18T16:21:55.266390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:39.383342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.828352+0000","last_change":"2024-03-18T16:21:21.409271+0000","last_active":"2024-03-18T16:21:54.828352+0000","last_peered":"2024-03-18T16:21:54.828352+0000","last_clean":"2024-03-18T16:21:54.828352+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:21:54.828352+0000","last_undegraded":"2024-03-18T16:21:54.828352+0000","last_fullsized":"2024-03-18T16:21:54.828352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064256+0000","last_change":"2024-03-18T16:21:29.139595+0000","last_active":"2024-03-18T16:21:55.064256+0000","last_peered":"2024-03-18T16:21:55.064256+0000","last_clean":"2024-03-18T16:21:55.064256+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:21:55.064256+0000","last_undegraded":"2024-03-18T16:21:55.064256+0000","last_fullsized":"2024-03-18T16:21:55.064256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:49.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266190+0000","last_change":"2024-03-18T16:21:23.000054+0000","last_active":"2024-03-18T16:21:55.266190+0000","last_peered":"2024-03-18T16:21:55.266190+0000","last_clean":"2024-03-18T16:21:55.266190+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:21:55.266190+0000","last_undegraded":"2024-03-18T16:21:55.266190+0000","last_fullsized":"2024-03-18T16:21:55.266190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:34:41.535042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148484+0000","last_change":"2024-03-18T16:21:21.399315+0000","last_active":"2024-03-18T16:21:55.148484+0000","last_peered":"2024-03-18T16:21:55.148484+0000","last_clean":"2024-03-18T16:21:55.148484+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:21:55.148484+0000","last_undegraded":"2024-03-18T16:21:55.148484+0000","last_fullsized":"2024-03-18T16:21:55.148484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:29.543106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147629+0000","last_change":"2024-03-18T16:21:29.065243+0000","last_active":"2024-03-18T16:21:55.147629+0000","last_peered":"2024-03-18T16:21:55.147629+0000","last_clean":"2024-03-18T16:21:55.147629+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:21:55.147629+0000","last_undegraded":"2024-03-18T16:21:55.147629+0000","last_fullsized":"2024-03-18T16:21:55.147629+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265799+0000","last_change":"2024-03-18T16:21:22.996080+0000","last_active":"2024-03-18T16:21:55.265799+0000","last_peered":"2024-03-18T16:21:55.265799+0000","last_clean":"2024-03-18T16:21:55.265799+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:21:55.265799+0000","last_undegraded":"2024-03-18T16:21:55.265799+0000","last_fullsized":"2024-03-18T16:21:55.265799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'10","reported_seq":40,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.832997+0000","last_change":"2024-03-18T16:21:21.404930+0000","last_active":"2024-03-18T16:22:05.832997+0000","last_peered":"2024-03-18T16:22:05.832997+0000","last_clean":"2024-03-18T16:22:05.832997+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:22:05.832997+0000","last_undegraded":"2024-03-18T16:22:05.832997+0000","last_fullsized":"2024-03-18T16:22:05.832997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:02.571905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278055+0000","last_change":"2024-03-18T16:21:29.079031+0000","last_active":"2024-03-18T16:21:55.278055+0000","last_peered":"2024-03-18T16:21:55.278055+0000","last_clean":"2024-03-18T16:21:55.278055+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:21:55.278055+0000","last_undegraded":"2024-03-18T16:21:55.278055+0000","last_fullsized":"2024-03-18T16:21:55.278055+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273519+0000","last_change":"2024-03-18T16:21:23.056832+0000","last_active":"2024-03-18T16:21:55.273519+0000","last_peered":"2024-03-18T16:21:55.273519+0000","last_clean":"2024-03-18T16:21:55.273519+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:21:55.273519+0000","last_undegraded":"2024-03-18T16:21:55.273519+0000","last_fullsized":"2024-03-18T16:21:55.273519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:56.124747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273585+0000","last_change":"2024-03-18T16:21:21.404392+0000","last_active":"2024-03-18T16:21:55.273585+0000","last_peered":"2024-03-18T16:21:55.273585+0000","last_clean":"2024-03-18T16:21:55.273585+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:21:55.273585+0000","last_undegraded":"2024-03-18T16:21:55.273585+0000","last_fullsized":"2024-03-18T16:21:55.273585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841565+0000","last_change":"2024-03-18T16:21:29.139956+0000","last_active":"2024-03-18T16:21:54.841565+0000","last_peered":"2024-03-18T16:21:54.841565+0000","last_clean":"2024-03-18T16:21:54.841565+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:21:54.841565+0000","last_undegraded":"2024-03-18T16:21:54.841565+0000","last_fullsized":"2024-03-18T16:21:54.841565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:52.356673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265953+0000","last_change":"2024-03-18T16:21:22.995275+0000","last_active":"2024-03-18T16:21:55.265953+0000","last_peered":"2024-03-18T16:21:55.265953+0000","last_clean":"2024-03-18T16:21:55.265953+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:21:55.265953+0000","last_undegraded":"2024-03-18T16:21:55.265953+0000","last_fullsized":"2024-03-18T16:21:55.265953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:48.024985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'3","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.057948+0000","last_change":"2024-03-18T16:21:21.412044+0000","last_active":"2024-03-18T16:22:06.057948+0000","last_peered":"2024-03-18T16:22:06.057948+0000","last_clean":"2024-03-18T16:22:06.057948+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:22:06.057948+0000","last_undegraded":"2024-03-18T16:22:06.057948+0000","last_fullsized":"2024-03-18T16:22:06.057948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.270056+0000","last_change":"2024-03-18T16:21:55.270056+0000","last_active":"2024-03-18T16:21:55.270056+0000","last_peered":"2024-03-18T16:21:55.270056+0000","last_clean":"2024-03-18T16:21:55.270056+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:21:55.270056+0000","last_undegraded":"2024-03-18T16:21:55.270056+0000","last_fullsized":"2024-03-18T16:21:55.270056+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:50.466826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277605+0000","last_change":"2024-03-18T16:21:23.056116+0000","last_active":"2024-03-18T16:21:55.277605+0000","last_peered":"2024-03-18T16:21:55.277605+0000","last_clean":"2024-03-18T16:21:55.277605+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:21:55.277605+0000","last_undegraded":"2024-03-18T16:21:55.277605+0000","last_fullsized":"2024-03-18T16:21:55.277605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:18.455315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.885447+0000","last_change":"2024-03-18T16:21:21.400378+0000","last_active":"2024-03-18T16:22:05.885447+0000","last_peered":"2024-03-18T16:22:05.885447+0000","last_clean":"2024-03-18T16:22:05.885447+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:22:05.885447+0000","last_undegraded":"2024-03-18T16:22:05.885447+0000","last_fullsized":"2024-03-18T16:22:05.885447+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064943+0000","last_change":"2024-03-18T16:21:29.141554+0000","last_active":"2024-03-18T16:21:55.064943+0000","last_peered":"2024-03-18T16:21:55.064943+0000","last_clean":"2024-03-18T16:21:55.064943+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:21:55.064943+0000","last_undegraded":"2024-03-18T16:21:55.064943+0000","last_fullsized":"2024-03-18T16:21:55.064943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:25.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266115+0000","last_change":"2024-03-18T16:21:22.996694+0000","last_active":"2024-03-18T16:21:55.266115+0000","last_peered":"2024-03-18T16:21:55.266115+0000","last_clean":"2024-03-18T16:21:55.266115+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:21:55.266115+0000","last_undegraded":"2024-03-18T16:21:55.266115+0000","last_fullsized":"2024-03-18T16:21:55.266115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:03.704090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'12","reported_seq":42,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.041654+0000","last_change":"2024-03-18T16:21:21.412149+0000","last_active":"2024-03-18T16:22:06.041654+0000","last_peered":"2024-03-18T16:22:06.041654+0000","last_clean":"2024-03-18T16:22:06.041654+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:22:06.041654+0000","last_undegraded":"2024-03-18T16:22:06.041654+0000","last_fullsized":"2024-03-18T16:22:06.041654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831282+0000","last_change":"2024-03-18T16:21:29.077350+0000","last_active":"2024-03-18T16:21:54.831282+0000","last_peered":"2024-03-18T16:21:54.831282+0000","last_clean":"2024-03-18T16:21:54.831282+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:21:54.831282+0000","last_undegraded":"2024-03-18T16:21:54.831282+0000","last_fullsized":"2024-03-18T16:21:54.831282+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148199+0000","last_change":"2024-03-18T16:21:22.992928+0000","last_active":"2024-03-18T16:21:55.148199+0000","last_peered":"2024-03-18T16:21:55.148199+0000","last_clean":"2024-03-18T16:21:55.148199+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:21:55.148199+0000","last_undegraded":"2024-03-18T16:21:55.148199+0000","last_fullsized":"2024-03-18T16:21:55.148199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:50.978267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834994+0000","last_change":"2024-03-18T16:21:21.403749+0000","last_active":"2024-03-18T16:21:54.834994+0000","last_peered":"2024-03-18T16:21:54.834994+0000","last_clean":"2024-03-18T16:21:54.834994+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:21:54.834994+0000","last_undegraded":"2024-03-18T16:21:54.834994+0000","last_fullsized":"2024-03-18T16:21:54.834994+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:29:30.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147816+0000","last_change":"2024-03-18T16:21:29.065378+0000","last_active":"2024-03-18T16:21:55.147816+0000","last_peered":"2024-03-18T16:21:55.147816+0000","last_clean":"2024-03-18T16:21:55.147816+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:21:55.147816+0000","last_undegraded":"2024-03-18T16:21:55.147816+0000","last_fullsized":"2024-03-18T16:21:55.147816+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:22:12.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278432+0000","last_change":"2024-03-18T16:21:23.054168+0000","last_active":"2024-03-18T16:21:55.278432+0000","last_peered":"2024-03-18T16:21:55.278432+0000","last_clean":"2024-03-18T16:21:55.278432+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:21:55.278432+0000","last_undegraded":"2024-03-18T16:21:55.278432+0000","last_fullsized":"2024-03-18T16:21:55.278432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.054769+0000","last_change":"2024-03-18T16:21:21.398307+0000","last_active":"2024-03-18T16:22:06.054769+0000","last_peered":"2024-03-18T16:22:06.054769+0000","last_clean":"2024-03-18T16:22:06.054769+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:22:06.054769+0000","last_undegraded":"2024-03-18T16:22:06.054769+0000","last_fullsized":"2024-03-18T16:22:06.054769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840228+0000","last_change":"2024-03-18T16:21:29.065855+0000","last_active":"2024-03-18T16:21:54.840228+0000","last_peered":"2024-03-18T16:21:54.840228+0000","last_clean":"2024-03-18T16:21:54.840228+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:21:54.840228+0000","last_undegraded":"2024-03-18T16:21:54.840228+0000","last_fullsized":"2024-03-18T16:21:54.840228+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:43.207302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064726+0000","last_change":"2024-03-18T16:21:22.993606+0000","last_active":"2024-03-18T16:21:55.064726+0000","last_peered":"2024-03-18T16:21:55.064726+0000","last_clean":"2024-03-18T16:21:55.064726+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:21:55.064726+0000","last_undegraded":"2024-03-18T16:21:55.064726+0000","last_fullsized":"2024-03-18T16:21:55.064726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:38:23.121609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'3","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.040632+0000","last_change":"2024-03-18T16:21:21.399588+0000","last_active":"2024-03-18T16:22:06.040632+0000","last_peered":"2024-03-18T16:22:06.040632+0000","last_clean":"2024-03-18T16:22:06.040632+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:22:06.040632+0000","last_undegraded":"2024-03-18T16:22:06.040632+0000","last_fullsized":"2024-03-18T16:22:06.040632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.843888+0000","last_change":"2024-03-18T16:21:29.144832+0000","last_active":"2024-03-18T16:21:54.843888+0000","last_peered":"2024-03-18T16:21:54.843888+0000","last_clean":"2024-03-18T16:21:54.843888+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:21:54.843888+0000","last_undegraded":"2024-03-18T16:21:54.843888+0000","last_fullsized":"2024-03-18T16:21:54.843888+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:10.103662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277724+0000","last_change":"2024-03-18T16:21:29.148215+0000","last_active":"2024-03-18T16:21:55.277724+0000","last_peered":"2024-03-18T16:21:55.277724+0000","last_clean":"2024-03-18T16:21:55.277724+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:21:55.277724+0000","last_undegraded":"2024-03-18T16:21:55.277724+0000","last_fullsized":"2024-03-18T16:21:55.277724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.829181+0000","last_change":"2024-03-18T16:21:22.980300+0000","last_active":"2024-03-18T16:21:54.829181+0000","last_peered":"2024-03-18T16:21:54.829181+0000","last_clean":"2024-03-18T16:21:54.829181+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:21:54.829181+0000","last_undegraded":"2024-03-18T16:21:54.829181+0000","last_fullsized":"2024-03-18T16:21:54.829181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064626+0000","last_change":"2024-03-18T16:21:21.329294+0000","last_active":"2024-03-18T16:21:55.064626+0000","last_peered":"2024-03-18T16:21:55.064626+0000","last_clean":"2024-03-18T16:21:55.064626+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:21:55.064626+0000","last_undegraded":"2024-03-18T16:21:55.064626+0000","last_fullsized":"2024-03-18T16:21:55.064626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273909+0000","last_change":"2024-03-18T16:21:29.076706+0000","last_active":"2024-03-18T16:21:55.273909+0000","last_peered":"2024-03-18T16:21:55.273909+0000","last_clean":"2024-03-18T16:21:55.273909+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:21:55.273909+0000","last_undegraded":"2024-03-18T16:21:55.273909+0000","last_fullsized":"2024-03-18T16:21:55.273909+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:57.880669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840470+0000","last_change":"2024-03-18T16:21:23.052989+0000","last_active":"2024-03-18T16:21:54.840470+0000","last_peered":"2024-03-18T16:21:54.840470+0000","last_clean":"2024-03-18T16:21:54.840470+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:21:54.840470+0000","last_undegraded":"2024-03-18T16:21:54.840470+0000","last_fullsized":"2024-03-18T16:21:54.840470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840520+0000","last_change":"2024-03-18T16:21:21.413020+0000","last_active":"2024-03-18T16:21:54.840520+0000","last_peered":"2024-03-18T16:21:54.840520+0000","last_clean":"2024-03-18T16:21:54.840520+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:21:54.840520+0000","last_undegraded":"2024-03-18T16:21:54.840520+0000","last_fullsized":"2024-03-18T16:21:54.840520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:49.237529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273671+0000","last_change":"2024-03-18T16:21:21.397354+0000","last_active":"2024-03-18T16:21:55.273671+0000","last_peered":"2024-03-18T16:21:55.273671+0000","last_clean":"2024-03-18T16:21:55.273671+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:21:55.273671+0000","last_undegraded":"2024-03-18T16:21:55.273671+0000","last_fullsized":"2024-03-18T16:21:55.273671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278026+0000","last_change":"2024-03-18T16:21:23.059441+0000","last_active":"2024-03-18T16:21:55.278026+0000","last_peered":"2024-03-18T16:21:55.278026+0000","last_clean":"2024-03-18T16:21:55.278026+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:21:55.278026+0000","last_undegraded":"2024-03-18T16:21:55.278026+0000","last_fullsized":"2024-03-18T16:21:55.278026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827518+0000","last_change":"2024-03-18T16:21:29.058800+0000","last_active":"2024-03-18T16:21:54.827518+0000","last_peered":"2024-03-18T16:21:54.827518+0000","last_clean":"2024-03-18T16:21:54.827518+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:21:54.827518+0000","last_undegraded":"2024-03-18T16:21:54.827518+0000","last_fullsized":"2024-03-18T16:21:54.827518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:20.279579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.074918+0000","last_change":"2024-03-18T16:21:21.412536+0000","last_active":"2024-03-18T16:22:06.074918+0000","last_peered":"2024-03-18T16:22:06.074918+0000","last_clean":"2024-03-18T16:22:06.074918+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:22:06.074918+0000","last_undegraded":"2024-03-18T16:22:06.074918+0000","last_fullsized":"2024-03-18T16:22:06.074918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148039+0000","last_change":"2024-03-18T16:21:22.974963+0000","last_active":"2024-03-18T16:21:55.148039+0000","last_peered":"2024-03-18T16:21:55.148039+0000","last_clean":"2024-03-18T16:21:55.148039+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:21:55.148039+0000","last_undegraded":"2024-03-18T16:21:55.148039+0000","last_fullsized":"2024-03-18T16:21:55.148039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:51.820583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.151933+0000","last_change":"2024-03-18T16:21:55.151933+0000","last_active":"2024-03-18T16:21:55.151933+0000","last_peered":"2024-03-18T16:21:55.151933+0000","last_clean":"2024-03-18T16:21:55.151933+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:21:55.151933+0000","last_undegraded":"2024-03-18T16:21:55.151933+0000","last_fullsized":"2024-03-18T16:21:55.151933+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:38.438575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273462+0000","last_change":"2024-03-18T16:21:23.056857+0000","last_active":"2024-03-18T16:21:55.273462+0000","last_peered":"2024-03-18T16:21:55.273462+0000","last_clean":"2024-03-18T16:21:55.273462+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:21:55.273462+0000","last_undegraded":"2024-03-18T16:21:55.273462+0000","last_fullsized":"2024-03-18T16:21:55.273462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:13.728335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827238+0000","last_change":"2024-03-18T16:21:22.949289+0000","last_active":"2024-03-18T16:21:54.827238+0000","last_peered":"2024-03-18T16:21:54.827238+0000","last_clean":"2024-03-18T16:21:54.827238+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:21:54.827238+0000","last_undegraded":"2024-03-18T16:21:54.827238+0000","last_fullsized":"2024-03-18T16:21:54.827238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:51:51.357680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.848377+0000","last_change":"2024-03-18T16:21:54.845858+0000","last_active":"2024-03-18T16:21:54.848377+0000","last_peered":"2024-03-18T16:21:54.848377+0000","last_clean":"2024-03-18T16:21:54.848377+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:21:54.848377+0000","last_undegraded":"2024-03-18T16:21:54.848377+0000","last_fullsized":"2024-03-18T16:21:54.848377+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277673+0000","last_change":"2024-03-18T16:21:23.059438+0000","last_active":"2024-03-18T16:21:55.277673+0000","last_peered":"2024-03-18T16:21:55.277673+0000","last_clean":"2024-03-18T16:21:55.277673+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:21:55.277673+0000","last_undegraded":"2024-03-18T16:21:55.277673+0000","last_fullsized":"2024-03-18T16:21:55.277673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841882+0000","last_change":"2024-03-18T16:21:22.849115+0000","last_active":"2024-03-18T16:21:54.841882+0000","last_peered":"2024-03-18T16:21:54.841882+0000","last_clean":"2024-03-18T16:21:54.841882+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:21:54.841882+0000","last_undegraded":"2024-03-18T16:21:54.841882+0000","last_fullsized":"2024-03-18T16:21:54.841882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:57.469261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834154+0000","last_change":"2024-03-18T16:21:22.980253+0000","last_active":"2024-03-18T16:21:54.834154+0000","last_peered":"2024-03-18T16:21:54.834154+0000","last_clean":"2024-03-18T16:21:54.834154+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:21:54.834154+0000","last_undegraded":"2024-03-18T16:21:54.834154+0000","last_fullsized":"2024-03-18T16:21:54.834154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273836+0000","last_change":"2024-03-18T16:21:23.042188+0000","last_active":"2024-03-18T16:21:55.273836+0000","last_peered":"2024-03-18T16:21:55.273836+0000","last_clean":"2024-03-18T16:21:55.273836+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:21:55.273836+0000","last_undegraded":"2024-03-18T16:21:55.273836+0000","last_fullsized":"2024-03-18T16:21:55.273836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841847+0000","last_change":"2024-03-18T16:21:22.851677+0000","last_active":"2024-03-18T16:21:54.841847+0000","last_peered":"2024-03-18T16:21:54.841847+0000","last_clean":"2024-03-18T16:21:54.841847+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:21:54.841847+0000","last_undegraded":"2024-03-18T16:21:54.841847+0000","last_fullsized":"2024-03-18T16:21:54.841847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064494+0000","last_change":"2024-03-18T16:21:22.947869+0000","last_active":"2024-03-18T16:21:55.064494+0000","last_peered":"2024-03-18T16:21:55.064494+0000","last_clean":"2024-03-18T16:21:55.064494+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:21:55.064494+0000","last_undegraded":"2024-03-18T16:21:55.064494+0000","last_fullsized":"2024-03-18T16:21:55.064494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840151+0000","last_change":"2024-03-18T16:21:22.851663+0000","last_active":"2024-03-18T16:21:54.840151+0000","last_peered":"2024-03-18T16:21:54.840151+0000","last_clean":"2024-03-18T16:21:54.840151+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:21:54.840151+0000","last_undegraded":"2024-03-18T16:21:54.840151+0000","last_fullsized":"2024-03-18T16:21:54.840151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.271992+0000","last_change":"2024-03-18T16:21:55.268883+0000","last_active":"2024-03-18T16:21:55.271992+0000","last_peered":"2024-03-18T16:21:55.271992+0000","last_clean":"2024-03-18T16:21:55.271992+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:21:55.271992+0000","last_undegraded":"2024-03-18T16:21:55.271992+0000","last_fullsized":"2024-03-18T16:21:55.271992+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277397+0000","last_change":"2024-03-18T16:21:23.059733+0000","last_active":"2024-03-18T16:21:55.277397+0000","last_peered":"2024-03-18T16:21:55.277397+0000","last_clean":"2024-03-18T16:21:55.277397+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:21:55.277397+0000","last_undegraded":"2024-03-18T16:21:55.277397+0000","last_fullsized":"2024-03-18T16:21:55.277397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273660+0000","last_change":"2024-03-18T16:21:23.042457+0000","last_active":"2024-03-18T16:21:55.273660+0000","last_peered":"2024-03-18T16:21:55.273660+0000","last_clean":"2024-03-18T16:21:55.273660+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:21:55.273660+0000","last_undegraded":"2024-03-18T16:21:55.273660+0000","last_fullsized":"2024-03-18T16:21:55.273660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:09.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.845542+0000","last_change":"2024-03-18T16:21:54.842423+0000","last_active":"2024-03-18T16:21:54.845542+0000","last_peered":"2024-03-18T16:21:54.845542+0000","last_clean":"2024-03-18T16:21:54.845542+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:21:54.845542+0000","last_undegraded":"2024-03-18T16:21:54.845542+0000","last_fullsized":"2024-03-18T16:21:54.845542+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:36.656058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.828079+0000","last_change":"2024-03-18T16:21:23.045061+0000","last_active":"2024-03-18T16:21:54.828079+0000","last_peered":"2024-03-18T16:21:54.828079+0000","last_clean":"2024-03-18T16:21:54.828079+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:21:54.828079+0000","last_undegraded":"2024-03-18T16:21:54.828079+0000","last_fullsized":"2024-03-18T16:21:54.828079+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:54.119675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065127+0000","last_change":"2024-03-18T16:21:22.993632+0000","last_active":"2024-03-18T16:21:55.065127+0000","last_peered":"2024-03-18T16:21:55.065127+0000","last_clean":"2024-03-18T16:21:55.065127+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:21:55.065127+0000","last_undegraded":"2024-03-18T16:21:55.065127+0000","last_fullsized":"2024-03-18T16:21:55.065127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:35:42.249037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064410+0000","last_change":"2024-03-18T16:21:23.056575+0000","last_active":"2024-03-18T16:21:55.064410+0000","last_peered":"2024-03-18T16:21:55.064410+0000","last_clean":"2024-03-18T16:21:55.064410+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:21:55.064410+0000","last_undegraded":"2024-03-18T16:21:55.064410+0000","last_fullsized":"2024-03-18T16:21:55.064410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831640+0000","last_change":"2024-03-18T16:21:22.973283+0000","last_active":"2024-03-18T16:21:54.831640+0000","last_peered":"2024-03-18T16:21:54.831640+0000","last_clean":"2024-03-18T16:21:54.831640+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:21:54.831640+0000","last_undegraded":"2024-03-18T16:21:54.831640+0000","last_fullsized":"2024-03-18T16:21:54.831640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277550+0000","last_change":"2024-03-18T16:21:23.057150+0000","last_active":"2024-03-18T16:21:55.277550+0000","last_peered":"2024-03-18T16:21:55.277550+0000","last_clean":"2024-03-18T16:21:55.277550+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:21:55.277550+0000","last_undegraded":"2024-03-18T16:21:55.277550+0000","last_fullsized":"2024-03-18T16:21:55.277550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841077+0000","last_change":"2024-03-18T16:21:23.047968+0000","last_active":"2024-03-18T16:21:54.841077+0000","last_peered":"2024-03-18T16:21:54.841077+0000","last_clean":"2024-03-18T16:21:54.841077+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:21:54.841077+0000","last_undegraded":"2024-03-18T16:21:54.841077+0000","last_fullsized":"2024-03-18T16:21:54.841077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148114+0000","last_change":"2024-03-18T16:21:23.041794+0000","last_active":"2024-03-18T16:21:55.148114+0000","last_peered":"2024-03-18T16:21:55.148114+0000","last_clean":"2024-03-18T16:21:55.148114+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:21:55.148114+0000","last_undegraded":"2024-03-18T16:21:55.148114+0000","last_fullsized":"2024-03-18T16:21:55.148114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:13.065494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273737+0000","last_change":"2024-03-18T16:21:23.036940+0000","last_active":"2024-03-18T16:21:55.273737+0000","last_peered":"2024-03-18T16:21:55.273737+0000","last_clean":"2024-03-18T16:21:55.273737+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:21:55.273737+0000","last_undegraded":"2024-03-18T16:21:55.273737+0000","last_fullsized":"2024-03-18T16:21:55.273737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:45:38.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273810+0000","last_change":"2024-03-18T16:21:23.042140+0000","last_active":"2024-03-18T16:21:55.273810+0000","last_peered":"2024-03-18T16:21:55.273810+0000","last_clean":"2024-03-18T16:21:55.273810+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:21:55.273810+0000","last_undegraded":"2024-03-18T16:21:55.273810+0000","last_fullsized":"2024-03-18T16:21:55.273810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266289+0000","last_change":"2024-03-18T16:21:22.995117+0000","last_active":"2024-03-18T16:21:55.266289+0000","last_peered":"2024-03-18T16:21:55.266289+0000","last_clean":"2024-03-18T16:21:55.266289+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:21:55.266289+0000","last_undegraded":"2024-03-18T16:21:55.266289+0000","last_fullsized":"2024-03-18T16:21:55.266289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:57.544738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.839997+0000","last_change":"2024-03-18T16:21:22.948123+0000","last_active":"2024-03-18T16:21:54.839997+0000","last_peered":"2024-03-18T16:21:54.839997+0000","last_clean":"2024-03-18T16:21:54.839997+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:21:54.839997+0000","last_undegraded":"2024-03-18T16:21:54.839997+0000","last_fullsized":"2024-03-18T16:21:54.839997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265696+0000","last_change":"2024-03-18T16:21:23.049877+0000","last_active":"2024-03-18T16:21:55.265696+0000","last_peered":"2024-03-18T16:21:55.265696+0000","last_clean":"2024-03-18T16:21:55.265696+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:21:55.265696+0000","last_undegraded":"2024-03-18T16:21:55.265696+0000","last_fullsized":"2024-03-18T16:21:55.265696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064503+0000","last_change":"2024-03-18T16:21:23.049483+0000","last_active":"2024-03-18T16:21:55.064503+0000","last_peered":"2024-03-18T16:21:55.064503+0000","last_clean":"2024-03-18T16:21:55.064503+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:21:55.064503+0000","last_undegraded":"2024-03-18T16:21:55.064503+0000","last_fullsized":"2024-03-18T16:21:55.064503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265858+0000","last_change":"2024-03-18T16:21:22.995726+0000","last_active":"2024-03-18T16:21:55.265858+0000","last_peered":"2024-03-18T16:21:55.265858+0000","last_clean":"2024-03-18T16:21:55.265858+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:21:55.265858+0000","last_undegraded":"2024-03-18T16:21:55.265858+0000","last_fullsized":"2024-03-18T16:21:55.265858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064410+0000","last_change":"2024-03-18T16:21:22.949640+0000","last_active":"2024-03-18T16:21:55.064410+0000","last_peered":"2024-03-18T16:21:55.064410+0000","last_clean":"2024-03-18T16:21:55.064410+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:21:55.064410+0000","last_undegraded":"2024-03-18T16:21:55.064410+0000","last_fullsized":"2024-03-18T16:21:55.064410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:21.734293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277697+0000","last_change":"2024-03-18T16:21:23.050203+0000","last_active":"2024-03-18T16:21:55.277697+0000","last_peered":"2024-03-18T16:21:55.277697+0000","last_clean":"2024-03-18T16:21:55.277697+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:21:55.277697+0000","last_undegraded":"2024-03-18T16:21:55.277697+0000","last_fullsized":"2024-03-18T16:21:55.277697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:21.982408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834586+0000","last_change":"2024-03-18T16:21:23.053003+0000","last_active":"2024-03-18T16:21:54.834586+0000","last_peered":"2024-03-18T16:21:54.834586+0000","last_clean":"2024-03-18T16:21:54.834586+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:21:54.834586+0000","last_undegraded":"2024-03-18T16:21:54.834586+0000","last_fullsized":"2024-03-18T16:21:54.834586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266295+0000","last_change":"2024-03-18T16:21:23.041863+0000","last_active":"2024-03-18T16:21:55.266295+0000","last_peered":"2024-03-18T16:21:55.266295+0000","last_clean":"2024-03-18T16:21:55.266295+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:21:55.266295+0000","last_undegraded":"2024-03-18T16:21:55.266295+0000","last_fullsized":"2024-03-18T16:21:55.266295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:26.190759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509456,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":393216,"data_stored":721186,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509457,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":129628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":133966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":130334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":389120,"data_stored":720530,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542160,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":129502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":323584,"data_stored":133434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":130264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:22:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T16:22:12.301 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T16:22:12.301 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-18T16:22:12.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-18T16:22:12.302 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-18T16:22:12.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.507+0000 7f9d11600640 1 Processor -- start 2024-03-18T16:22:12.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.507+0000 7f9d11600640 1 -- start start 2024-03-18T16:22:12.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.509+0000 7f9d11600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9d0c15ae70 0x7f9d0c0ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:12.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.509+0000 7f9d11600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9d0c0c9be0 0x7f9d0c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:12.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.509+0000 7f9d11600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d0c154ab0 con 0x7f9d0c0d05a0 2024-03-18T16:22:12.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.509+0000 7f9d11600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9d0c06d2f0 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.509+0000 7f9d11600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9d0c06d5d0 con 0x7f9d0c15ae70 2024-03-18T16:22:12.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.510+0000 7f9d0be00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9d0c15ae70 0x7f9d0c0ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:12.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.510+0000 7f9d0b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9d0c0c9be0 0x7f9d0c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:12.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.511+0000 7f9d10c00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f9d0c0d05a0 0x7f9d0c0cc4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58470/0 (socket says 172.21.15.102:58470) 2024-03-18T16:22:12.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.511+0000 7f9d10c00640 1 -- 172.21.15.102:0/251171573 learned_addr learned my addr 172.21.15.102:0/251171573 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:12.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.513+0000 7f9d0b400640 1 -- 172.21.15.102:0/251171573 >> v2:172.21.15.154:3300/0 conn(0x7f9d0c15ae70 msgr2=0x7f9d0c0ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:12.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.513+0000 7f9d0b400640 1 --2- 172.21.15.102:0/251171573 >> v2:172.21.15.154:3300/0 conn(0x7f9d0c15ae70 0x7f9d0c0ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.513+0000 7f9d0b400640 1 -- 172.21.15.102:0/251171573 >> v1:172.21.15.154:6789/0 conn(0x7f9d0c0d05a0 legacy=0x7f9d0c0cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:12.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.513+0000 7f9d0b400640 1 -- 172.21.15.102:0/251171573 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0c15b250 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.515+0000 7f9d0b400640 1 --2- 172.21.15.102:0/251171573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d0c0c9be0 0x7f9d0c0d0060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0cd5c0 tx=0x7f9cf802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acf25531ffc9ceb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:12.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.518+0000 7f9d0aa00640 1 -- 172.21.15.102:0/251171573 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8035020 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.520+0000 7f9d0aa00640 1 -- 172.21.15.102:0/251171573 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cf8031830 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.520+0000 7f9d0aa00640 1 -- 172.21.15.102:0/251171573 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8031c00 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.521+0000 7f9d11600640 1 -- 172.21.15.102:0/251171573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d0c0c9be0 msgr2=0x7f9d0c0d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:12.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.521+0000 7f9d11600640 1 --2- 172.21.15.102:0/251171573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d0c0c9be0 0x7f9d0c0d0060 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0cd5c0 tx=0x7f9cf802b8d0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.521+0000 7f9d11600640 1 -- 172.21.15.102:0/251171573 shutdown_connections 2024-03-18T16:22:12.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.521+0000 7f9d11600640 1 --2- 172.21.15.102:0/251171573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d0c0c9be0 0x7f9d0c0d0060 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.521+0000 7f9d11600640 1 --2- 172.21.15.102:0/251171573 >> v2:172.21.15.154:3300/0 conn(0x7f9d0c15ae70 0x7f9d0c0ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.521+0000 7f9d11600640 1 -- 172.21.15.102:0/251171573 >> 172.21.15.102:0/251171573 conn(0x7f9d0c0673a0 msgr2=0x7f9d0c0677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:12.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.521+0000 7f9d11600640 1 -- 172.21.15.102:0/251171573 shutdown_connections 2024-03-18T16:22:12.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.521+0000 7f9d11600640 1 -- 172.21.15.102:0/251171573 wait complete. 2024-03-18T16:22:12.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.524+0000 7f9d11600640 1 Processor -- start 2024-03-18T16:22:12.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.524+0000 7f9d11600640 1 -- start start 2024-03-18T16:22:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.526+0000 7f9d11600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0c0c9be0 0x7f9d0c152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.526+0000 7f9d11600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d0c0d05a0 0x7f9d0c0c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.526+0000 7f9d11600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d0c15ae70 0x7f9d0c0c6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.526+0000 7f9d11600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c06de70 con 0x7f9d0c0d05a0 2024-03-18T16:22:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.526+0000 7f9d11600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0c06db90 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.526+0000 7f9d11600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9d0c06e150 con 0x7f9d0c15ae70 2024-03-18T16:22:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.526+0000 7f9d0be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0c0c9be0 0x7f9d0c152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:12.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.526+0000 7f9d0b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d0c0d05a0 0x7f9d0c0c5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:12.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.526+0000 7f9d10c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d0c15ae70 0x7f9d0c0c6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:12.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.527+0000 7f9d0be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0c0c9be0 0x7f9d0c152250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47336/0 (socket says 172.21.15.102:47336) 2024-03-18T16:22:12.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.527+0000 7f9d0be00640 1 -- 172.21.15.102:0/1261202098 learned_addr learned my addr 172.21.15.102:0/1261202098 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:12.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.530+0000 7f9d0be00640 1 -- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d0c15ae70 msgr2=0x7f9d0c0c6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:12.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.530+0000 7f9d0be00640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d0c15ae70 0x7f9d0c0c6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.530+0000 7f9d0be00640 1 -- 172.21.15.102:0/1261202098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d0c0d05a0 msgr2=0x7f9d0c0c5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:12.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.530+0000 7f9d0be00640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d0c0d05a0 0x7f9d0c0c5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:12.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.530+0000 7f9d0be00640 1 -- 172.21.15.102:0/1261202098 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0c1568d0 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.533+0000 7f9d0be00640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0c0c9be0 0x7f9d0c152250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e9b0 tx=0x7f9cfc00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:12.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.537+0000 7f9d03e00640 1 -- 172.21.15.102:0/1261202098 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc017070 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.537+0000 7f9d03e00640 1 -- 172.21.15.102:0/1261202098 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cfc00c9a0 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.537+0000 7f9d03e00640 1 -- 172.21.15.102:0/1261202098 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc00cd90 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.538+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0c156bd0 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.538+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d0c157180 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.539+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d0c074fd0 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.558+0000 7f9d03e00640 1 -- 172.21.15.102:0/1261202098 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9cfc01a070 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.559+0000 7f9d03e00640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9cec06bc20 0x7f9cec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:12.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.559+0000 7f9d0b400640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9cec06bc20 0x7f9cec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:12.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.560+0000 7f9d0b400640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9cec06bc20 0x7f9cec06e0e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8036000 tx=0x7f9cf803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:12.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.561+0000 7f9d03e00640 1 -- 172.21.15.102:0/1261202098 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f9cfc057770 con 0x7f9d0c0c9be0 2024-03-18T16:22:12.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:12.819+0000 7f9d03e00640 1 -- 172.21.15.102:0/1261202098 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9d0c1568d0 con 0x7f9d0c0c9be0 2024-03-18T16:22:13.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.090+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f9d0c0b5a90 con 0x7f9d0c0c9be0 2024-03-18T16:22:13.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.227+0000 7f9d03e00640 1 -- 172.21.15.102:0/1261202098 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172586 (secure 0 0 0) 0x7f9cfc0596d0 con 0x7f9d0c0c9be0 2024-03-18T16:22:13.298 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9cec06bc20 msgr2=0x7f9cec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9cec06bc20 0x7f9cec06e0e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9cf8036000 tx=0x7f9cf803c040 comp rx=0 tx=0).stop 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0c0c9be0 msgr2=0x7f9d0c152250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0c0c9be0 0x7f9d0c152250 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e9b0 tx=0x7f9cfc00ee70 comp rx=0 tx=0).stop 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 shutdown_connections 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9cec06bc20 0x7f9cec06e0e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d0c15ae70 0x7f9d0c0c6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d0c0d05a0 0x7f9d0c0c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 --2- 172.21.15.102:0/1261202098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d0c0c9be0 0x7f9d0c152250 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 >> 172.21.15.102:0/1261202098 conn(0x7f9d0c0673a0 msgr2=0x7f9d0c153db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 shutdown_connections 2024-03-18T16:22:13.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.232+0000 7f9d11600640 1 -- 172.21.15.102:0/1261202098 wait complete. 2024-03-18T16:22:13.301 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":2819855665},{"type":"v1","addr":"172.21.15.154:6801","nonce":2819855665}]},"active_addr":"172.21.15.154:6801/2819855665","active_change":"2024-03-18T16:20:52.440334+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":2397945274}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":3355454093}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":2881010551}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":2990855632}]}]} 2024-03-18T16:22:13.304 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-18T16:22:13.304 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-18T16:22:13.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:22:13.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.514+0000 7fbdb8600640 1 Processor -- start 2024-03-18T16:22:13.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.515+0000 7fbdb8600640 1 -- start start 2024-03-18T16:22:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.517+0000 7fbdb8600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb00c8740 0x7fbdb0072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.517+0000 7fbdb8600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbdb00c8b20 0x7fbdb015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.517+0000 7fbdb8600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbdb0158760 0x7fbdb01572f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.517+0000 7fbdb8600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbdb015a860 con 0x7fbdb00c8b20 2024-03-18T16:22:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.517+0000 7fbdb8600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbdb0072840 con 0x7fbdb0158760 2024-03-18T16:22:13.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.517+0000 7fbdb8600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbdb015b7f0 con 0x7fbdb00c8740 2024-03-18T16:22:13.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.540+0000 7fbdb5c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb00c8740 0x7fbdb0072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:13.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.541+0000 7fbdb5c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb00c8740 0x7fbdb0072300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34030/0 (socket says 172.21.15.102:34030) 2024-03-18T16:22:13.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.541+0000 7fbdb5c00640 1 -- 172.21.15.102:0/1846055415 learned_addr learned my addr 172.21.15.102:0/1846055415 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:13.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.543+0000 7fbdb5c00640 1 -- 172.21.15.102:0/1846055415 >> v2:172.21.15.154:3300/0 conn(0x7fbdb0158760 msgr2=0x7fbdb01572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:13.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.543+0000 7fbdb5c00640 1 --2- 172.21.15.102:0/1846055415 >> v2:172.21.15.154:3300/0 conn(0x7fbdb0158760 0x7fbdb01572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.543+0000 7fbdb5c00640 1 -- 172.21.15.102:0/1846055415 >> v2:172.21.15.102:3300/0 conn(0x7fbdb00c8b20 msgr2=0x7fbdb015b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:13.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.543+0000 7fbdb5c00640 1 --2- 172.21.15.102:0/1846055415 >> v2:172.21.15.102:3300/0 conn(0x7fbdb00c8b20 0x7fbdb015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.543+0000 7fbdb5c00640 1 -- 172.21.15.102:0/1846055415 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb0157830 con 0x7fbdb00c8740 2024-03-18T16:22:13.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.546+0000 7fbdb5c00640 1 --2- 172.21.15.102:0/1846055415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb00c8740 0x7fbdb0072300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00b1f60 tx=0x7fbda402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf0953127e374bfe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.550+0000 7fbdafe00640 1 -- 172.21.15.102:0/1846055415 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda4035020 con 0x7fbdb00c8740 2024-03-18T16:22:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.550+0000 7fbdafe00640 1 -- 172.21.15.102:0/1846055415 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda4031870 con 0x7fbdb00c8740 2024-03-18T16:22:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.550+0000 7fbdafe00640 1 -- 172.21.15.102:0/1846055415 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda4031c40 con 0x7fbdb00c8740 2024-03-18T16:22:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.551+0000 7fbdb8600640 1 -- 172.21.15.102:0/1846055415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb00c8740 msgr2=0x7fbdb0072300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.551+0000 7fbdb8600640 1 --2- 172.21.15.102:0/1846055415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb00c8740 0x7fbdb0072300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00b1f60 tx=0x7fbda402b910 comp rx=0 tx=0).stop 2024-03-18T16:22:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.551+0000 7fbdb8600640 1 -- 172.21.15.102:0/1846055415 shutdown_connections 2024-03-18T16:22:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.551+0000 7fbdb8600640 1 --2- 172.21.15.102:0/1846055415 >> v2:172.21.15.154:3300/0 conn(0x7fbdb0158760 0x7fbdb01572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.551+0000 7fbdb8600640 1 --2- 172.21.15.102:0/1846055415 >> v2:172.21.15.102:3300/0 conn(0x7fbdb00c8b20 0x7fbdb015b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.551+0000 7fbdb8600640 1 --2- 172.21.15.102:0/1846055415 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb00c8740 0x7fbdb0072300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.551+0000 7fbdb8600640 1 -- 172.21.15.102:0/1846055415 >> 172.21.15.102:0/1846055415 conn(0x7fbdb00700d0 msgr2=0x7fbdb006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.551+0000 7fbdb8600640 1 -- 172.21.15.102:0/1846055415 shutdown_connections 2024-03-18T16:22:13.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.551+0000 7fbdb8600640 1 -- 172.21.15.102:0/1846055415 wait complete. 2024-03-18T16:22:13.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.554+0000 7fbdb8600640 1 Processor -- start 2024-03-18T16:22:13.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.554+0000 7fbdb8600640 1 -- start start 2024-03-18T16:22:13.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.555+0000 7fbdb8600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdb00c8740 0x7fbdb00ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:13.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.555+0000 7fbdb8600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb00c8b20 0x7fbdb014d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:13.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.555+0000 7fbdb8600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb0158760 0x7fbdb014f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:13.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.556+0000 7fbdb8600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb0158080 con 0x7fbdb00c8740 2024-03-18T16:22:13.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.556+0000 7fbdb8600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbdb0157da0 con 0x7fbdb00c8b20 2024-03-18T16:22:13.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.556+0000 7fbdb8600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbdb015aab0 con 0x7fbdb0158760 2024-03-18T16:22:13.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.556+0000 7fbdb5200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb00c8b20 0x7fbdb014d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:13.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.556+0000 7fbdb6600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb0158760 0x7fbdb014f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.556+0000 7fbdb5200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb00c8b20 0x7fbdb014d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47354/0 (socket says 172.21.15.102:47354) 2024-03-18T16:22:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.556+0000 7fbdb5200640 1 -- 172.21.15.102:0/803469864 learned_addr learned my addr 172.21.15.102:0/803469864 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:13.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.560+0000 7fbdb5200640 1 -- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb0158760 msgr2=0x7fbdb014f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:13.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.560+0000 7fbdb5200640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb0158760 0x7fbdb014f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.560+0000 7fbdb5200640 1 -- 172.21.15.102:0/803469864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdb00c8740 msgr2=0x7fbdb00ca710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:13.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.560+0000 7fbdb5200640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdb00c8740 0x7fbdb00ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:13.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.560+0000 7fbdb5200640 1 -- 172.21.15.102:0/803469864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdb01eff00 con 0x7fbdb00c8b20 2024-03-18T16:22:13.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.560+0000 7fbdb6600640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb0158760 0x7fbdb014f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:22:13.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.565+0000 7fbdb5200640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb00c8b20 0x7fbdb014d490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbda000b410 tx=0x7fbda000b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:13.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.569+0000 7fbdad600640 1 -- 172.21.15.102:0/803469864 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda0015020 con 0x7fbdb00c8b20 2024-03-18T16:22:13.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.569+0000 7fbdad600640 1 -- 172.21.15.102:0/803469864 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda0011850 con 0x7fbdb00c8b20 2024-03-18T16:22:13.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.569+0000 7fbdad600640 1 -- 172.21.15.102:0/803469864 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda0011c40 con 0x7fbdb00c8b20 2024-03-18T16:22:13.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.569+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdb014e6d0 con 0x7fbdb00c8b20 2024-03-18T16:22:13.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.570+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdb014ec90 con 0x7fbdb00c8b20 2024-03-18T16:22:13.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.571+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdb0074fd0 con 0x7fbdb00c8b20 2024-03-18T16:22:13.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.590+0000 7fbdad600640 1 -- 172.21.15.102:0/803469864 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbda0018020 con 0x7fbdb00c8b20 2024-03-18T16:22:13.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.591+0000 7fbdad600640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd9406bc50 0x7fbd9406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:13.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.591+0000 7fbdb5c00640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd9406bc50 0x7fbd9406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:13.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.592+0000 7fbdb5c00640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd9406bc50 0x7fbd9406e110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00b1f60 tx=0x7fbda4030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:13.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.594+0000 7fbdad600640 1 -- 172.21.15.102:0/803469864 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fbda0011de0 con 0x7fbdb00c8b20 2024-03-18T16:22:13.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:13.845+0000 7fbdad600640 1 -- 172.21.15.102:0/803469864 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbda001f020 con 0x7fbdb00c8b20 2024-03-18T16:22:14.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.061+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbdb014e210 con 0x7fbdb00c8b20 2024-03-18T16:22:14.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.101+0000 7fbdad600640 1 -- 172.21.15.102:0/803469864 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18907 (secure 0 0 0) 0x7fbda0059aa0 con 0x7fbdb00c8b20 2024-03-18T16:22:14.103 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:14.103 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":29,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:54.749120+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:22:14.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd9406bc50 msgr2=0x7fbd9406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:14.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd9406bc50 0x7fbd9406e110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00b1f60 tx=0x7fbda4030000 comp rx=0 tx=0).stop 2024-03-18T16:22:14.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb00c8b20 msgr2=0x7fbdb014d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb00c8b20 0x7fbdb014d490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbda000b410 tx=0x7fbda000b8d0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 shutdown_connections 2024-03-18T16:22:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd9406bc50 0x7fbd9406e110 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbdb0158760 0x7fbdb014f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdb00c8b20 0x7fbdb014d490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 --2- 172.21.15.102:0/803469864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdb00c8740 0x7fbdb00ca710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 >> 172.21.15.102:0/803469864 conn(0x7fbdb00700d0 msgr2=0x7fbdb00c4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.103+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 shutdown_connections 2024-03-18T16:22:14.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.104+0000 7fbdb8600640 1 -- 172.21.15.102:0/803469864 wait complete. 2024-03-18T16:22:14.129 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-18T16:22:14.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:22:14.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.456+0000 7f630b200640 1 Processor -- start 2024-03-18T16:22:14.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.456+0000 7f630b200640 1 -- start start 2024-03-18T16:22:14.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.457+0000 7f630b200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f63040c9be0 0x7f63040d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.457+0000 7f630b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f63040d05a0 0x7f63040cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.457+0000 7f630b200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6304154ab0 con 0x7f630415ae70 2024-03-18T16:22:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.457+0000 7f630b200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f63040caba0 con 0x7f63040d05a0 2024-03-18T16:22:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.457+0000 7f630b200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f630406d2f0 con 0x7f63040c9be0 2024-03-18T16:22:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.458+0000 7f6309e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f630415ae70 0x7f63040ca490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58474/0 (socket says 172.21.15.102:58474) 2024-03-18T16:22:14.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.459+0000 7f6309e00640 1 -- 172.21.15.102:0/2238687396 learned_addr learned my addr 172.21.15.102:0/2238687396 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:14.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.459+0000 7f6309400640 1 --2- 172.21.15.102:0/2238687396 >> v2:172.21.15.154:3300/0 conn(0x7f63040c9be0 0x7f63040d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:14.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.462+0000 7f630a800640 1 --2- 172.21.15.102:0/2238687396 >> v2:172.21.15.102:3300/0 conn(0x7f63040d05a0 0x7f63040cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.462+0000 7f6309400640 1 -- 172.21.15.102:0/2238687396 >> v2:172.21.15.102:3300/0 conn(0x7f63040d05a0 msgr2=0x7f63040cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.462+0000 7f6309400640 1 --2- 172.21.15.102:0/2238687396 >> v2:172.21.15.102:3300/0 conn(0x7f63040d05a0 0x7f63040cc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.462+0000 7f6309400640 1 -- 172.21.15.102:0/2238687396 >> v1:172.21.15.154:6789/0 conn(0x7f630415ae70 legacy=0x7f63040ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:14.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.462+0000 7f6309400640 1 -- 172.21.15.102:0/2238687396 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630415b250 con 0x7f63040c9be0 2024-03-18T16:22:14.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.468+0000 7f6309400640 1 --2- 172.21.15.102:0/2238687396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040c9be0 0x7f63040d0060 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6304157f40 tx=0x7f62f402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d94a171bb9577f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:14.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.475+0000 7f6308a00640 1 -- 172.21.15.102:0/2238687396 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f4035020 con 0x7f63040c9be0 2024-03-18T16:22:14.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.475+0000 7f6308a00640 1 -- 172.21.15.102:0/2238687396 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62f4031830 con 0x7f63040c9be0 2024-03-18T16:22:14.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.475+0000 7f6308a00640 1 -- 172.21.15.102:0/2238687396 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f4031c00 con 0x7f63040c9be0 2024-03-18T16:22:14.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.476+0000 7f630b200640 1 -- 172.21.15.102:0/2238687396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040c9be0 msgr2=0x7f63040d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:14.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.476+0000 7f630b200640 1 --2- 172.21.15.102:0/2238687396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040c9be0 0x7f63040d0060 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6304157f40 tx=0x7f62f402b8d0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.476+0000 7f630b200640 1 -- 172.21.15.102:0/2238687396 shutdown_connections 2024-03-18T16:22:14.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.476+0000 7f630b200640 1 --2- 172.21.15.102:0/2238687396 >> v2:172.21.15.102:3300/0 conn(0x7f63040d05a0 0x7f63040cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.476+0000 7f630b200640 1 --2- 172.21.15.102:0/2238687396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040c9be0 0x7f63040d0060 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.476+0000 7f630b200640 1 -- 172.21.15.102:0/2238687396 >> 172.21.15.102:0/2238687396 conn(0x7f63040673a0 msgr2=0x7f63040677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:14.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.476+0000 7f630b200640 1 -- 172.21.15.102:0/2238687396 shutdown_connections 2024-03-18T16:22:14.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.476+0000 7f630b200640 1 -- 172.21.15.102:0/2238687396 wait complete. 2024-03-18T16:22:14.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.478+0000 7f630b200640 1 Processor -- start 2024-03-18T16:22:14.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.479+0000 7f630b200640 1 -- start start 2024-03-18T16:22:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.480+0000 7f630b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63040c9be0 0x7f63040c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.480+0000 7f630b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040d05a0 0x7f6304153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.480+0000 7f630b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f630415ae70 0x7f63041543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.480+0000 7f630b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f630406db90 con 0x7f63040c9be0 2024-03-18T16:22:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.480+0000 7f630b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f630406d8b0 con 0x7f63040d05a0 2024-03-18T16:22:14.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.480+0000 7f630b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f630406de70 con 0x7f630415ae70 2024-03-18T16:22:14.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.481+0000 7f630a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f630415ae70 0x7f63041543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:14.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.482+0000 7f630a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f630415ae70 0x7f63041543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34054/0 (socket says 172.21.15.102:34054) 2024-03-18T16:22:14.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.482+0000 7f630a800640 1 -- 172.21.15.102:0/1320504894 learned_addr learned my addr 172.21.15.102:0/1320504894 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:14.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.485+0000 7f6309400640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040d05a0 0x7f6304153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.487+0000 7f6309e00640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63040c9be0 0x7f63040c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.488+0000 7f630a800640 1 -- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040d05a0 msgr2=0x7f6304153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.488+0000 7f630a800640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040d05a0 0x7f6304153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.488+0000 7f630a800640 1 -- 172.21.15.102:0/1320504894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63040c9be0 msgr2=0x7f63040c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.488+0000 7f630a800640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63040c9be0 0x7f63040c5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.488+0000 7f630a800640 1 -- 172.21.15.102:0/1320504894 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63040c4660 con 0x7f630415ae70 2024-03-18T16:22:14.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.488+0000 7f6309400640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040d05a0 0x7f6304153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:22:14.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.492+0000 7f630a800640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f630415ae70 0x7f63041543a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f630000cc30 tx=0x7f630000a5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.545+0000 7f62fe000640 1 -- 172.21.15.102:0/1320504894 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630000ce40 con 0x7f630415ae70 2024-03-18T16:22:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.545+0000 7f62fe000640 1 -- 172.21.15.102:0/1320504894 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6300003990 con 0x7f630415ae70 2024-03-18T16:22:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.545+0000 7f62fe000640 1 -- 172.21.15.102:0/1320504894 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300003d40 con 0x7f630415ae70 2024-03-18T16:22:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.545+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63040c6320 con 0x7f630415ae70 2024-03-18T16:22:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.545+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63040c6590 con 0x7f630415ae70 2024-03-18T16:22:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.547+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6304074fd0 con 0x7f630415ae70 2024-03-18T16:22:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.577+0000 7f62fe000640 1 -- 172.21.15.102:0/1320504894 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6300003ee0 con 0x7f630415ae70 2024-03-18T16:22:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.578+0000 7f62fe000640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62e806bc20 0x7f62e806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:14.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.579+0000 7f6309e00640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62e806bc20 0x7f62e806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:14.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.580+0000 7f6309e00640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62e806bc20 0x7f62e806e0e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f62f8000f60 tx=0x7f62f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:14.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.828+0000 7f62fe000640 1 -- 172.21.15.102:0/1320504894 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f63000577e0 con 0x7f630415ae70 2024-03-18T16:22:14.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:14.846+0000 7f62fe000640 1 -- 172.21.15.102:0/1320504894 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6300014030 con 0x7f630415ae70 2024-03-18T16:22:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.083+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6304157e40 con 0x7f630415ae70 2024-03-18T16:22:15.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.129+0000 7f62fe000640 1 -- 172.21.15.102:0/1320504894 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18907 (secure 0 0 0) 0x7f6300059e40 con 0x7f630415ae70 2024-03-18T16:22:15.132 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:15.132 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":29,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:54.749120+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.132+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62e806bc20 msgr2=0x7f62e806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.132+0000 7f630b200640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62e806bc20 0x7f62e806e0e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f62f8000f60 tx=0x7f62f8009040 comp rx=0 tx=0).stop 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f630415ae70 msgr2=0x7f63041543a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f630415ae70 0x7f63041543a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f630000cc30 tx=0x7f630000a5c0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 shutdown_connections 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62e806bc20 0x7f62e806e0e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f630415ae70 0x7f63041543a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63040d05a0 0x7f6304153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 --2- 172.21.15.102:0/1320504894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63040c9be0 0x7f63040c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 >> 172.21.15.102:0/1320504894 conn(0x7f63040673a0 msgr2=0x7f63040c6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 shutdown_connections 2024-03-18T16:22:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.133+0000 7f630b200640 1 -- 172.21.15.102:0/1320504894 wait complete. 2024-03-18T16:22:15.152 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-18T16:22:15.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-18T16:22:15.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-18T16:22:15.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-18T16:22:15.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-18T16:22:15.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-18T16:22:15.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-18T16:22:15.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-18T16:22:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.502+0000 7fdbb4e00640 1 Processor -- start 2024-03-18T16:22:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.502+0000 7fdbb4e00640 1 -- start start 2024-03-18T16:22:15.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.505+0000 7fdbb4e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdbb00ccd20 0x7fdbb015aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.505+0000 7fdbb4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb015b000 0x7fdbb00cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.505+0000 7fdbb4e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb00ce0d0 con 0x7fdbb0154960 2024-03-18T16:22:15.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.506+0000 7fdbb4e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdbb015a630 con 0x7fdbb00ccd20 2024-03-18T16:22:15.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.506+0000 7fdbb4e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb00ce3b0 con 0x7fdbb015b000 2024-03-18T16:22:15.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.506+0000 7fdbafe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb015b000 0x7fdbb00cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.506+0000 7fdbaea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdbb00ccd20 0x7fdbb015aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.506+0000 7fdbafe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb015b000 0x7fdbb00cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34070/0 (socket says 172.21.15.102:34070) 2024-03-18T16:22:15.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.506+0000 7fdbafe00640 1 -- 172.21.15.102:0/4007111170 learned_addr learned my addr 172.21.15.102:0/4007111170 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.506+0000 7fdbaf400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fdbb0154960 0x7fdbb00ca510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58478/0 (socket says 172.21.15.102:58478) 2024-03-18T16:22:15.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.519+0000 7fdbafe00640 1 -- 172.21.15.102:0/4007111170 >> v2:172.21.15.154:3300/0 conn(0x7fdbb00ccd20 msgr2=0x7fdbb015aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.519+0000 7fdbafe00640 1 --2- 172.21.15.102:0/4007111170 >> v2:172.21.15.154:3300/0 conn(0x7fdbb00ccd20 0x7fdbb015aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.522+0000 7fdbafe00640 1 -- 172.21.15.102:0/4007111170 >> v1:172.21.15.154:6789/0 conn(0x7fdbb0154960 legacy=0x7fdbb00ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.522+0000 7fdbafe00640 1 -- 172.21.15.102:0/4007111170 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb00ce690 con 0x7fdbb015b000 2024-03-18T16:22:15.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.522+0000 7fdbaea00640 1 --2- 172.21.15.102:0/4007111170 >> v2:172.21.15.154:3300/0 conn(0x7fdbb00ccd20 0x7fdbb015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:22:15.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.532+0000 7fdbafe00640 1 --2- 172.21.15.102:0/4007111170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb015b000 0x7fdbb00cec50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009870 tx=0x7fdb9c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=146126671abc2554 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.537+0000 7fdbae000640 1 -- 172.21.15.102:0/4007111170 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c02f830 con 0x7fdbb015b000 2024-03-18T16:22:15.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.541+0000 7fdbae000640 1 -- 172.21.15.102:0/4007111170 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9c02f9d0 con 0x7fdbb015b000 2024-03-18T16:22:15.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.541+0000 7fdbae000640 1 -- 172.21.15.102:0/4007111170 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c02fda0 con 0x7fdbb015b000 2024-03-18T16:22:15.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.546+0000 7fdbb4e00640 1 -- 172.21.15.102:0/4007111170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb015b000 msgr2=0x7fdbb00cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.546+0000 7fdbb4e00640 1 --2- 172.21.15.102:0/4007111170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb015b000 0x7fdbb00cec50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009870 tx=0x7fdb9c02ba10 comp rx=0 tx=0).stop 2024-03-18T16:22:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.548+0000 7fdbb4e00640 1 -- 172.21.15.102:0/4007111170 shutdown_connections 2024-03-18T16:22:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.548+0000 7fdbb4e00640 1 --2- 172.21.15.102:0/4007111170 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb015b000 0x7fdbb00cec50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.548+0000 7fdbb4e00640 1 --2- 172.21.15.102:0/4007111170 >> v2:172.21.15.154:3300/0 conn(0x7fdbb00ccd20 0x7fdbb015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.548+0000 7fdbb4e00640 1 -- 172.21.15.102:0/4007111170 >> 172.21.15.102:0/4007111170 conn(0x7fdbb00b16a0 msgr2=0x7fdbb00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.551+0000 7fdbb4e00640 1 -- 172.21.15.102:0/4007111170 shutdown_connections 2024-03-18T16:22:15.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.551+0000 7fdbb4e00640 1 -- 172.21.15.102:0/4007111170 wait complete. 2024-03-18T16:22:15.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.553+0000 7fdbb4e00640 1 Processor -- start 2024-03-18T16:22:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.553+0000 7fdbb4e00640 1 -- start start 2024-03-18T16:22:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.556+0000 7fdbb4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb00ccd20 0x7fdbb0156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.556+0000 7fdbb4e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb0154960 0x7fdbb014e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.561+0000 7fdbb4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb015b000 0x7fdbb0150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.561+0000 7fdbb4e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb00cf190 con 0x7fdbb015b000 2024-03-18T16:22:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.561+0000 7fdbb4e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb00cac20 con 0x7fdbb0154960 2024-03-18T16:22:15.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.561+0000 7fdbb4e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb00cf310 con 0x7fdbb00ccd20 2024-03-18T16:22:15.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.573+0000 7f97e7200640 1 Processor -- start 2024-03-18T16:22:15.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.573+0000 7f97e7200640 1 -- start start 2024-03-18T16:22:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.575+0000 7f97e7200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f97e80c8f50 0x7f97e80c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.575+0000 7f97e7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e8150670 0x7f97e80c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.575+0000 7f97e7200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97e806f6c0 0x7f97e806fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.575+0000 7f97e7200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f97e80c5e70 con 0x7f97e80c8f50 2024-03-18T16:22:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.575+0000 7f97e7200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f97e80c4ec0 con 0x7f97e806f6c0 2024-03-18T16:22:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.575+0000 7f97e7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f97e80c7450 con 0x7f97e8150670 2024-03-18T16:22:15.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.582+0000 7f31cf200640 1 Processor -- start 2024-03-18T16:22:15.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.583+0000 7ff1fe200640 1 Processor -- start 2024-03-18T16:22:15.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.583+0000 7ff1fe200640 1 -- start start 2024-03-18T16:22:15.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.586+0000 7ff1fe200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00f6ae0 0x7ff1f004b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.587+0000 7ff1fe200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1f0006990 con 0x7ff1f00fc980 2024-03-18T16:22:15.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.587+0000 7ff1fe200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1f0006c70 con 0x7ff1f004b5d0 2024-03-18T16:22:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.587+0000 7ff1fe200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f00075a0 con 0x7ff1f00f6ae0 2024-03-18T16:22:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.582+0000 7f31cf200640 1 -- start start 2024-03-18T16:22:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.584+0000 7f31cf200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d00c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.584+0000 7f31cf200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f31d00c7b70 0x7f31d006f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.584+0000 7f31cf200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31d00c7730 con 0x7f31d00c7f90 2024-03-18T16:22:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.584+0000 7f31cf200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f31d00c5e70 con 0x7f31d00c7b70 2024-03-18T16:22:15.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.584+0000 7f31cf200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f31d00c4ec0 con 0x7f31d01508e0 2024-03-18T16:22:15.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.595+0000 7fdbaf400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb00ccd20 0x7fdbb0156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.597+0000 7ff1f7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00f6ae0 0x7ff1f004b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.595+0000 7f97e5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e8150670 0x7f97e80c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.597+0000 7ff1f7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00f6ae0 0x7ff1f004b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34090/0 (socket says 172.21.15.102:34090) 2024-03-18T16:22:15.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.597+0000 7ff1f7e00640 1 -- 172.21.15.102:0/4136220899 learned_addr learned my addr 172.21.15.102:0/4136220899 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.597+0000 7f97e5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e8150670 0x7f97e80c7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34088/0 (socket says 172.21.15.102:34088) 2024-03-18T16:22:15.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.597+0000 7f97e5400640 1 -- 172.21.15.102:0/1470273909 learned_addr learned my addr 172.21.15.102:0/1470273909 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.598+0000 7fdbaf400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb00ccd20 0x7fdbb0156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34080/0 (socket says 172.21.15.102:34080) 2024-03-18T16:22:15.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.598+0000 7fdbaf400640 1 -- 172.21.15.102:0/3413385339 learned_addr learned my addr 172.21.15.102:0/3413385339 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.599+0000 7f97e6800640 1 --2- 172.21.15.102:0/1470273909 >> v2:172.21.15.154:3300/0 conn(0x7f97e806f6c0 0x7f97e806fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.601+0000 7f97e5400640 1 -- 172.21.15.102:0/1470273909 >> v2:172.21.15.154:3300/0 conn(0x7f97e806f6c0 msgr2=0x7f97e806fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.601+0000 7f97e5400640 1 --2- 172.21.15.102:0/1470273909 >> v2:172.21.15.154:3300/0 conn(0x7f97e806f6c0 0x7f97e806fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.599+0000 7fdbaea00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb0154960 0x7fdbb014e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.601+0000 7f97e5400640 1 -- 172.21.15.102:0/1470273909 >> v2:172.21.15.102:3300/0 conn(0x7f97e80c8f50 msgr2=0x7f97e80c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.602+0000 7ff1f7e00640 1 -- 172.21.15.102:0/4136220899 >> v1:172.21.15.154:6789/0 conn(0x7ff1f004b5d0 legacy=0x7ff1f0045220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.602+0000 7ff1f7e00640 1 -- 172.21.15.102:0/4136220899 >> v1:172.21.15.102:6789/0 conn(0x7ff1f00fc980 legacy=0x7ff1f0005e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.601+0000 7f97e6800640 1 --2- 172.21.15.102:0/1470273909 >> v2:172.21.15.154:3300/0 conn(0x7f97e806f6c0 0x7f97e806fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:22:15.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.602+0000 7fdbafe00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb015b000 0x7fdbb0150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.602+0000 7f97e5400640 1 --2- 172.21.15.102:0/1470273909 >> v2:172.21.15.102:3300/0 conn(0x7f97e80c8f50 0x7f97e80c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.602+0000 7f97e5400640 1 -- 172.21.15.102:0/1470273909 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e80c4590 con 0x7f97e8150670 2024-03-18T16:22:15.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.604+0000 7f31cde00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d00c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.604+0000 7f31cde00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d00c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34092/0 (socket says 172.21.15.102:34092) 2024-03-18T16:22:15.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.604+0000 7f31cde00640 1 -- 172.21.15.102:0/740497510 learned_addr learned my addr 172.21.15.102:0/740497510 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.605+0000 7f31cd400640 1 --2- 172.21.15.102:0/740497510 >> v2:172.21.15.154:3300/0 conn(0x7f31d00c7b70 0x7f31d006f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.602+0000 7ff1f7e00640 1 -- 172.21.15.102:0/4136220899 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f0006f50 con 0x7ff1f00f6ae0 2024-03-18T16:22:15.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.605+0000 7fdbaea00640 1 -- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb00ccd20 msgr2=0x7fdbb0156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.605+0000 7fdbaea00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb00ccd20 0x7fdbb0156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.605+0000 7fdbaea00640 1 -- 172.21.15.102:0/3413385339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb015b000 msgr2=0x7fdbb0150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.605+0000 7fdbaea00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb015b000 0x7fdbb0150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.605+0000 7fdbaea00640 1 -- 172.21.15.102:0/3413385339 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb008bc50 con 0x7fdbb0154960 2024-03-18T16:22:15.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.610+0000 7f97e5400640 1 --2- 172.21.15.102:0/1470273909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e8150670 0x7f97e80c7b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f97e80c97c0 tx=0x7f97d802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d617d3fb146023a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.611+0000 7ff1f7e00640 1 --2- 172.21.15.102:0/4136220899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00f6ae0 0x7ff1f004b090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0098a0 tx=0x7ff1ec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4a73c53ac39e750 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.612+0000 7f31cd400640 1 -- 172.21.15.102:0/740497510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 msgr2=0x7f31d00c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.612+0000 7f31cd400640 1 --2- 172.21.15.102:0/740497510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d00c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.612+0000 7f31cd400640 1 -- 172.21.15.102:0/740497510 >> v1:172.21.15.154:6789/0 conn(0x7f31d00c7f90 legacy=0x7f31d0159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.613+0000 7f31cd400640 1 -- 172.21.15.102:0/740497510 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d00c7450 con 0x7f31d00c7b70 2024-03-18T16:22:15.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.614+0000 7fdbaea00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb0154960 0x7fdbb014e3b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdba000e900 tx=0x7fdba000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.615+0000 7f7371a00640 1 Processor -- start 2024-03-18T16:22:15.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.615+0000 7f7371a00640 1 -- start start 2024-03-18T16:22:15.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.618+0000 7f7371a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f736c153b40 0x7f736c1570e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.618+0000 7f7371a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f736c072c20 con 0x7f736c0c8070 2024-03-18T16:22:15.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.619+0000 7f7371a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f736c0726a0 con 0x7f736c0c7c30 2024-03-18T16:22:15.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.619+0000 7f7371a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f736c157620 con 0x7f736c153b40 2024-03-18T16:22:15.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.620+0000 7f31cd400640 1 --2- 172.21.15.102:0/740497510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d00c7b70 0x7f31d006f6c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009d90 tx=0x7f31c002bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c91dfe3d7bc85510 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.620+0000 7fdba7400640 1 -- 172.21.15.102:0/3413385339 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba0017070 con 0x7fdbb0154960 2024-03-18T16:22:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.620+0000 7fdba7400640 1 -- 172.21.15.102:0/3413385339 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba000ca00 con 0x7fdbb0154960 2024-03-18T16:22:15.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.620+0000 7fdba7400640 1 -- 172.21.15.102:0/3413385339 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba000cdf0 con 0x7fdbb0154960 2024-03-18T16:22:15.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.620+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb01e55f0 con 0x7fdbb0154960 2024-03-18T16:22:15.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.621+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb0157750 con 0x7fdbb0154960 2024-03-18T16:22:15.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.623+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdbb01e53a0 con 0x7fdbb0154960 2024-03-18T16:22:15.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.623+0000 7f736a000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f736c0c7c30 0x7f736c0c8640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58514/0 (socket says 172.21.15.102:58514) 2024-03-18T16:22:15.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.623+0000 7f736a000640 1 -- 172.21.15.102:0/608371274 learned_addr learned my addr 172.21.15.102:0/608371274 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:15.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.624+0000 7f31cca00640 1 -- 172.21.15.102:0/740497510 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c0039070 con 0x7f31d00c7b70 2024-03-18T16:22:15.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.624+0000 7f31cca00640 1 -- 172.21.15.102:0/740497510 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c002cc70 con 0x7f31d00c7b70 2024-03-18T16:22:15.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.624+0000 7f31cca00640 1 -- 172.21.15.102:0/740497510 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c002ce60 con 0x7f31d00c7b70 2024-03-18T16:22:15.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.625+0000 7f31cf200640 1 -- 172.21.15.102:0/740497510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d00c7b70 msgr2=0x7f31d006f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.625+0000 7f31cf200640 1 --2- 172.21.15.102:0/740497510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d00c7b70 0x7f31d006f6c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f31c0009d90 tx=0x7f31c002bec0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.625+0000 7fcea7000640 1 Processor -- start 2024-03-18T16:22:15.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.625+0000 7fcea7000640 1 -- start start 2024-03-18T16:22:15.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.626+0000 7ff1f7400640 1 -- 172.21.15.102:0/4136220899 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec034070 con 0x7ff1f00f6ae0 2024-03-18T16:22:15.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.626+0000 7ff1f7400640 1 -- 172.21.15.102:0/4136220899 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ec030840 con 0x7ff1f00f6ae0 2024-03-18T16:22:15.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.626+0000 7ff1f7400640 1 -- 172.21.15.102:0/4136220899 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ec034070 con 0x7ff1f00f6ae0 2024-03-18T16:22:15.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.627+0000 7f31cf200640 1 -- 172.21.15.102:0/740497510 shutdown_connections 2024-03-18T16:22:15.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.627+0000 7f31cf200640 1 --2- 172.21.15.102:0/740497510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d00c7b70 0x7f31d006f6c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.627+0000 7f31cf200640 1 --2- 172.21.15.102:0/740497510 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d00c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.627+0000 7f31cf200640 1 -- 172.21.15.102:0/740497510 >> 172.21.15.102:0/740497510 conn(0x7f31d00c8430 msgr2=0x7f31d00c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.627+0000 7ff1fe200640 1 -- 172.21.15.102:0/4136220899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00f6ae0 msgr2=0x7ff1f004b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.627+0000 7ff1fe200640 1 --2- 172.21.15.102:0/4136220899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00f6ae0 0x7ff1f004b090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0098a0 tx=0x7ff1ec02b980 comp rx=0 tx=0).stop 2024-03-18T16:22:15.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.628+0000 7ff1fe200640 1 -- 172.21.15.102:0/4136220899 shutdown_connections 2024-03-18T16:22:15.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.628+0000 7ff1fe200640 1 --2- 172.21.15.102:0/4136220899 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00f6ae0 0x7ff1f004b090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.628+0000 7ff1fe200640 1 -- 172.21.15.102:0/4136220899 >> 172.21.15.102:0/4136220899 conn(0x7ff1f0050180 msgr2=0x7ff1f0050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.628+0000 7ff1fe200640 1 -- 172.21.15.102:0/4136220899 shutdown_connections 2024-03-18T16:22:15.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.628+0000 7f97e4a00640 1 -- 172.21.15.102:0/1470273909 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8035020 con 0x7f97e8150670 2024-03-18T16:22:15.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.628+0000 7f97e4a00640 1 -- 172.21.15.102:0/1470273909 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d8031800 con 0x7f97e8150670 2024-03-18T16:22:15.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.628+0000 7f97e4a00640 1 -- 172.21.15.102:0/1470273909 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d8031c10 con 0x7f97e8150670 2024-03-18T16:22:15.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.629+0000 7f31cf200640 1 -- 172.21.15.102:0/740497510 shutdown_connections 2024-03-18T16:22:15.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.629+0000 7f31cf200640 1 -- 172.21.15.102:0/740497510 wait complete. 2024-03-18T16:22:15.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.630+0000 7f97e7200640 1 -- 172.21.15.102:0/1470273909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e8150670 msgr2=0x7f97e80c7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.630+0000 7f97e7200640 1 --2- 172.21.15.102:0/1470273909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e8150670 0x7f97e80c7b70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f97e80c97c0 tx=0x7f97d802b8a0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.628+0000 7ff1fe200640 1 -- 172.21.15.102:0/4136220899 wait complete. 2024-03-18T16:22:15.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.630+0000 7fcea7000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcea00cec50 0x7fcea00cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.630+0000 7fcea7000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea00ce0d0 con 0x7fcea00ccd20 2024-03-18T16:22:15.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.630+0000 7fcea7000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcea00ce3b0 con 0x7fcea0154960 2024-03-18T16:22:15.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.630+0000 7fcea7000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fcea015a630 con 0x7fcea00cec50 2024-03-18T16:22:15.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.629+0000 7f736aa00640 1 --2- 172.21.15.102:0/608371274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f736c153b40 0x7f736c1570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.630+0000 7f97e7200640 1 -- 172.21.15.102:0/1470273909 shutdown_connections 2024-03-18T16:22:15.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.631+0000 7f97e7200640 1 --2- 172.21.15.102:0/1470273909 >> v2:172.21.15.154:3300/0 conn(0x7f97e806f6c0 0x7f97e806fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.631+0000 7f97e7200640 1 --2- 172.21.15.102:0/1470273909 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e8150670 0x7f97e80c7b70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.631+0000 7f97e7200640 1 --2- 172.21.15.102:0/1470273909 >> v2:172.21.15.102:3300/0 conn(0x7f97e80c8f50 0x7f97e80c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.631+0000 7f97e7200640 1 -- 172.21.15.102:0/1470273909 >> 172.21.15.102:0/1470273909 conn(0x7f97e80c8430 msgr2=0x7f97e80c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.631+0000 7ff1fe200640 1 Processor -- start 2024-03-18T16:22:15.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.631+0000 7f97e7200640 1 -- 172.21.15.102:0/1470273909 shutdown_connections 2024-03-18T16:22:15.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.631+0000 7f97e7200640 1 -- 172.21.15.102:0/1470273909 wait complete. 2024-03-18T16:22:15.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.632+0000 7ff1fe200640 1 -- start start 2024-03-18T16:22:15.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.632+0000 7f31cf200640 1 Processor -- start 2024-03-18T16:22:15.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.632+0000 7f31cf200640 1 -- start start 2024-03-18T16:22:15.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.634+0000 7f8972400640 1 Processor -- start 2024-03-18T16:22:15.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.634+0000 7f31cf200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31d00c7b70 0x7f31d020e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.634+0000 7f31cf200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d00c7f90 0x7f31d020eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.635+0000 7f31cf200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d0156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.635+0000 7f31cf200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f31d00c4b50 con 0x7f31d00c7b70 2024-03-18T16:22:15.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.635+0000 7f97e7200640 1 Processor -- start 2024-03-18T16:22:15.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.636+0000 7ff1fe200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f004b5d0 0x7ff1f0048a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.634+0000 7f8972400640 1 -- start start 2024-03-18T16:22:15.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.636+0000 7f8972400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 0x7f896c1570e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.636+0000 7f8972400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f896c0c7c30 0x7f896c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f8972400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f896c072c20 con 0x7f896c06b9b0 2024-03-18T16:22:15.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f8972400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f896c157620 con 0x7f896c0c7c30 2024-03-18T16:22:15.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f8972400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f896c073770 con 0x7f896c153b40 2024-03-18T16:22:15.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f736aa00640 1 -- 172.21.15.102:0/608371274 >> v1:172.21.15.154:6789/0 conn(0x7f736c0c7c30 legacy=0x7f736c0c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f736aa00640 1 -- 172.21.15.102:0/608371274 >> v1:172.21.15.102:6789/0 conn(0x7f736c0c8070 legacy=0x7f736c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f736aa00640 1 -- 172.21.15.102:0/608371274 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736c072940 con 0x7f736c153b40 2024-03-18T16:22:15.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f31ce800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d0156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.636+0000 7f97e7200640 1 -- start start 2024-03-18T16:22:15.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f97e7200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806f6c0 0x7f97e813dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f97e7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e80c8f50 0x7f97e80cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f97e7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e8150670 0x7f97e80cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f97e7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f97e8150a90 con 0x7f97e8150670 2024-03-18T16:22:15.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f97e7200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97e80c4b50 con 0x7f97e806f6c0 2024-03-18T16:22:15.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f97e7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f97e80c80b0 con 0x7f97e80c8f50 2024-03-18T16:22:15.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.637+0000 7f31cf200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31d00c4870 con 0x7f31d00c7f90 2024-03-18T16:22:15.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.638+0000 7f31cf200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f31d00c8f50 con 0x7f31d01508e0 2024-03-18T16:22:15.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.638+0000 7f896b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 0x7f896c1570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.636+0000 7ff1fe200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1f00f6ae0 0x7ff1f0049ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.636+0000 7ff1fe200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00fc980 0x7ff1f00472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.636+0000 7ff1fe200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f004c8a0 con 0x7ff1f00f6ae0 2024-03-18T16:22:15.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.636+0000 7ff1fe200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f004c720 con 0x7ff1f004b5d0 2024-03-18T16:22:15.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.636+0000 7ff1fe200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f00f6f00 con 0x7ff1f00fc980 2024-03-18T16:22:15.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.638+0000 7f31ce800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d0156610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34114/0 (socket says 172.21.15.102:34114) 2024-03-18T16:22:15.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.638+0000 7f31ce800640 1 -- 172.21.15.102:0/631553492 learned_addr learned my addr 172.21.15.102:0/631553492 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.638+0000 7f896b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 0x7f896c1570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34130/0 (socket says 172.21.15.102:34130) 2024-03-18T16:22:15.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.638+0000 7ff1fd800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00fc980 0x7ff1f00472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.638+0000 7f896b400640 1 -- 172.21.15.102:0/2867692820 learned_addr learned my addr 172.21.15.102:0/2867692820 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.640+0000 7ff1fd800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00fc980 0x7ff1f00472a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34132/0 (socket says 172.21.15.102:34132) 2024-03-18T16:22:15.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.640+0000 7ff1fd800640 1 -- 172.21.15.102:0/3751796633 learned_addr learned my addr 172.21.15.102:0/3751796633 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.643+0000 7f31ce800640 1 -- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d00c7f90 msgr2=0x7f31d020eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.643+0000 7f31ce800640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d00c7f90 0x7f31d020eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.643+0000 7f31ce800640 1 -- 172.21.15.102:0/631553492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31d00c7b70 msgr2=0x7f31d020e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.643+0000 7f31ce800640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31d00c7b70 0x7f31d020e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.643+0000 7f31ce800640 1 -- 172.21.15.102:0/631553492 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d0157690 con 0x7f31d01508e0 2024-03-18T16:22:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.645+0000 7f896b400640 1 -- 172.21.15.102:0/2867692820 >> v2:172.21.15.154:3300/0 conn(0x7f896c0c7c30 msgr2=0x7f896c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.645+0000 7f896b400640 1 --2- 172.21.15.102:0/2867692820 >> v2:172.21.15.154:3300/0 conn(0x7f896c0c7c30 0x7f896c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.645+0000 7f896b400640 1 -- 172.21.15.102:0/2867692820 >> v1:172.21.15.154:6789/0 conn(0x7f896c06b9b0 legacy=0x7f896c0c8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.645+0000 7f896b400640 1 -- 172.21.15.102:0/2867692820 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896c0726a0 con 0x7f896c153b40 2024-03-18T16:22:15.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.652+0000 7fdba7400640 1 -- 172.21.15.102:0/3413385339 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdba001a070 con 0x7fdbb0154960 2024-03-18T16:22:15.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.652+0000 7ff1fd800640 1 -- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f004b5d0 msgr2=0x7ff1f0048a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.652+0000 7ff1fd800640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1f004b5d0 0x7ff1f0048a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.653+0000 7ff1fd800640 1 -- 172.21.15.102:0/3751796633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1f00f6ae0 msgr2=0x7ff1f0049ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.653+0000 7ff1fd800640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1f00f6ae0 0x7ff1f0049ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.653+0000 7ff1fd800640 1 -- 172.21.15.102:0/3751796633 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f0049170 con 0x7ff1f00fc980 2024-03-18T16:22:15.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.653+0000 7f831fc00640 1 Processor -- start 2024-03-18T16:22:15.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.654+0000 7f831fc00640 1 -- start start 2024-03-18T16:22:15.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.654+0000 7fdba7400640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9006bc50 0x7fdb9006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.654+0000 7fdbaf400640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9006bc50 0x7fdb9006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.655+0000 7fdbaf400640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9006bc50 0x7fdb9006e110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdba80012e0 tx=0x7fdba8001790 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.656+0000 7f31ce800640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d0156610 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c3f0 tx=0x7f31bc00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.656+0000 7f831fc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8318154930 0x7f831815b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.656+0000 7f831fc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f83180ca950 0x7f83180cade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.656+0000 7f831fc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83180ca480 con 0x7f8318071e60 2024-03-18T16:22:15.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.656+0000 7f831fc00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f831815db30 con 0x7f83180ca950 2024-03-18T16:22:15.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.656+0000 7f831fc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f831815dd80 con 0x7f8318154930 2024-03-18T16:22:15.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.656+0000 7fdba7400640 1 -- 172.21.15.102:0/3413385339 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fdba00574f0 con 0x7fdbb0154960 2024-03-18T16:22:15.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.657+0000 7f896b400640 1 --2- 172.21.15.102:0/2867692820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 0x7f896c1570e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f895c009870 tx=0x7f895c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe70a1f22f71beba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.657+0000 7fdba7400640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7fdb900755e0 0x7fdb90077a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.657+0000 7fdba7400640 1 -- 172.21.15.102:0/3413385339 --> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb90078150 con 0x7fdb900755e0 2024-03-18T16:22:15.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.657+0000 7fdbafe00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7fdb900755e0 0x7fdb90077a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.658+0000 7fdbafe00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7fdb900755e0 0x7fdb90077a80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.660+0000 7fdba7400640 1 -- 172.21.15.102:0/3413385339 <== osd.0 v2:172.21.15.102:6809/3520461741 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb90078150 con 0x7fdb900755e0 2024-03-18T16:22:15.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.660+0000 7f97e5e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806f6c0 0x7f97e813dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.661+0000 7f831dc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f83180ca950 0x7f83180cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.660+0000 7f97e5e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806f6c0 0x7f97e813dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47422/0 (socket says 172.21.15.102:47422) 2024-03-18T16:22:15.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.660+0000 7f97e5e00640 1 -- 172.21.15.102:0/2606369379 learned_addr learned my addr 172.21.15.102:0/2606369379 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.662+0000 7f831dc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f83180ca950 0x7f83180cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47440/0 (socket says 172.21.15.102:47440) 2024-03-18T16:22:15.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.662+0000 7f831dc00640 1 -- 172.21.15.102:0/1085054098 learned_addr learned my addr 172.21.15.102:0/1085054098 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.663+0000 7ff1fd800640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00fc980 0x7ff1f00472a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b440 tx=0x7ff1e400b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.665+0000 7f31c6000640 1 -- 172.21.15.102:0/631553492 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc0052d0 con 0x7f31d01508e0 2024-03-18T16:22:15.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.665+0000 7f31c6000640 1 -- 172.21.15.102:0/631553492 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31bc005470 con 0x7f31d01508e0 2024-03-18T16:22:15.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.666+0000 7f31c6000640 1 -- 172.21.15.102:0/631553492 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc007260 con 0x7f31d01508e0 2024-03-18T16:22:15.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.666+0000 7f97e5e00640 1 -- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e80c8f50 msgr2=0x7f97e80cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.666+0000 7f97e5e00640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e80c8f50 0x7f97e80cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.666+0000 7f97e5e00640 1 -- 172.21.15.102:0/2606369379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e8150670 msgr2=0x7f97e80cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.666+0000 7f97e5e00640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e8150670 0x7f97e80cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.666+0000 7f97e5e00640 1 -- 172.21.15.102:0/2606369379 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e8159720 con 0x7f97e806f6c0 2024-03-18T16:22:15.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.667+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d0157970 con 0x7f31d01508e0 2024-03-18T16:22:15.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.667+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d0157f30 con 0x7f31d01508e0 2024-03-18T16:22:15.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.668+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f31d004f5b0 con 0x7f31d01508e0 2024-03-18T16:22:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.670+0000 7f736aa00640 1 --2- 172.21.15.102:0/608371274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f736c153b40 0x7f736c1570e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f736c072870 tx=0x7f735c02cd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46b4658c401d4d97 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.670+0000 7f831dc00640 1 -- 172.21.15.102:0/1085054098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8318154930 msgr2=0x7f831815b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.670+0000 7f831dc00640 1 --2- 172.21.15.102:0/1085054098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8318154930 0x7f831815b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.670+0000 7f831dc00640 1 -- 172.21.15.102:0/1085054098 >> v1:172.21.15.102:6789/0 conn(0x7f8318071e60 legacy=0x7f83180c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:15.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.670+0000 7f831dc00640 1 -- 172.21.15.102:0/1085054098 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83180cb320 con 0x7f83180ca950 2024-03-18T16:22:15.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.672+0000 7f896a000640 1 -- 172.21.15.102:0/2867692820 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c034070 con 0x7f896c153b40 2024-03-18T16:22:15.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.672+0000 7f896a000640 1 -- 172.21.15.102:0/2867692820 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f895c030840 con 0x7f896c153b40 2024-03-18T16:22:15.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.672+0000 7f896a000640 1 -- 172.21.15.102:0/2867692820 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c030c10 con 0x7f896c153b40 2024-03-18T16:22:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.673+0000 7f8972400640 1 -- 172.21.15.102:0/2867692820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 msgr2=0x7f896c1570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.673+0000 7f8972400640 1 --2- 172.21.15.102:0/2867692820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 0x7f896c1570e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f895c009870 tx=0x7f895c02b980 comp rx=0 tx=0).stop 2024-03-18T16:22:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.673+0000 7f8972400640 1 -- 172.21.15.102:0/2867692820 shutdown_connections 2024-03-18T16:22:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.673+0000 7f8972400640 1 --2- 172.21.15.102:0/2867692820 >> v2:172.21.15.154:3300/0 conn(0x7f896c0c7c30 0x7f896c06d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.673+0000 7f8972400640 1 --2- 172.21.15.102:0/2867692820 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 0x7f896c1570e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.673+0000 7f8972400640 1 -- 172.21.15.102:0/2867692820 >> 172.21.15.102:0/2867692820 conn(0x7f896c0700d0 msgr2=0x7f896c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.674+0000 7f97e5e00640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806f6c0 0x7f97e813dbb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f97d400b4a0 tx=0x7f97d400b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.674+0000 7f8972400640 1 -- 172.21.15.102:0/2867692820 shutdown_connections 2024-03-18T16:22:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.674+0000 7f8972400640 1 -- 172.21.15.102:0/2867692820 wait complete. 2024-03-18T16:22:15.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.675+0000 7f831dc00640 1 --2- 172.21.15.102:0/1085054098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83180ca950 0x7f83180cade0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8310009e50 tx=0x7f831002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c0b7d841c8d84a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.675+0000 7fcea5200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcea00ccd20 0x7fcea015aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59978/0 (socket says 172.21.15.102:59978) 2024-03-18T16:22:15.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.676+0000 7fcea5200640 1 -- 172.21.15.102:0/4148901798 learned_addr learned my addr 172.21.15.102:0/4148901798 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:15.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.677+0000 7f8972400640 1 Processor -- start 2024-03-18T16:22:15.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.677+0000 7fcea6600640 1 --2- 172.21.15.102:0/4148901798 >> v2:172.21.15.102:3300/0 conn(0x7fcea00cec50 0x7fcea00cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.677+0000 7f8972400640 1 -- start start 2024-03-18T16:22:15.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.679+0000 7f8972400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f896c06b9b0 0x7f896c0cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.679+0000 7f8972400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f896c0c7c30 0x7f896c144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.679+0000 7f8972400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 0x7f896c1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.679+0000 7f8972400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f896c073400 con 0x7f896c06b9b0 2024-03-18T16:22:15.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.679+0000 7f8972400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f896c073120 con 0x7f896c0c7c30 2024-03-18T16:22:15.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.679+0000 7f8972400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f896c153f20 con 0x7f896c153b40 2024-03-18T16:22:15.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.683+0000 7fcea6600640 1 -- 172.21.15.102:0/4148901798 >> v1:172.21.15.154:6789/0 conn(0x7fcea0154960 legacy=0x7fcea00ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.683+0000 7fcea6600640 1 -- 172.21.15.102:0/4148901798 >> v1:172.21.15.102:6789/0 conn(0x7fcea00ccd20 legacy=0x7fcea015aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.683+0000 7fcea6600640 1 -- 172.21.15.102:0/4148901798 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea00ce690 con 0x7fcea00cec50 2024-03-18T16:22:15.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.685+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 --> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdbb01e5b10 con 0x7fdb900755e0 2024-03-18T16:22:15.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.685+0000 7ff1f4c00640 1 -- 172.21.15.102:0/3751796633 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4015020 con 0x7ff1f00fc980 2024-03-18T16:22:15.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.686+0000 7ff1f4c00640 1 -- 172.21.15.102:0/3751796633 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e4011830 con 0x7ff1f00fc980 2024-03-18T16:22:15.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.686+0000 7ff1f4c00640 1 -- 172.21.15.102:0/3751796633 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4011c20 con 0x7ff1f00fc980 2024-03-18T16:22:15.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.686+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f00479e0 con 0x7ff1f00fc980 2024-03-18T16:22:15.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.686+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f00e4270 con 0x7ff1f00fc980 2024-03-18T16:22:15.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.687+0000 7fdba7400640 1 -- 172.21.15.102:0/3413385339 <== osd.0 v2:172.21.15.102:6809/3520461741 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdb9c02f7b0 con 0x7fdb900755e0 2024-03-18T16:22:15.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.687+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff1f0005390 con 0x7ff1f00fc980 2024-03-18T16:22:15.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.687+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7fdb900755e0 msgr2=0x7fdb90077a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.687+0000 7fdbb4e00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7fdb900755e0 0x7fdb90077a80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.688+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9006bc50 msgr2=0x7fdb9006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.688+0000 7fdbb4e00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9006bc50 0x7fdb9006e110 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdba80012e0 tx=0x7fdba8001790 comp rx=0 tx=0).stop 2024-03-18T16:22:15.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.688+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb0154960 msgr2=0x7fdbb014e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.688+0000 7fdbb4e00640 1 --2- 172.21.15.102:0/3413385339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb0154960 0x7fdbb014e3b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdba000e900 tx=0x7fdba000edc0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.688+0000 7fdbafe00640 1 -- 172.21.15.102:0/3413385339 reap_dead start 2024-03-18T16:22:15.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.688+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 shutdown_connections 2024-03-18T16:22:15.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.688+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 >> 172.21.15.102:0/3413385339 conn(0x7fdbb00b16a0 msgr2=0x7fdbb014f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.688+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 shutdown_connections 2024-03-18T16:22:15.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.688+0000 7fdbb4e00640 1 -- 172.21.15.102:0/3413385339 wait complete. 2024-03-18T16:22:15.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.692+0000 7f896b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f896c06b9b0 0x7f896c0cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.693+0000 7f8317400640 1 -- 172.21.15.102:0/1085054098 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831002cb20 con 0x7f83180ca950 2024-03-18T16:22:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.693+0000 7f8317400640 1 -- 172.21.15.102:0/1085054098 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831002ccc0 con 0x7f83180ca950 2024-03-18T16:22:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.693+0000 7f8317400640 1 -- 172.21.15.102:0/1085054098 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310034610 con 0x7f83180ca950 2024-03-18T16:22:15.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.694+0000 7f896b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f896c06b9b0 0x7f896c0cb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51802/0 (socket says 172.21.15.102:51802) 2024-03-18T16:22:15.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.694+0000 7f896b400640 1 -- 172.21.15.102:0/4130139706 learned_addr learned my addr 172.21.15.102:0/4130139706 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.694+0000 7f831fc00640 1 -- 172.21.15.102:0/1085054098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83180ca950 msgr2=0x7f83180cade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.694+0000 7f831fc00640 1 --2- 172.21.15.102:0/1085054098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83180ca950 0x7f83180cade0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8310009e50 tx=0x7f831002bf80 comp rx=0 tx=0).stop 2024-03-18T16:22:15.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.695+0000 7f831fc00640 1 -- 172.21.15.102:0/1085054098 shutdown_connections 2024-03-18T16:22:15.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.695+0000 7f831fc00640 1 --2- 172.21.15.102:0/1085054098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83180ca950 0x7f83180cade0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.695+0000 7f831fc00640 1 --2- 172.21.15.102:0/1085054098 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8318154930 0x7f831815b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.695+0000 7f831fc00640 1 -- 172.21.15.102:0/1085054098 >> 172.21.15.102:0/1085054098 conn(0x7f83180700d0 msgr2=0x7f831806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.695+0000 7f831fc00640 1 -- 172.21.15.102:0/1085054098 shutdown_connections 2024-03-18T16:22:15.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.695+0000 7f831fc00640 1 -- 172.21.15.102:0/1085054098 wait complete. 2024-03-18T16:22:15.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.696+0000 7fcea6600640 1 --2- 172.21.15.102:0/4148901798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcea00cec50 0x7fcea00cf0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0097a0 tx=0x7fce8c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aefd0ad711b3e7d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.697+0000 7f896aa00640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f896c0c7c30 0x7f896c144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.697+0000 7f831fc00640 1 Processor -- start 2024-03-18T16:22:15.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.697+0000 7f97de000640 1 -- 172.21.15.102:0/2606369379 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4014070 con 0x7f97e806f6c0 2024-03-18T16:22:15.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.697+0000 7f97de000640 1 -- 172.21.15.102:0/2606369379 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d4010610 con 0x7f97e806f6c0 2024-03-18T16:22:15.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.697+0000 7f97de000640 1 -- 172.21.15.102:0/2606369379 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4010a00 con 0x7f97e806f6c0 2024-03-18T16:22:15.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.697+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e80c99b0 con 0x7f97e806f6c0 2024-03-18T16:22:15.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.698+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e80c9f70 con 0x7f97e806f6c0 2024-03-18T16:22:15.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.699+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f97e804f5b0 con 0x7f97e806f6c0 2024-03-18T16:22:15.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.698+0000 7f831fc00640 1 -- start start 2024-03-18T16:22:15.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.699+0000 7f831fc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8318071e60 0x7f83181f77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.699+0000 7f831fc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83180ca950 0x7f83181598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.699+0000 7f831fc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8318154930 0x7f8318159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.699+0000 7f831fc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f831815e620 con 0x7f8318154930 2024-03-18T16:22:15.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.699+0000 7f831fc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f831815e340 con 0x7f8318071e60 2024-03-18T16:22:15.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.699+0000 7f831fc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f831815e900 con 0x7f83180ca950 2024-03-18T16:22:15.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.701+0000 7f896b400640 1 -- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 msgr2=0x7f896c1591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.701+0000 7f896b400640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f896c153b40 0x7f896c1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.701+0000 7f896b400640 1 -- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f896c0c7c30 msgr2=0x7f896c144840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.701+0000 7f896b400640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f896c0c7c30 0x7f896c144840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.701+0000 7f896b400640 1 -- 172.21.15.102:0/4130139706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896c151900 con 0x7f896c06b9b0 2024-03-18T16:22:15.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.702+0000 7f31c6000640 1 -- 172.21.15.102:0/631553492 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f31bc007400 con 0x7f31d01508e0 2024-03-18T16:22:15.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.704+0000 7f31c6000640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f31b006bc50 0x7f31b006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.704+0000 7f31cde00640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f31b006bc50 0x7f31b006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.705+0000 7f31cde00640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f31b006bc50 0x7f31b006e110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f31c8006f40 tx=0x7f31c8006e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.705+0000 7f31c6000640 1 -- 172.21.15.102:0/631553492 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f31bc056ed0 con 0x7f31d01508e0 2024-03-18T16:22:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.706+0000 7f31c6000640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f31b00755e0 0x7f31b0077a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.706+0000 7f31c6000640 1 -- 172.21.15.102:0/631553492 --> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31b0078150 con 0x7f31b00755e0 2024-03-18T16:22:15.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.706+0000 7f31cd400640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f31b00755e0 0x7f31b0077a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.707+0000 7f31cd400640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f31b00755e0 0x7f31b0077a80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.710 INFO:teuthology.orchestra.run.smithi102.stdout:64424509457 2024-03-18T16:22:15.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-18T16:22:15.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.708+0000 7f31c6000640 1 -- 172.21.15.102:0/631553492 <== osd.2 v2:172.21.15.102:6824/2928547335 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31b0078150 con 0x7f31b00755e0 2024-03-18T16:22:15.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.709+0000 7f896b400640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f896c06b9b0 0x7f896c0cb6a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f895c009990 tx=0x7f895c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.710+0000 7fce9fe00640 1 -- 172.21.15.102:0/4148901798 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8c02f810 con 0x7fcea00cec50 2024-03-18T16:22:15.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.710+0000 7fce9fe00640 1 -- 172.21.15.102:0/4148901798 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce8c02f9b0 con 0x7fcea00cec50 2024-03-18T16:22:15.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.710+0000 7fce9fe00640 1 -- 172.21.15.102:0/4148901798 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8c02fd80 con 0x7fcea00cec50 2024-03-18T16:22:15.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.712+0000 7f31c6000640 1 -- 172.21.15.102:0/631553492 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f31bc0550d0 con 0x7f31d01508e0 2024-03-18T16:22:15.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.711+0000 7fcea7000640 1 -- 172.21.15.102:0/4148901798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcea00cec50 msgr2=0x7fcea00cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.711+0000 7fcea7000640 1 --2- 172.21.15.102:0/4148901798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcea00cec50 0x7fcea00cf0a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0097a0 tx=0x7fce8c02b9f0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.711+0000 7fcea7000640 1 -- 172.21.15.102:0/4148901798 shutdown_connections 2024-03-18T16:22:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.711+0000 7fcea7000640 1 --2- 172.21.15.102:0/4148901798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcea00cec50 0x7fcea00cf0a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.711+0000 7fcea7000640 1 -- 172.21.15.102:0/4148901798 >> 172.21.15.102:0/4148901798 conn(0x7fcea00b16a0 msgr2=0x7fcea00b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.711+0000 7fcea7000640 1 -- 172.21.15.102:0/4148901798 shutdown_connections 2024-03-18T16:22:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.711+0000 7fcea7000640 1 -- 172.21.15.102:0/4148901798 wait complete. 2024-03-18T16:22:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.713+0000 7fcea7000640 1 Processor -- start 2024-03-18T16:22:15.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.711+0000 7f831dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8318154930 0x7f8318159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.713+0000 7fcea7000640 1 -- start start 2024-03-18T16:22:15.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.714+0000 7f8317e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83180ca950 0x7f83181598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.714+0000 7f7369600640 1 -- 172.21.15.102:0/608371274 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c03a070 con 0x7f736c153b40 2024-03-18T16:22:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.714+0000 7f7369600640 1 -- 172.21.15.102:0/608371274 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f735c02dab0 con 0x7f736c153b40 2024-03-18T16:22:15.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.714+0000 7f7369600640 1 -- 172.21.15.102:0/608371274 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c035440 con 0x7f736c153b40 2024-03-18T16:22:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.715+0000 7fcea7000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcea00ccd20 0x7fcea00c7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.715+0000 7fcea7000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcea00cec50 0x7fcea01f7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.715+0000 7fcea7000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcea0154960 0x7fcea00b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.715+0000 7fcea7000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcea00cf5e0 con 0x7fcea00ccd20 2024-03-18T16:22:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.715+0000 7fcea7000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcea00cac20 con 0x7fcea0154960 2024-03-18T16:22:15.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.715+0000 7fcea7000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcea0150350 con 0x7fcea00cec50 2024-03-18T16:22:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.716+0000 7f7371a00640 1 -- 172.21.15.102:0/608371274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f736c153b40 msgr2=0x7f736c1570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.716+0000 7f7371a00640 1 --2- 172.21.15.102:0/608371274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f736c153b40 0x7f736c1570e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f736c072870 tx=0x7f735c02cd00 comp rx=0 tx=0).stop 2024-03-18T16:22:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.716+0000 7f7371a00640 1 -- 172.21.15.102:0/608371274 shutdown_connections 2024-03-18T16:22:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.716+0000 7f7371a00640 1 --2- 172.21.15.102:0/608371274 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f736c153b40 0x7f736c1570e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.716+0000 7f7371a00640 1 -- 172.21.15.102:0/608371274 >> 172.21.15.102:0/608371274 conn(0x7f736c0700d0 msgr2=0x7f736c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.716+0000 7f7371a00640 1 -- 172.21.15.102:0/608371274 shutdown_connections 2024-03-18T16:22:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.717+0000 7f7371a00640 1 -- 172.21.15.102:0/608371274 wait complete. 2024-03-18T16:22:15.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.717+0000 7f8963400640 1 -- 172.21.15.102:0/4130139706 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c048070 con 0x7f896c06b9b0 2024-03-18T16:22:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.717+0000 7f8963400640 1 -- 172.21.15.102:0/4130139706 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f895c030640 con 0x7f896c06b9b0 2024-03-18T16:22:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.717+0000 7f8963400640 1 -- 172.21.15.102:0/4130139706 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c034040 con 0x7f896c06b9b0 2024-03-18T16:22:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.717+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896c158720 con 0x7f896c06b9b0 2024-03-18T16:22:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.717+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896c0cc330 con 0x7f896c06b9b0 2024-03-18T16:22:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.719+0000 7f7371a00640 1 Processor -- start 2024-03-18T16:22:15.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.719+0000 7f831dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8318154930 0x7f8318159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51812/0 (socket says 172.21.15.102:51812) 2024-03-18T16:22:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.719+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f896c074fd0 con 0x7f896c06b9b0 2024-03-18T16:22:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.719+0000 7f831dc00640 1 -- 172.21.15.102:0/1685128764 learned_addr learned my addr 172.21.15.102:0/1685128764 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.719+0000 7f7371a00640 1 -- start start 2024-03-18T16:22:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.721+0000 7f7371a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f736c0c7c30 0x7f736c0ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.721+0000 7f7371a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f736c0c8070 0x7f736c1ecfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.721+0000 7f7371a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f736c153b40 0x7f736c0cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.721+0000 7f7371a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f736c073400 con 0x7f736c153b40 2024-03-18T16:22:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.721+0000 7f7371a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f736c073770 con 0x7f736c0c8070 2024-03-18T16:22:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.721+0000 7f7371a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f736c153f20 con 0x7f736c0c7c30 2024-03-18T16:22:15.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.728+0000 7f97de000640 1 -- 172.21.15.102:0/2606369379 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f97d4017070 con 0x7f97e806f6c0 2024-03-18T16:22:15.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.729+0000 7fcea5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcea00ccd20 0x7fcea00c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.729+0000 7fcea5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcea00ccd20 0x7fcea00c7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51826/0 (socket says 172.21.15.102:51826) 2024-03-18T16:22:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.729+0000 7fcea5c00640 1 -- 172.21.15.102:0/2045915942 learned_addr learned my addr 172.21.15.102:0/2045915942 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.730+0000 7f97de000640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 0x7f97c806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.730+0000 7f97e5400640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 0x7f97c806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.731+0000 7f97e5400640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 0x7f97c806e110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f97d8036000 tx=0x7f97d8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.735+0000 7f831dc00640 1 -- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83180ca950 msgr2=0x7f83181598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.735+0000 7f831dc00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83180ca950 0x7f83181598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.735+0000 7f831dc00640 1 -- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8318071e60 msgr2=0x7f83181f77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.735+0000 7f831dc00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8318071e60 0x7f83181f77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.735+0000 7f831dc00640 1 -- 172.21.15.102:0/1685128764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83181f8070 con 0x7f8318154930 2024-03-18T16:22:15.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.736+0000 7f97de000640 1 -- 172.21.15.102:0/2606369379 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f97d4056e20 con 0x7f97e806f6c0 2024-03-18T16:22:15.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.736+0000 7f736b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f736c153b40 0x7f736c0cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.736+0000 7f97de000640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f97c80755f0 0x7f97c8077a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.737+0000 7f97de000640 1 -- 172.21.15.102:0/2606369379 --> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97c8078160 con 0x7f97c80755f0 2024-03-18T16:22:15.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.736+0000 7f736b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f736c153b40 0x7f736c0cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51834/0 (socket says 172.21.15.102:51834) 2024-03-18T16:22:15.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.736+0000 7f736b400640 1 -- 172.21.15.102:0/3245192536 learned_addr learned my addr 172.21.15.102:0/3245192536 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:15.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.737+0000 7f97e6800640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f97c80755f0 0x7f97c8077a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.738+0000 7f97e6800640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f97c80755f0 0x7f97c8077a90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.738+0000 7f736a000640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f736c0c8070 0x7f736c1ecfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.740+0000 7f97de000640 1 -- 172.21.15.102:0/2606369379 <== osd.1 v2:172.21.15.102:6800/1867629855 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97c8078160 con 0x7f97c80755f0 2024-03-18T16:22:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.740+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 --> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f31d00c7950 con 0x7f31b00755e0 2024-03-18T16:22:15.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.741+0000 7fcea6600640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcea0154960 0x7fcea00b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.741+0000 7f31c6000640 1 -- 172.21.15.102:0/631553492 <== osd.2 v2:172.21.15.102:6824/2928547335 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f31d00c7950 con 0x7f31b00755e0 2024-03-18T16:22:15.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.741+0000 7ff1f4c00640 1 -- 172.21.15.102:0/3751796633 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff1e4018020 con 0x7ff1f00fc980 2024-03-18T16:22:15.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.742+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f31b00755e0 msgr2=0x7f31b0077a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.742+0000 7f31cf200640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f31b00755e0 0x7f31b0077a80 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.742+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f31b006bc50 msgr2=0x7f31b006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.742+0000 7f31cf200640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f31b006bc50 0x7f31b006e110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f31c8006f40 tx=0x7f31c8006e90 comp rx=0 tx=0).stop 2024-03-18T16:22:15.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.742+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 msgr2=0x7f31d0156610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.742+0000 7f31cf200640 1 --2- 172.21.15.102:0/631553492 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f31d01508e0 0x7f31d0156610 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c3f0 tx=0x7f31bc00c8b0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.742+0000 7f31ce800640 1 -- 172.21.15.102:0/631553492 reap_dead start 2024-03-18T16:22:15.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.742+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 shutdown_connections 2024-03-18T16:22:15.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.742+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 >> 172.21.15.102:0/631553492 conn(0x7f31d00c8430 msgr2=0x7f31d00c8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.743+0000 7ff1f4c00640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff1d806bc20 0x7ff1d806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.743+0000 7ff1fce00640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff1d806bc20 0x7ff1d806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.743+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 shutdown_connections 2024-03-18T16:22:15.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.743+0000 7f31cf200640 1 -- 172.21.15.102:0/631553492 wait complete. 2024-03-18T16:22:15.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.743+0000 7ff1fce00640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff1d806bc20 0x7ff1d806e0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8075b80 tx=0x7ff1f8073c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.745+0000 7f736a000640 1 -- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f736c0c7c30 msgr2=0x7f736c0ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.745+0000 7f736a000640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f736c0c7c30 0x7f736c0ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.745+0000 7f736a000640 1 -- 172.21.15.102:0/3245192536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f736c153b40 msgr2=0x7f736c0cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.745+0000 7f736a000640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f736c153b40 0x7f736c0cafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.745+0000 7f736a000640 1 -- 172.21.15.102:0/3245192536 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736c0c9a50 con 0x7f736c0c8070 2024-03-18T16:22:15.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.746+0000 7ff1f4c00640 1 -- 172.21.15.102:0/3751796633 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7ff1e4057350 con 0x7ff1f00fc980 2024-03-18T16:22:15.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.746+0000 7ff1f4c00640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7ff1d80755c0 0x7ff1d8077a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.746+0000 7ff1f4c00640 1 -- 172.21.15.102:0/3751796633 --> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1d8078130 con 0x7ff1d80755c0 2024-03-18T16:22:15.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.746+0000 7fcea5200640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcea00cec50 0x7fcea01f7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.748+0000 7ff1f7e00640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7ff1d80755c0 0x7ff1d8077a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.749+0000 7ff1f7e00640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7ff1d80755c0 0x7ff1d8077a60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.749+0000 7f736a000640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f736c0c8070 0x7f736c1ecfe0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f736400caa0 tx=0x7f736400cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.750+0000 7ff1f4c00640 1 -- 172.21.15.102:0/3751796633 <== osd.4 v2:172.21.15.154:6826/2919562209 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1d8078130 con 0x7ff1d80755c0 2024-03-18T16:22:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.754+0000 7fcea6600640 1 -- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcea00cec50 msgr2=0x7fcea01f7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.754+0000 7fcea6600640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcea00cec50 0x7fcea01f7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.754+0000 7fcea6600640 1 -- 172.21.15.102:0/2045915942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcea00ccd20 msgr2=0x7fcea00c7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.754+0000 7fcea6600640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcea00ccd20 0x7fcea00c7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.754+0000 7fcea6600640 1 -- 172.21.15.102:0/2045915942 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea0150610 con 0x7fcea0154960 2024-03-18T16:22:15.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.760+0000 7fcea6600640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcea0154960 0x7fcea00b0870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fce8c033000 tx=0x7fce8c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.761+0000 7f7362a00640 1 -- 172.21.15.102:0/3245192536 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364017070 con 0x7f736c0c8070 2024-03-18T16:22:15.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.761+0000 7f7362a00640 1 -- 172.21.15.102:0/3245192536 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f736400ab40 con 0x7f736c0c8070 2024-03-18T16:22:15.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.761+0000 7f7362a00640 1 -- 172.21.15.102:0/3245192536 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364012400 con 0x7f736c0c8070 2024-03-18T16:22:15.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.764+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736c0cc430 con 0x7f736c0c8070 2024-03-18T16:22:15.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.765+0000 7fce9d600640 1 -- 172.21.15.102:0/2045915942 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8c02fe60 con 0x7fcea0154960 2024-03-18T16:22:15.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.765+0000 7fce9d600640 1 -- 172.21.15.102:0/2045915942 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce8c037be0 con 0x7fcea0154960 2024-03-18T16:22:15.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.765+0000 7fce9d600640 1 -- 172.21.15.102:0/2045915942 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce8c0423f0 con 0x7fcea0154960 2024-03-18T16:22:15.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.764+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f736c0c9d10 con 0x7f736c0c8070 2024-03-18T16:22:15.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.765+0000 7f8963400640 1 -- 172.21.15.102:0/4130139706 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f895c04b070 con 0x7f896c06b9b0 2024-03-18T16:22:15.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.766+0000 7f8963400640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f894c06bc20 0x7f894c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.769 INFO:teuthology.orchestra.run.smithi102.stdout:60129542162 2024-03-18T16:22:15.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-18T16:22:15.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.767+0000 7fcea7000640 1 -- 172.21.15.102:0/2045915942 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcea0159d60 con 0x7fcea0154960 2024-03-18T16:22:15.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.768+0000 7f896aa00640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f894c06bc20 0x7f894c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.767+0000 7fcea7000640 1 -- 172.21.15.102:0/2045915942 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcea00b1420 con 0x7fcea0154960 2024-03-18T16:22:15.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.769+0000 7f896aa00640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f894c06bc20 0x7f894c06e0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f896c1f1f00 tx=0x7f896400d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.770+0000 7fcea7000640 1 -- 172.21.15.102:0/2045915942 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcea015a3b0 con 0x7fcea0154960 2024-03-18T16:22:15.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.770+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f736c074fd0 con 0x7f736c0c8070 2024-03-18T16:22:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.770+0000 7f8963400640 1 -- 172.21.15.102:0/4130139706 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f895c0307f0 con 0x7f896c06b9b0 2024-03-18T16:22:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.771+0000 7f8963400640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f894c076290 0x7f894c078730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.771+0000 7f8963400640 1 -- 172.21.15.102:0/4130139706 --> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f894c078e00 con 0x7f894c076290 2024-03-18T16:22:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.771+0000 7f896be00640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f894c076290 0x7f894c078730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.772+0000 7f896be00640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f894c076290 0x7f894c078730 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.773+0000 7f8963400640 1 -- 172.21.15.102:0/4130139706 <== osd.6 v2:172.21.15.154:6810/1929522630 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f894c078e00 con 0x7f894c076290 2024-03-18T16:22:15.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.776+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 --> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f97e8157280 con 0x7f97c80755f0 2024-03-18T16:22:15.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.777+0000 7f97de000640 1 -- 172.21.15.102:0/2606369379 <== osd.1 v2:172.21.15.102:6800/1867629855 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f97e8157280 con 0x7f97c80755f0 2024-03-18T16:22:15.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.779+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f97c80755f0 msgr2=0x7f97c8077a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.779+0000 7f97e7200640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f97c80755f0 0x7f97c8077a90 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.779+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 msgr2=0x7f97c806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.779+0000 7f97e7200640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 0x7f97c806e110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f97d8036000 tx=0x7f97d8030000 comp rx=0 tx=0).stop 2024-03-18T16:22:15.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.780+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806f6c0 msgr2=0x7f97e813dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.780+0000 7f97e7200640 1 --2- 172.21.15.102:0/2606369379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806f6c0 0x7f97e813dbb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f97d400b4a0 tx=0x7f97d400b960 comp rx=0 tx=0).stop 2024-03-18T16:22:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.780+0000 7f97e6800640 1 -- 172.21.15.102:0/2606369379 reap_dead start 2024-03-18T16:22:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.780+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 shutdown_connections 2024-03-18T16:22:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.780+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 >> 172.21.15.102:0/2606369379 conn(0x7f97e80c8430 msgr2=0x7f97e80c8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.780+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 shutdown_connections 2024-03-18T16:22:15.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.780+0000 7f97e7200640 1 -- 172.21.15.102:0/2606369379 wait complete. 2024-03-18T16:22:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.784+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 --> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff1f004a4f0 con 0x7ff1d80755c0 2024-03-18T16:22:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.785+0000 7ff1f4c00640 1 -- 172.21.15.102:0/3751796633 <== osd.4 v2:172.21.15.154:6826/2919562209 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff1f004a4f0 con 0x7ff1d80755c0 2024-03-18T16:22:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.786+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7ff1d80755c0 msgr2=0x7ff1d8077a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.786+0000 7ff1fe200640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7ff1d80755c0 0x7ff1d8077a60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.787+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff1d806bc20 msgr2=0x7ff1d806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.787+0000 7ff1fe200640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff1d806bc20 0x7ff1d806e0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8075b80 tx=0x7ff1f8073c90 comp rx=0 tx=0).stop 2024-03-18T16:22:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.787+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00fc980 msgr2=0x7ff1f00472a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.787+0000 7ff1fe200640 1 --2- 172.21.15.102:0/3751796633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1f00fc980 0x7ff1f00472a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b440 tx=0x7ff1e400b900 comp rx=0 tx=0).stop 2024-03-18T16:22:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.787+0000 7ff1fd800640 1 -- 172.21.15.102:0/3751796633 reap_dead start 2024-03-18T16:22:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.787+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 shutdown_connections 2024-03-18T16:22:15.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.787+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 >> 172.21.15.102:0/3751796633 conn(0x7ff1f0050180 msgr2=0x7ff1f00fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.787+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 shutdown_connections 2024-03-18T16:22:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.788+0000 7ff1fe200640 1 -- 172.21.15.102:0/3751796633 wait complete. 2024-03-18T16:22:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.788+0000 7f831dc00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8318154930 0x7f8318159e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8310009e50 tx=0x7f8310004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.796+0000 7f7362a00640 1 -- 172.21.15.102:0/3245192536 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f736401a070 con 0x7f736c0c8070 2024-03-18T16:22:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.797+0000 7f7362a00640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f734c06bcd0 0x7f734c06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.798+0000 7f7362a00640 1 -- 172.21.15.102:0/3245192536 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f736408b230 con 0x7f736c0c8070 2024-03-18T16:22:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.798+0000 7f7362a00640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f734c0756e0 0x7f734c077b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.798+0000 7f7362a00640 1 -- 172.21.15.102:0/3245192536 --> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f734c078250 con 0x7f734c0756e0 2024-03-18T16:22:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.799+0000 7f736aa00640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f734c06bcd0 0x7f734c06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.799+0000 7f736b400640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f734c0756e0 0x7f734c077b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.800+0000 7f736aa00640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f734c06bcd0 0x7f734c06e190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f735c037000 tx=0x7f735c02f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.801+0000 7f736b400640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f734c0756e0 0x7f734c077b80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.802+0000 7f7362a00640 1 -- 172.21.15.102:0/3245192536 <== osd.5 v2:172.21.15.154:6802/1669301852 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f734c078250 con 0x7f734c0756e0 2024-03-18T16:22:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.808+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 --> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f896c072df0 con 0x7f894c076290 2024-03-18T16:22:15.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.808+0000 7f8963400640 1 -- 172.21.15.102:0/4130139706 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f895c078220 con 0x7f896c06b9b0 2024-03-18T16:22:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.810+0000 7f8963400640 1 -- 172.21.15.102:0/4130139706 <== osd.6 v2:172.21.15.154:6810/1929522630 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f896c072df0 con 0x7f894c076290 2024-03-18T16:22:15.813 INFO:teuthology.orchestra.run.smithi102.stdout:64424509459 2024-03-18T16:22:15.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-18T16:22:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.812+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f894c076290 msgr2=0x7f894c078730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.812+0000 7f8972400640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f894c076290 0x7f894c078730 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.812+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f894c06bc20 msgr2=0x7f894c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.812+0000 7f8972400640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f894c06bc20 0x7f894c06e0e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f896c1f1f00 tx=0x7f896400d040 comp rx=0 tx=0).stop 2024-03-18T16:22:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.812+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f896c06b9b0 msgr2=0x7f896c0cb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.812+0000 7f8972400640 1 --2- 172.21.15.102:0/4130139706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f896c06b9b0 0x7f896c0cb6a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f895c009990 tx=0x7f895c004730 comp rx=0 tx=0).stop 2024-03-18T16:22:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.812+0000 7f896be00640 1 -- 172.21.15.102:0/4130139706 reap_dead start 2024-03-18T16:22:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.813+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 shutdown_connections 2024-03-18T16:22:15.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.813+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 >> 172.21.15.102:0/4130139706 conn(0x7f896c0700d0 msgr2=0x7f896c0c8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.820 INFO:teuthology.orchestra.run.smithi102.stdout:64424509459 2024-03-18T16:22:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.814+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 shutdown_connections 2024-03-18T16:22:15.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.814+0000 7f8972400640 1 -- 172.21.15.102:0/4130139706 wait complete. 2024-03-18T16:22:15.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-18T16:22:15.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.823+0000 7fce9d600640 1 -- 172.21.15.102:0/2045915942 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fce8c049060 con 0x7fcea0154960 2024-03-18T16:22:15.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.826+0000 7fce9d600640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce8006bc20 0x7fce8006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.826+0000 7fce9d600640 1 -- 172.21.15.102:0/2045915942 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fce8c0ad230 con 0x7fcea0154960 2024-03-18T16:22:15.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.830+0000 7fcea5c00640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce8006bc20 0x7fce8006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.830+0000 7fce9d600640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fce80075630 0x7fce80077ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.830+0000 7fce9d600640 1 -- 172.21.15.102:0/2045915942 --> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce8c043e50 con 0x7fce80075630 2024-03-18T16:22:15.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.831+0000 7fcea5200640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fce80075630 0x7fce80077ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.832+0000 7fcea5c00640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce8006bc20 0x7fce8006e0e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fce94007fd0 tx=0x7fce94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.832+0000 7fcea5200640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fce80075630 0x7fce80077ad0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.834+0000 7fce9d600640 1 -- 172.21.15.102:0/2045915942 <== osd.3 v2:172.21.15.102:6808/1215178908 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce8c043e50 con 0x7fce80075630 2024-03-18T16:22:15.848 INFO:teuthology.orchestra.run.smithi102.stdout:55834574866 2024-03-18T16:22:15.849 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-18T16:22:15.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.848+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 --> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f736c0ca160 con 0x7f734c0756e0 2024-03-18T16:22:15.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.851+0000 7f7362a00640 1 -- 172.21.15.102:0/3245192536 <== osd.5 v2:172.21.15.154:6802/1669301852 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f736c0ca160 con 0x7f734c0756e0 2024-03-18T16:22:15.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.856+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f734c0756e0 msgr2=0x7f734c077b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.856+0000 7f7371a00640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f734c0756e0 0x7f734c077b80 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.858+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f734c06bcd0 msgr2=0x7f734c06e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.858+0000 7f7371a00640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f734c06bcd0 0x7f734c06e190 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f735c037000 tx=0x7f735c02f000 comp rx=0 tx=0).stop 2024-03-18T16:22:15.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.858+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f736c0c8070 msgr2=0x7f736c1ecfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.858+0000 7f7371a00640 1 --2- 172.21.15.102:0/3245192536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f736c0c8070 0x7f736c1ecfe0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f736400caa0 tx=0x7f736400cea0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.858+0000 7f736b400640 1 -- 172.21.15.102:0/3245192536 reap_dead start 2024-03-18T16:22:15.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.858+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 shutdown_connections 2024-03-18T16:22:15.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.858+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 >> 172.21.15.102:0/3245192536 conn(0x7f736c0700d0 msgr2=0x7f736c06df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.859+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 shutdown_connections 2024-03-18T16:22:15.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.859+0000 7f7371a00640 1 -- 172.21.15.102:0/3245192536 wait complete. 2024-03-18T16:22:15.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.864+0000 7fcea7000640 1 -- 172.21.15.102:0/2045915942 --> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcea004f5b0 con 0x7fce80075630 2024-03-18T16:22:15.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.866+0000 7fce9d600640 1 -- 172.21.15.102:0/2045915942 <== osd.3 v2:172.21.15.102:6808/1215178908 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcea004f5b0 con 0x7fce80075630 2024-03-18T16:22:15.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.869+0000 7fce92a00640 1 -- 172.21.15.102:0/2045915942 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fce80075630 msgr2=0x7fce80077ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.869+0000 7fce92a00640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fce80075630 0x7fce80077ad0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.871+0000 7fce92a00640 1 -- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce8006bc20 msgr2=0x7fce8006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.871+0000 7fce92a00640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce8006bc20 0x7fce8006e0e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fce94007fd0 tx=0x7fce94009040 comp rx=0 tx=0).stop 2024-03-18T16:22:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.871+0000 7fce92a00640 1 -- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcea0154960 msgr2=0x7fcea00b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.871+0000 7fce92a00640 1 --2- 172.21.15.102:0/2045915942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcea0154960 0x7fcea00b0870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fce8c033000 tx=0x7fce8c004730 comp rx=0 tx=0).stop 2024-03-18T16:22:15.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.871+0000 7fcea6600640 1 -- 172.21.15.102:0/2045915942 reap_dead start 2024-03-18T16:22:15.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.872+0000 7fce92a00640 1 -- 172.21.15.102:0/2045915942 shutdown_connections 2024-03-18T16:22:15.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.872+0000 7fce92a00640 1 -- 172.21.15.102:0/2045915942 >> 172.21.15.102:0/2045915942 conn(0x7fcea00b16a0 msgr2=0x7fcea006d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:15.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.873+0000 7fce92a00640 1 -- 172.21.15.102:0/2045915942 shutdown_connections 2024-03-18T16:22:15.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.873+0000 7fce92a00640 1 -- 172.21.15.102:0/2045915942 wait complete. 2024-03-18T16:22:15.890 INFO:teuthology.orchestra.run.smithi102.stdout:55834574866 2024-03-18T16:22:15.890 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-18T16:22:15.903 INFO:teuthology.orchestra.run.smithi102.stdout:64424509457 2024-03-18T16:22:15.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-18T16:22:15.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.921+0000 7f8314c00640 1 -- 172.21.15.102:0/1685128764 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310043020 con 0x7f8318154930 2024-03-18T16:22:15.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.921+0000 7f8314c00640 1 -- 172.21.15.102:0/1685128764 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831003e450 con 0x7f8318154930 2024-03-18T16:22:15.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.921+0000 7f8314c00640 1 -- 172.21.15.102:0/1685128764 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310009510 con 0x7f8318154930 2024-03-18T16:22:15.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.923+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83181f8350 con 0x7f8318154930 2024-03-18T16:22:15.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.923+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8318156240 con 0x7f8318154930 2024-03-18T16:22:15.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.930+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8318074fd0 con 0x7f8318154930 2024-03-18T16:22:15.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.977+0000 7f8314c00640 1 -- 172.21.15.102:0/1685128764 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8310046020 con 0x7f8318154930 2024-03-18T16:22:15.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.978+0000 7f8314c00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f806bc20 0x7f82f806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.981+0000 7f831d200640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f806bc20 0x7f82f806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.983+0000 7f8314c00640 1 -- 172.21.15.102:0/1685128764 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f8310079c50 con 0x7f8318154930 2024-03-18T16:22:15.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.984+0000 7f8314c00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f82f80756b0 0x7f82f8077b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.984+0000 7f8314c00640 1 -- 172.21.15.102:0/1685128764 --> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8310039e50 con 0x7f82f80756b0 2024-03-18T16:22:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.984+0000 7f8314c00640 1 -- 172.21.15.102:0/1685128764 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8310077e50 con 0x7f8318154930 2024-03-18T16:22:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.984+0000 7f831d200640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f806bc20 0x7f82f806e0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f830c009670 tx=0x7f830c006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.984+0000 7f8317e00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f82f80756b0 0x7f82f8077b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.985+0000 7f8317e00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f82f80756b0 0x7f82f8077b50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:15.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:15.989+0000 7f8314c00640 1 -- 172.21.15.102:0/1685128764 <== osd.7 v2:172.21.15.154:6818/2630476699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8310039e50 con 0x7f82f80756b0 2024-03-18T16:22:16.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.006+0000 7fe618800640 1 Processor -- start 2024-03-18T16:22:16.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.006+0000 7fe618800640 1 -- start start 2024-03-18T16:22:16.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.008+0000 7fe618800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6101547a0 0x7fe61015e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.008+0000 7fe618800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe6100c98d0 0x7fe6100c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.008+0000 7fe618800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe610071e20 0x7fe610072240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.008+0000 7fe618800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe61015b510 con 0x7fe6100c98d0 2024-03-18T16:22:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.008+0000 7fe618800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe61015a6f0 con 0x7fe610071e20 2024-03-18T16:22:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.008+0000 7fe618800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe61015a990 con 0x7fe6101547a0 2024-03-18T16:22:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.009+0000 7fe616800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe610071e20 0x7fe610072240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.009+0000 7fe616800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe610071e20 0x7fe610072240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47490/0 (socket says 172.21.15.102:47490) 2024-03-18T16:22:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.009+0000 7fe616800640 1 -- 172.21.15.102:0/3477513184 learned_addr learned my addr 172.21.15.102:0/3477513184 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.011+0000 7fe616800640 1 -- 172.21.15.102:0/3477513184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6101547a0 msgr2=0x7fe61015e690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:22:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.011+0000 7fe616800640 1 --2- 172.21.15.102:0/3477513184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6101547a0 0x7fe61015e690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.011+0000 7fe616800640 1 -- 172.21.15.102:0/3477513184 >> v2:172.21.15.102:3300/0 conn(0x7fe6100c98d0 msgr2=0x7fe6100c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.011+0000 7fe616800640 1 --2- 172.21.15.102:0/3477513184 >> v2:172.21.15.102:3300/0 conn(0x7fe6100c98d0 0x7fe6100c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.011+0000 7fe616800640 1 -- 172.21.15.102:0/3477513184 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe61015ac70 con 0x7fe610071e20 2024-03-18T16:22:16.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.018+0000 7fe616800640 1 --2- 172.21.15.102:0/3477513184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe610071e20 0x7fe610072240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe6100cba70 tx=0x7fe60c02bfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=815da806520ff3f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.023+0000 7fe614a00640 1 -- 172.21.15.102:0/3477513184 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c02cb40 con 0x7fe610071e20 2024-03-18T16:22:16.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.023+0000 7fe614a00640 1 -- 172.21.15.102:0/3477513184 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe60c02cce0 con 0x7fe610071e20 2024-03-18T16:22:16.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.023+0000 7fe614a00640 1 -- 172.21.15.102:0/3477513184 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c034610 con 0x7fe610071e20 2024-03-18T16:22:16.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.031+0000 7fe618800640 1 -- 172.21.15.102:0/3477513184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe610071e20 msgr2=0x7fe610072240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.031+0000 7fe618800640 1 --2- 172.21.15.102:0/3477513184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe610071e20 0x7fe610072240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe6100cba70 tx=0x7fe60c02bfa0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.031+0000 7fe618800640 1 -- 172.21.15.102:0/3477513184 shutdown_connections 2024-03-18T16:22:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.031+0000 7fe618800640 1 --2- 172.21.15.102:0/3477513184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe610071e20 0x7fe610072240 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.031+0000 7fe618800640 1 --2- 172.21.15.102:0/3477513184 >> v2:172.21.15.102:3300/0 conn(0x7fe6100c98d0 0x7fe6100c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.031+0000 7fe618800640 1 --2- 172.21.15.102:0/3477513184 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6101547a0 0x7fe61015e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.031+0000 7fe618800640 1 -- 172.21.15.102:0/3477513184 >> 172.21.15.102:0/3477513184 conn(0x7fe6100700d0 msgr2=0x7fe61006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.032+0000 7fe618800640 1 -- 172.21.15.102:0/3477513184 shutdown_connections 2024-03-18T16:22:16.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.032+0000 7fe618800640 1 -- 172.21.15.102:0/3477513184 wait complete. 2024-03-18T16:22:16.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.038+0000 7fe618800640 1 Processor -- start 2024-03-18T16:22:16.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.039+0000 7fe618800640 1 -- start start 2024-03-18T16:22:16.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.040+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 --> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8318072980 con 0x7f82f80756b0 2024-03-18T16:22:16.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.041+0000 7fe618800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe610071e20 0x7fe610156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.041+0000 7fe618800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6100c98d0 0x7fe610158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.041+0000 7fe618800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6101547a0 0x7fe6101586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.041+0000 7fe618800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe610154b80 con 0x7fe610071e20 2024-03-18T16:22:16.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.041+0000 7fe618800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe61015b230 con 0x7fe6101547a0 2024-03-18T16:22:16.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.041+0000 7fe618800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe6100ca7f0 con 0x7fe6100c98d0 2024-03-18T16:22:16.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.042+0000 7fe616800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6101547a0 0x7fe6101586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.042+0000 7f8314c00640 1 -- 172.21.15.102:0/1685128764 <== osd.7 v2:172.21.15.154:6818/2630476699 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8318072980 con 0x7f82f80756b0 2024-03-18T16:22:16.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.042+0000 7fe616800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6101547a0 0x7fe6101586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47506/0 (socket says 172.21.15.102:47506) 2024-03-18T16:22:16.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.042+0000 7fe616800640 1 -- 172.21.15.102:0/1169430718 learned_addr learned my addr 172.21.15.102:0/1169430718 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.045+0000 7fe615400640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6100c98d0 0x7fe610158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.045+0000 7fe615e00640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe610071e20 0x7fe610156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.046+0000 7fe616800640 1 -- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6100c98d0 msgr2=0x7fe610158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.046+0000 7fe616800640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6100c98d0 0x7fe610158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.046+0000 7fe616800640 1 -- 172.21.15.102:0/1169430718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe610071e20 msgr2=0x7fe610156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.046+0000 7fe616800640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe610071e20 0x7fe610156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.046+0000 7fe616800640 1 -- 172.21.15.102:0/1169430718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6101f0650 con 0x7fe6101547a0 2024-03-18T16:22:16.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.047+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f82f80756b0 msgr2=0x7f82f8077b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.047+0000 7f831fc00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f82f80756b0 0x7f82f8077b50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.048+0000 7fe616800640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6101547a0 0x7fe6101586c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe60c02c5b0 tx=0x7fe60c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.053+0000 7fe60a000640 1 -- 172.21.15.102:0/1169430718 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c043020 con 0x7fe6101547a0 2024-03-18T16:22:16.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.053+0000 7fe60a000640 1 -- 172.21.15.102:0/1169430718 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe60c03e450 con 0x7fe6101547a0 2024-03-18T16:22:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.053+0000 7fe60a000640 1 -- 172.21.15.102:0/1169430718 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c043020 con 0x7fe6101547a0 2024-03-18T16:22:16.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.054+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6101f0930 con 0x7fe6101547a0 2024-03-18T16:22:16.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.058+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6101f0ef0 con 0x7fe6101547a0 2024-03-18T16:22:16.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.058+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f806bc20 msgr2=0x7f82f806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.058+0000 7f831fc00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f806bc20 0x7f82f806e0e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f830c009670 tx=0x7f830c006bf0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.058+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8318154930 msgr2=0x7f8318159e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.058+0000 7f831fc00640 1 --2- 172.21.15.102:0/1685128764 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8318154930 0x7f8318159e10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8310009e50 tx=0x7f8310004a40 comp rx=0 tx=0).stop 2024-03-18T16:22:16.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.058+0000 7f831dc00640 1 -- 172.21.15.102:0/1685128764 reap_dead start 2024-03-18T16:22:16.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.058+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 shutdown_connections 2024-03-18T16:22:16.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.059+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 >> 172.21.15.102:0/1685128764 conn(0x7f83180700d0 msgr2=0x7f8318153770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.059+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 shutdown_connections 2024-03-18T16:22:16.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.060+0000 7f831fc00640 1 -- 172.21.15.102:0/1685128764 wait complete. 2024-03-18T16:22:16.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.059+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe610074fd0 con 0x7fe6101547a0 2024-03-18T16:22:16.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.084+0000 7fe60a000640 1 -- 172.21.15.102:0/1169430718 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe60c046020 con 0x7fe6101547a0 2024-03-18T16:22:16.087 INFO:teuthology.orchestra.run.smithi102.stdout:64424509458 2024-03-18T16:22:16.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-18T16:22:16.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.086+0000 7fe60a000640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f406bdd0 0x7fe5f406e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.087+0000 7fe615e00640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f406bdd0 0x7fe5f406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.091+0000 7fe615e00640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f406bdd0 0x7fe5f406e290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe600006fd0 tx=0x7fe600008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.100+0000 7f5865400640 1 Processor -- start 2024-03-18T16:22:16.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.101+0000 7f5865400640 1 -- start start 2024-03-18T16:22:16.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.103+0000 7f5865400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f58600c9d00 0x7f58600cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.105+0000 7f5865400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f58600d0060 0x7f58600d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.105+0000 7f5865400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58600b3560 con 0x7f58600b2a60 2024-03-18T16:22:16.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.106+0000 7f5865400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f58600b37c0 con 0x7f58600d0060 2024-03-18T16:22:16.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.106+0000 7f5865400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f586015ea90 con 0x7f58600c9d00 2024-03-18T16:22:16.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.107+0000 7f585de00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f58600b2a60 0x7f586015a650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59996/0 (socket says 172.21.15.102:59996) 2024-03-18T16:22:16.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.107+0000 7f585de00640 1 -- 172.21.15.102:0/3772957139 learned_addr learned my addr 172.21.15.102:0/3772957139 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:16.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.110+0000 7f585f200640 1 --2- 172.21.15.102:0/3772957139 >> v2:172.21.15.102:3300/0 conn(0x7f58600d0060 0x7f58600d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.116+0000 7f585d400640 1 -- 172.21.15.102:0/3772957139 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 814854662 0 0) 0x7f58600b3560 con 0x7f58600b2a60 2024-03-18T16:22:16.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.116+0000 7f585d400640 1 -- 172.21.15.102:0/3772957139 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5844003520 con 0x7f58600b2a60 2024-03-18T16:22:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.120+0000 7f585f200640 1 -- 172.21.15.102:0/3772957139 >> v2:172.21.15.154:3300/0 conn(0x7f58600c9d00 msgr2=0x7f58600cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.120+0000 7f585f200640 1 --2- 172.21.15.102:0/3772957139 >> v2:172.21.15.154:3300/0 conn(0x7f58600c9d00 0x7f58600cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.120+0000 7f585f200640 1 -- 172.21.15.102:0/3772957139 >> v1:172.21.15.102:6789/0 conn(0x7f58600b2a60 legacy=0x7f586015a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.120+0000 7f585f200640 1 -- 172.21.15.102:0/3772957139 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58600b3280 con 0x7f58600d0060 2024-03-18T16:22:16.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.137+0000 7f585f200640 1 --2- 172.21.15.102:0/3772957139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 0x7f58600d0470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f586008d6c0 tx=0x7f585802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e6c85ec53e82351 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.141+0000 7f585d400640 1 -- 172.21.15.102:0/3772957139 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858039070 con 0x7f58600d0060 2024-03-18T16:22:16.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.146+0000 7f585d400640 1 -- 172.21.15.102:0/3772957139 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f585802cd30 con 0x7f58600d0060 2024-03-18T16:22:16.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.146+0000 7f585d400640 1 -- 172.21.15.102:0/3772957139 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858034640 con 0x7f58600d0060 2024-03-18T16:22:16.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.147+0000 7f5865400640 1 -- 172.21.15.102:0/3772957139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 msgr2=0x7f58600d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.147+0000 7f5865400640 1 --2- 172.21.15.102:0/3772957139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 0x7f58600d0470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f586008d6c0 tx=0x7f585802bf80 comp rx=0 tx=0).stop 2024-03-18T16:22:16.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.149+0000 7f5865400640 1 -- 172.21.15.102:0/3772957139 shutdown_connections 2024-03-18T16:22:16.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.149+0000 7f5865400640 1 --2- 172.21.15.102:0/3772957139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 0x7f58600d0470 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.149+0000 7f5865400640 1 --2- 172.21.15.102:0/3772957139 >> v2:172.21.15.154:3300/0 conn(0x7f58600c9d00 0x7f58600cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.149+0000 7f5865400640 1 -- 172.21.15.102:0/3772957139 >> 172.21.15.102:0/3772957139 conn(0x7f586008bfc0 msgr2=0x7f586008c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.149+0000 7f5865400640 1 -- 172.21.15.102:0/3772957139 shutdown_connections 2024-03-18T16:22:16.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.149+0000 7f5865400640 1 -- 172.21.15.102:0/3772957139 wait complete. 2024-03-18T16:22:16.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.153+0000 7f5865400640 1 Processor -- start 2024-03-18T16:22:16.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.153+0000 7f5865400640 1 -- start start 2024-03-18T16:22:16.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.155+0000 7f5865400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58600b2a60 0x7f58601f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.155+0000 7f5865400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600c9d00 0x7f586015a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.155+0000 7f5865400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 0x7f586015d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.157+0000 7f585f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 0x7f586015d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.157+0000 7f585f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 0x7f586015d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51872/0 (socket says 172.21.15.102:51872) 2024-03-18T16:22:16.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.158+0000 7f585f200640 1 -- 172.21.15.102:0/661566562 learned_addr learned my addr 172.21.15.102:0/661566562 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.159+0000 7f585e800640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58600b2a60 0x7f58601f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.160+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f586015e290 con 0x7f58600d0060 2024-03-18T16:22:16.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.160+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f586015dfb0 con 0x7f58600c9d00 2024-03-18T16:22:16.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.160+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f586015e570 con 0x7f58600b2a60 2024-03-18T16:22:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.166+0000 7f585e800640 1 -- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600c9d00 msgr2=0x7f586015a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.166+0000 7f585e800640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600c9d00 0x7f586015a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.166+0000 7f585e800640 1 -- 172.21.15.102:0/661566562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 msgr2=0x7f586015d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.167+0000 7f585e800640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 0x7f586015d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.167+0000 7f585e800640 1 -- 172.21.15.102:0/661566562 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58600c7650 con 0x7f58600b2a60 2024-03-18T16:22:16.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.171+0000 7f585e800640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58600b2a60 0x7f58601f4d90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f584c00b440 tx=0x7f584c00b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.175+0000 7f5856a00640 1 -- 172.21.15.102:0/661566562 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c011830 con 0x7f58600b2a60 2024-03-18T16:22:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.175+0000 7f5856a00640 1 -- 172.21.15.102:0/661566562 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f584c016070 con 0x7f58600b2a60 2024-03-18T16:22:16.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.175+0000 7f5856a00640 1 -- 172.21.15.102:0/661566562 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584c011c10 con 0x7f58600b2a60 2024-03-18T16:22:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.178+0000 7fe60a000640 1 -- 172.21.15.102:0/1169430718 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fe60c079800 con 0x7fe6101547a0 2024-03-18T16:22:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.176+0000 7f9fa5a00640 1 Processor -- start 2024-03-18T16:22:16.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.177+0000 7f9fa5a00640 1 -- start start 2024-03-18T16:22:16.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.179+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58600c7930 con 0x7f58600b2a60 2024-03-18T16:22:16.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.179+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f586015a160 con 0x7f58600b2a60 2024-03-18T16:22:16.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.179+0000 7f9fa5a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f98045720 0x7f9f98045b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.179+0000 7f9fa5a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9f980de3b0 0x7f9f980de800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.179+0000 7f9fa5a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f98046090 con 0x7f9f980d5090 2024-03-18T16:22:16.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.179+0000 7f9fa5a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9f98007aa0 con 0x7f9f98045720 2024-03-18T16:22:16.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.179+0000 7f9fa5a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9f98006e90 con 0x7f9f980de3b0 2024-03-18T16:22:16.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.181+0000 7f9f9f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f98045720 0x7f9f98045b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.182+0000 7f9f9f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9f98045720 0x7f9f98045b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51886/0 (socket says 172.21.15.102:51886) 2024-03-18T16:22:16.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.182+0000 7f9f9f400640 1 -- 172.21.15.102:0/3080070990 learned_addr learned my addr 172.21.15.102:0/3080070990 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.184+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58600cf8c0 con 0x7f58600b2a60 2024-03-18T16:22:16.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.186+0000 7f9f9f400640 1 -- 172.21.15.102:0/3080070990 >> v2:172.21.15.154:3300/0 conn(0x7f9f980de3b0 msgr2=0x7f9f980de800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.186+0000 7f9f9f400640 1 --2- 172.21.15.102:0/3080070990 >> v2:172.21.15.154:3300/0 conn(0x7f9f980de3b0 0x7f9f980de800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.186+0000 7f9f9f400640 1 -- 172.21.15.102:0/3080070990 >> v1:172.21.15.154:6789/0 conn(0x7f9f980d5090 legacy=0x7f9f9804b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.187+0000 7f9f9f400640 1 -- 172.21.15.102:0/3080070990 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f98007170 con 0x7f9f98045720 2024-03-18T16:22:16.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.189+0000 7f0f1f200640 1 Processor -- start 2024-03-18T16:22:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.189+0000 7f0f1f200640 1 -- start start 2024-03-18T16:22:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.191+0000 7f0f1f200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0f20154960 0x7f0f200ca510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.191+0000 7f0f1f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f200caa50 0x7f0f2015aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.191+0000 7f0f1f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f200ce0d0 con 0x7f0f200ccd20 2024-03-18T16:22:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.191+0000 7f0f1f200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0f2015a630 con 0x7f0f20154960 2024-03-18T16:22:16.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.191+0000 7f0f1f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0f200ce3b0 con 0x7f0f200caa50 2024-03-18T16:22:16.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.192+0000 7f9f9f400640 1 --2- 172.21.15.102:0/3080070990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f98045720 0x7f9f98045b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800e6a0 tx=0x7f9f8c02ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818d57efc6dd4a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.197+0000 7f0f1e800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0f200ccd20 0x7f0f200cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60002/0 (socket says 172.21.15.102:60002) 2024-03-18T16:22:16.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.197+0000 7f0f1e800640 1 -- 172.21.15.102:0/2629316428 learned_addr learned my addr 172.21.15.102:0/2629316428 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:16.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.200+0000 7f9f9ea00640 1 -- 172.21.15.102:0/3080070990 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c03a070 con 0x7f9f98045720 2024-03-18T16:22:16.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.200+0000 7f9f9ea00640 1 -- 172.21.15.102:0/3080070990 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8c02daa0 con 0x7f9f98045720 2024-03-18T16:22:16.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.200+0000 7f9f9ea00640 1 -- 172.21.15.102:0/3080070990 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c035440 con 0x7f9f98045720 2024-03-18T16:22:16.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.202+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3080070990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f98045720 msgr2=0x7f9f98045b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.202+0000 7f9fa5a00640 1 --2- 172.21.15.102:0/3080070990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f98045720 0x7f9f98045b50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800e6a0 tx=0x7f9f8c02ccf0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.203+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3080070990 shutdown_connections 2024-03-18T16:22:16.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.203+0000 7f9fa5a00640 1 --2- 172.21.15.102:0/3080070990 >> v2:172.21.15.154:3300/0 conn(0x7f9f980de3b0 0x7f9f980de800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.203+0000 7f9fa5a00640 1 --2- 172.21.15.102:0/3080070990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f98045720 0x7f9f98045b50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.203+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3080070990 >> 172.21.15.102:0/3080070990 conn(0x7f9f980505d0 msgr2=0x7f9f980509e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.204+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3080070990 shutdown_connections 2024-03-18T16:22:16.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.204+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3080070990 wait complete. 2024-03-18T16:22:16.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.205+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1452371270 0 0) 0x7f0f200ce0d0 con 0x7f0f200ccd20 2024-03-18T16:22:16.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.205+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f04003520 con 0x7f0f200ccd20 2024-03-18T16:22:16.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.207+0000 7f9fa5a00640 1 Processor -- start 2024-03-18T16:22:16.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.208+0000 7f9fa5a00640 1 -- start start 2024-03-18T16:22:16.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.210+0000 7f9fa5a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f98045720 0x7f9f98036440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.210+0000 7f9fa5a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f980d5090 0x7f9f980d62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.211+0000 7f9fa5a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f980de3b0 0x7f9f980c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.211+0000 7f9fa5a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9f980d5470 con 0x7f9f980d5090 2024-03-18T16:22:16.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.211+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3046255054 0 0) 0x7f0f04003520 con 0x7f0f200ccd20 2024-03-18T16:22:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.212+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f200caa50 msgr2=0x7f0f2015aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.212+0000 7f0f1ca00640 1 --2- 172.21.15.102:0/2629316428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f200caa50 0x7f0f2015aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.212+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 >> v2:172.21.15.154:3300/0 conn(0x7f0f20154960 msgr2=0x7f0f200ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.212+0000 7f9fa5a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9f98007730 con 0x7f9f980de3b0 2024-03-18T16:22:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.212+0000 7f9fa5a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9f9804cb50 con 0x7f9f98045720 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.212+0000 7f0f1ca00640 1 --2- 172.21.15.102:0/2629316428 >> v2:172.21.15.154:3300/0 conn(0x7f0f20154960 0x7f0f200ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.212+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f200ce690 con 0x7f0f200ccd20 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.212+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f0f180019f0 con 0x7f0f200ccd20 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.212+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 >> v1:172.21.15.102:6789/0 conn(0x7f0f200ccd20 legacy=0x7f0f200cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.213+0000 7f0f1ca00640 1 --2- 172.21.15.102:0/2629316428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f040039b0 0x7f0f04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.213+0000 7f0f1ca00640 1 --2- 172.21.15.102:0/2629316428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f200caa50 0x7f0f04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.213+0000 7f0f1ca00640 1 --2- 172.21.15.102:0/2629316428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f04004e60 0x7f0f0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.213+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2015a630 con 0x7f0f040039b0 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.213+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f200ce3b0 con 0x7f0f04004e60 2024-03-18T16:22:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.213+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0f04005620 con 0x7f0f200caa50 2024-03-18T16:22:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.214+0000 7f9f9f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f980d5090 0x7f9f980d62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.214+0000 7f9f9f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f980d5090 0x7f9f980d62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51890/0 (socket says 172.21.15.102:51890) 2024-03-18T16:22:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.214+0000 7f9f9f400640 1 -- 172.21.15.102:0/3779793869 learned_addr learned my addr 172.21.15.102:0/3779793869 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.216+0000 7fce4e800640 1 Processor -- start 2024-03-18T16:22:16.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.216+0000 7fce4e800640 1 -- start start 2024-03-18T16:22:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.217+0000 7fce4e800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fce480c5050 0x7fce4815a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.217+0000 7fce4e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fce480cc1f0 0x7fce480caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.217+0000 7fce4e800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce4815daa0 con 0x7fce48154780 2024-03-18T16:22:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.217+0000 7fce4e800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fce4815e970 con 0x7fce480cc1f0 2024-03-18T16:22:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.217+0000 7fce4e800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fce4815aad0 con 0x7fce480c5050 2024-03-18T16:22:16.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.218+0000 7f0f1de00640 1 --2- 172.21.15.102:0/2629316428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f040039b0 0x7f0f04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.219+0000 7fce47400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fce48154780 0x7fce4815e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60006/0 (socket says 172.21.15.102:60006) 2024-03-18T16:22:16.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.219+0000 7fce47400640 1 -- 172.21.15.102:0/1265008769 learned_addr learned my addr 172.21.15.102:0/1265008769 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:16.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.223+0000 7f0f1de00640 1 -- 172.21.15.102:0/2629316428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f200caa50 msgr2=0x7f0f04004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.223+0000 7f0f1de00640 1 --2- 172.21.15.102:0/2629316428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f200caa50 0x7f0f04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.223+0000 7f0f1de00640 1 -- 172.21.15.102:0/2629316428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f04004e60 msgr2=0x7f0f0400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.223+0000 7f0f1de00640 1 --2- 172.21.15.102:0/2629316428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f04004e60 0x7f0f0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.223+0000 7f0f1de00640 1 -- 172.21.15.102:0/2629316428 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f04003520 con 0x7f0f040039b0 2024-03-18T16:22:16.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.230+0000 7f0f1de00640 1 --2- 172.21.15.102:0/2629316428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f040039b0 0x7f0f04003dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009860 tx=0x7f0f04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f11292dc3300f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.235+0000 7f9f9f400640 1 -- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f98045720 msgr2=0x7f9f98036440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.235+0000 7f9f9f400640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f98045720 0x7f9f98036440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.235+0000 7f9f9f400640 1 -- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f980de3b0 msgr2=0x7f9f980c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.235+0000 7f9f9f400640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f980de3b0 0x7f9f980c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.236+0000 7f9f9f400640 1 -- 172.21.15.102:0/3779793869 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f98171090 con 0x7f9f980d5090 2024-03-18T16:22:16.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.241+0000 7f9f9f400640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f980d5090 0x7f9f980d62e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c000c00 tx=0x7f9f8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.245+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c037070 con 0x7f0f040039b0 2024-03-18T16:22:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.245+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f0c032600 con 0x7f0f040039b0 2024-03-18T16:22:16.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.245+0000 7f0f1ca00640 1 -- 172.21.15.102:0/2629316428 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c037070 con 0x7f0f040039b0 2024-03-18T16:22:16.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.246+0000 7f0f1f200640 1 -- 172.21.15.102:0/2629316428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f040039b0 msgr2=0x7f0f04003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.246+0000 7f0f1f200640 1 --2- 172.21.15.102:0/2629316428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f040039b0 0x7f0f04003dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009860 tx=0x7f0f04005830 comp rx=0 tx=0).stop 2024-03-18T16:22:16.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.246+0000 7f0f1e800640 1 -- 172.21.15.102:0/2629316428 reap_dead start 2024-03-18T16:22:16.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.246+0000 7f0f1f200640 1 -- 172.21.15.102:0/2629316428 shutdown_connections 2024-03-18T16:22:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.248+0000 7fce47e00640 1 --2- 172.21.15.102:0/1265008769 >> v2:172.21.15.102:3300/0 conn(0x7fce480cc1f0 0x7fce480caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.246+0000 7f0f1f200640 1 -- 172.21.15.102:0/2629316428 >> 172.21.15.102:0/2629316428 conn(0x7f0f200b16a0 msgr2=0x7f0f200b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.247+0000 7f0f1f200640 1 -- 172.21.15.102:0/2629316428 shutdown_connections 2024-03-18T16:22:16.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.248+0000 7f0f1f200640 1 -- 172.21.15.102:0/2629316428 wait complete. 2024-03-18T16:22:16.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.249+0000 7fce46000640 1 -- 172.21.15.102:0/1265008769 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588135643 0 0) 0x7fce4815daa0 con 0x7fce48154780 2024-03-18T16:22:16.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.249+0000 7fce46000640 1 -- 172.21.15.102:0/1265008769 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce2c003520 con 0x7fce48154780 2024-03-18T16:22:16.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.251+0000 7f9f93e00640 1 -- 172.21.15.102:0/3779793869 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c047480 con 0x7f9f980d5090 2024-03-18T16:22:16.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.251+0000 7f0f1f200640 1 Processor -- start 2024-03-18T16:22:16.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.251+0000 7f9f93e00640 1 -- 172.21.15.102:0/3779793869 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f8c047640 con 0x7f9f980d5090 2024-03-18T16:22:16.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.252+0000 7f0f1f200640 1 -- start start 2024-03-18T16:22:16.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.251+0000 7f9f93e00640 1 -- 172.21.15.102:0/3779793869 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c047a30 con 0x7f9f980d5090 2024-03-18T16:22:16.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.252+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f980d8220 con 0x7f9f980d5090 2024-03-18T16:22:16.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.252+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f980caf40 con 0x7f9f980d5090 2024-03-18T16:22:16.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.253+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f7c005290 con 0x7f9f980d5090 2024-03-18T16:22:16.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.255+0000 7f0f1f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f040039b0 0x7f0f200c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.256+0000 7f0f1f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f2015d3e0 0x7f0f201f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.256+0000 7f0f1f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f2015b560 0x7f0f201f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.256+0000 7f0f1f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0f200cf4e0 con 0x7f0f2015d3e0 2024-03-18T16:22:16.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.256+0000 7f0f1f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f200cf360 con 0x7f0f040039b0 2024-03-18T16:22:16.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.257+0000 7f0f1f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0f200cf660 con 0x7f0f2015b560 2024-03-18T16:22:16.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.271+0000 7f076ae00640 1 Processor -- start 2024-03-18T16:22:16.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.271+0000 7f076ae00640 1 -- start start 2024-03-18T16:22:16.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.273+0000 7f076ae00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c5050 0x7f076415a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.273+0000 7f076ae00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f076415daa0 con 0x7f07640cc1f0 2024-03-18T16:22:16.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.273+0000 7f076ae00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f076415aad0 con 0x7f0764154780 2024-03-18T16:22:16.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.273+0000 7f076ae00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f076415e970 con 0x7f07640c5050 2024-03-18T16:22:16.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.277+0000 7efe96600640 1 Processor -- start 2024-03-18T16:22:16.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.277+0000 7efe96600640 1 -- start start 2024-03-18T16:22:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.279+0000 7efe96600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe90154780 0x7efe9015a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.279+0000 7efe96600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe900cc1f0 0x7efe900caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.279+0000 7efe96600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe9015daa0 con 0x7efe900c5050 2024-03-18T16:22:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.279+0000 7efe96600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efe9015e970 con 0x7efe90154780 2024-03-18T16:22:16.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.279+0000 7efe96600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efe9015aad0 con 0x7efe900cc1f0 2024-03-18T16:22:16.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.348+0000 7fcdfcc00640 1 Processor -- start 2024-03-18T16:22:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.348+0000 7fcdfcc00640 1 -- start start 2024-03-18T16:22:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.349+0000 7fe60a000640 1 -- 172.21.15.102:0/1169430718 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe60c04d030 con 0x7fe6101547a0 2024-03-18T16:22:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.349+0000 7fcdfcc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcdf80c8740 0x7fcdf8072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.350+0000 7fcdfcc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcdf8158760 0x7fcdf815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.350+0000 7fcdfcc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf8157ac0 con 0x7fcdf80c8b20 2024-03-18T16:22:16.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.350+0000 7fcdfcc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fcdf815a860 con 0x7fcdf8158760 2024-03-18T16:22:16.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.350+0000 7fcdfcc00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcdf8072840 con 0x7fcdf80c8740 2024-03-18T16:22:16.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.358+0000 7fce46a00640 1 --2- 172.21.15.102:0/1265008769 >> v2:172.21.15.154:3300/0 conn(0x7fce480c5050 0x7fce4815a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.359+0000 7efe8f400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe90154780 0x7efe9015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.359+0000 7efe8f400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe90154780 0x7efe9015a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47596/0 (socket says 172.21.15.102:47596) 2024-03-18T16:22:16.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.359+0000 7efe8f400640 1 -- 172.21.15.102:0/3800325289 learned_addr learned my addr 172.21.15.102:0/3800325289 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.362+0000 7fce46a00640 1 -- 172.21.15.102:0/1265008769 >> v2:172.21.15.102:3300/0 conn(0x7fce480cc1f0 msgr2=0x7fce480caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.362+0000 7fce46a00640 1 --2- 172.21.15.102:0/1265008769 >> v2:172.21.15.102:3300/0 conn(0x7fce480cc1f0 0x7fce480caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.362+0000 7fce46a00640 1 -- 172.21.15.102:0/1265008769 >> v1:172.21.15.102:6789/0 conn(0x7fce48154780 legacy=0x7fce4815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.362+0000 7fce46a00640 1 -- 172.21.15.102:0/1265008769 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4815adb0 con 0x7fce480c5050 2024-03-18T16:22:16.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.365+0000 7efe8f400640 1 -- 172.21.15.102:0/3800325289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe900cc1f0 msgr2=0x7efe900caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.365+0000 7efe8f400640 1 --2- 172.21.15.102:0/3800325289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe900cc1f0 0x7efe900caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.365+0000 7efe8f400640 1 -- 172.21.15.102:0/3800325289 >> v1:172.21.15.102:6789/0 conn(0x7efe900c5050 legacy=0x7efe9015e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.365+0000 7efe8f400640 1 -- 172.21.15.102:0/3800325289 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe9015adb0 con 0x7efe90154780 2024-03-18T16:22:16.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.368+0000 7fce46a00640 1 --2- 172.21.15.102:0/1265008769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce480c5050 0x7fce4815a490 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce4806d770 tx=0x7fce3402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f13ef9c2f606c4f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.369+0000 7efe8f400640 1 --2- 172.21.15.102:0/3800325289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 0x7efe9015a490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efe90067e80 tx=0x7efe8002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c61f3593bd7352d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.371+0000 7f0f1de00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f040039b0 0x7f0f200c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.374+0000 7f0763e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0764154780 0x7f076415e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58556/0 (socket says 172.21.15.102:58556) 2024-03-18T16:22:16.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.374+0000 7f0763e00640 1 -- 172.21.15.102:0/372642409 learned_addr learned my addr 172.21.15.102:0/372642409 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:16.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.377+0000 7f0f1de00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f040039b0 0x7f0f200c8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47582/0 (socket says 172.21.15.102:47582) 2024-03-18T16:22:16.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.377+0000 7f0f1de00640 1 -- 172.21.15.102:0/425205957 learned_addr learned my addr 172.21.15.102:0/425205957 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.381+0000 7efe8e000640 1 -- 172.21.15.102:0/3800325289 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe80035020 con 0x7efe90154780 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.381+0000 7efe8e000640 1 -- 172.21.15.102:0/3800325289 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe80031640 con 0x7efe90154780 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.381+0000 7efe8e000640 1 -- 172.21.15.102:0/3800325289 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8002d040 con 0x7efe90154780 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.382+0000 7efe96600640 1 -- 172.21.15.102:0/3800325289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 msgr2=0x7efe9015a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.382+0000 7efe96600640 1 --2- 172.21.15.102:0/3800325289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 0x7efe9015a490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7efe90067e80 tx=0x7efe8002b8f0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.382+0000 7efe96600640 1 -- 172.21.15.102:0/3800325289 shutdown_connections 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.382+0000 7efe96600640 1 --2- 172.21.15.102:0/3800325289 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe900cc1f0 0x7efe900caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.382+0000 7efe96600640 1 --2- 172.21.15.102:0/3800325289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 0x7efe9015a490 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.382+0000 7efe96600640 1 -- 172.21.15.102:0/3800325289 >> 172.21.15.102:0/3800325289 conn(0x7efe90067160 msgr2=0x7efe90067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.382+0000 7efe96600640 1 -- 172.21.15.102:0/3800325289 shutdown_connections 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.382+0000 7efe96600640 1 -- 172.21.15.102:0/3800325289 wait complete. 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.384+0000 7efe96600640 1 Processor -- start 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.384+0000 7efe96600640 1 -- start start 2024-03-18T16:22:16.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.385+0000 7efe96600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe900c5050 0x7efe901e5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.385+0000 7efe96600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe900cc1f0 0x7efe9015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.385+0000 7efe96600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 0x7efe900cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.385+0000 7efe96600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efe9015b370 con 0x7efe900cc1f0 2024-03-18T16:22:16.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.385+0000 7efe96600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe9015dd40 con 0x7efe90154780 2024-03-18T16:22:16.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.385+0000 7efe96600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efe90154b60 con 0x7efe900c5050 2024-03-18T16:22:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.386+0000 7f0f1de00640 1 -- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f2015b560 msgr2=0x7f0f201f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.386+0000 7f0f1de00640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f2015b560 0x7f0f201f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.386+0000 7f0f1de00640 1 -- 172.21.15.102:0/425205957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f2015d3e0 msgr2=0x7f0f201f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.386+0000 7f0f1de00640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f2015d3e0 0x7f0f201f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.386+0000 7f0f1de00640 1 -- 172.21.15.102:0/425205957 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f200c7030 con 0x7f0f040039b0 2024-03-18T16:22:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.390+0000 7f0f1de00640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f040039b0 0x7f0f200c8b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009980 tx=0x7f0f0c0047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.390+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4047407430 0 0) 0x7f076415aad0 con 0x7f0764154780 2024-03-18T16:22:16.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.391+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0748003520 con 0x7f0764154780 2024-03-18T16:22:16.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.391+0000 7fcdf5e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcdf80c8740 0x7fcdf8072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.391+0000 7efe8fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 0x7efe900cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2968450453 0 0) 0x7f0748003520 con 0x7f0764154780 2024-03-18T16:22:16.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c5050 msgr2=0x7f076415a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 --2- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c5050 0x7f076415a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 >> v1:172.21.15.102:6789/0 conn(0x7f07640cc1f0 legacy=0x7f07640caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7fcdf5e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcdf80c8740 0x7fcdf8072300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47598/0 (socket says 172.21.15.102:47598) 2024-03-18T16:22:16.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7fcdf5e00640 1 -- 172.21.15.102:0/2538213402 learned_addr learned my addr 172.21.15.102:0/2538213402 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076415adb0 con 0x7f0764154780 2024-03-18T16:22:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f0754001670 con 0x7f0764154780 2024-03-18T16:22:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 >> v1:172.21.15.154:6789/0 conn(0x7f0764154780 legacy=0x7f076415e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 --2- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c5050 0x7f07480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 --2- 172.21.15.102:0/372642409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07480040c0 0x7f0748004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 --2- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0748004d90 0x7f0748005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f076415e970 con 0x7f07480040c0 2024-03-18T16:22:16.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f074800bc90 con 0x7f0748004d90 2024-03-18T16:22:16.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.398+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f074800be90 con 0x7f07640c5050 2024-03-18T16:22:16.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7efe8fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 0x7efe900cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47600/0 (socket says 172.21.15.102:47600) 2024-03-18T16:22:16.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.397+0000 7efe8fe00640 1 -- 172.21.15.102:0/2914274166 learned_addr learned my addr 172.21.15.102:0/2914274166 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.401+0000 7efe8fe00640 1 -- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe900c5050 msgr2=0x7efe901e5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.401+0000 7efe8fe00640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe900c5050 0x7efe901e5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.401+0000 7efe8fe00640 1 -- 172.21.15.102:0/2914274166 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe900cc1f0 msgr2=0x7efe9015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.401+0000 7efe8fe00640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe900cc1f0 0x7efe9015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.401+0000 7efe8fe00640 1 -- 172.21.15.102:0/2914274166 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe901e4be0 con 0x7efe90154780 2024-03-18T16:22:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.406+0000 7efe8fe00640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 0x7efe900cc8b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efe8800b9d0 tx=0x7efe8800be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.408+0000 7fcdf5e00640 1 -- 172.21.15.102:0/2538213402 >> v2:172.21.15.102:3300/0 conn(0x7fcdf8158760 msgr2=0x7fcdf815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.408+0000 7fcdf5e00640 1 --2- 172.21.15.102:0/2538213402 >> v2:172.21.15.102:3300/0 conn(0x7fcdf8158760 0x7fcdf815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.408+0000 7fcdf5e00640 1 -- 172.21.15.102:0/2538213402 >> v1:172.21.15.102:6789/0 conn(0x7fcdf80c8b20 legacy=0x7fcdf81572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.408+0000 7fcdf5e00640 1 -- 172.21.15.102:0/2538213402 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf8157da0 con 0x7fcdf80c8740 2024-03-18T16:22:16.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.408+0000 7f0768e00640 1 --2- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0748004d90 0x7f0748005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.413+0000 7f0768e00640 1 -- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c5050 msgr2=0x7f07480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.413+0000 7f0768e00640 1 --2- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c5050 0x7f07480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.413+0000 7f0768e00640 1 -- 172.21.15.102:0/372642409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07480040c0 msgr2=0x7f0748004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:16.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.413+0000 7f0768e00640 1 --2- 172.21.15.102:0/372642409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07480040c0 0x7f0748004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.413+0000 7f0768e00640 1 -- 172.21.15.102:0/372642409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f076415adb0 con 0x7f0748004d90 2024-03-18T16:22:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.417+0000 7efe87400640 1 -- 172.21.15.102:0/2914274166 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88019070 con 0x7efe90154780 2024-03-18T16:22:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.417+0000 7efe87400640 1 -- 172.21.15.102:0/2914274166 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe8800cbf0 con 0x7efe90154780 2024-03-18T16:22:16.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.417+0000 7efe87400640 1 -- 172.21.15.102:0/2914274166 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe88014440 con 0x7efe90154780 2024-03-18T16:22:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.417+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe900b2860 con 0x7efe90154780 2024-03-18T16:22:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.417+0000 7fcdf5e00640 1 --2- 172.21.15.102:0/2538213402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdf80c8740 0x7fcdf8072300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8072a00 tx=0x7fcde802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=392d7223488be6b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.417+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe901e66e0 con 0x7efe90154780 2024-03-18T16:22:16.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.419+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe90074fd0 con 0x7efe90154780 2024-03-18T16:22:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.419+0000 7fce46000640 1 -- 172.21.15.102:0/1265008769 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34035020 con 0x7fce480c5050 2024-03-18T16:22:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.419+0000 7fce46000640 1 -- 172.21.15.102:0/1265008769 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce34031830 con 0x7fce480c5050 2024-03-18T16:22:16.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.419+0000 7fce46000640 1 -- 172.21.15.102:0/1265008769 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce34031c00 con 0x7fce480c5050 2024-03-18T16:22:16.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.419+0000 7f0768e00640 1 --2- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0748004d90 0x7f0748005210 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f075c001d50 tx=0x7f0748005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d3a4310432933e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.420+0000 7fce4e800640 1 -- 172.21.15.102:0/1265008769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce480c5050 msgr2=0x7fce4815a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.420+0000 7fce4e800640 1 --2- 172.21.15.102:0/1265008769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce480c5050 0x7fce4815a490 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fce4806d770 tx=0x7fce3402b8d0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.420+0000 7fce4e800640 1 -- 172.21.15.102:0/1265008769 shutdown_connections 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.420+0000 7fce4e800640 1 --2- 172.21.15.102:0/1265008769 >> v2:172.21.15.102:3300/0 conn(0x7fce480cc1f0 0x7fce480caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.420+0000 7fce4e800640 1 --2- 172.21.15.102:0/1265008769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce480c5050 0x7fce4815a490 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.420+0000 7fce4e800640 1 -- 172.21.15.102:0/1265008769 >> 172.21.15.102:0/1265008769 conn(0x7fce48067160 msgr2=0x7fce48067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.420+0000 7fce4e800640 1 -- 172.21.15.102:0/1265008769 shutdown_connections 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.420+0000 7fce4e800640 1 -- 172.21.15.102:0/1265008769 wait complete. 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.422+0000 7fce4e800640 1 Processor -- start 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.422+0000 7fce4e800640 1 -- start start 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.423+0000 7fce4e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce480c5050 0x7fce481f8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.423+0000 7fce4e800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce480cc1f0 0x7fce4815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.423+0000 7fce4e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce48154780 0x7fce480ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.423+0000 7fce4e800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fce4815b370 con 0x7fce480c5050 2024-03-18T16:22:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.423+0000 7fce4e800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fce4815dd40 con 0x7fce480cc1f0 2024-03-18T16:22:16.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.423+0000 7fce4e800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fce48154b60 con 0x7fce48154780 2024-03-18T16:22:16.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.431+0000 7f0f16000640 1 -- 172.21.15.102:0/425205957 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c042020 con 0x7f0f040039b0 2024-03-18T16:22:16.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.431+0000 7f0f16000640 1 -- 172.21.15.102:0/425205957 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f0c003800 con 0x7f0f040039b0 2024-03-18T16:22:16.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.431+0000 7f0f16000640 1 -- 172.21.15.102:0/425205957 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c037070 con 0x7f0f040039b0 2024-03-18T16:22:16.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.431+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f200c9860 con 0x7f0f040039b0 2024-03-18T16:22:16.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.431+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2015cc10 con 0x7f0f040039b0 2024-03-18T16:22:16.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.432+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f2004f5b0 con 0x7f0f040039b0 2024-03-18T16:22:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.442+0000 7f5856a00640 1 -- 172.21.15.102:0/661566562 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f584c01b460 con 0x7f58600b2a60 2024-03-18T16:22:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.443+0000 7f5856a00640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f584006bc20 0x7f584006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.444+0000 7f585de00640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f584006bc20 0x7f584006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.444+0000 7f585de00640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f584006bc20 0x7f584006e0e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5850000f60 tx=0x7f58500078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.446+0000 7efe87400640 1 -- 172.21.15.102:0/2914274166 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efe8801c070 con 0x7efe90154780 2024-03-18T16:22:16.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.446+0000 7f5856a00640 1 -- 172.21.15.102:0/661566562 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f584c0575b0 con 0x7f58600b2a60 2024-03-18T16:22:16.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.447+0000 7efe87400640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe7006bc20 0x7efe7006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.447+0000 7efe8f400640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe7006bc20 0x7efe7006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.447+0000 7efe8f400640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe7006bc20 0x7efe7006e0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efe80036000 tx=0x7efe80030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.452+0000 7efe87400640 1 -- 172.21.15.102:0/2914274166 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7efe88057150 con 0x7efe90154780 2024-03-18T16:22:16.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.457+0000 7f5856a00640 1 -- 172.21.15.102:0/661566562 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f58600cf8c0 con 0x7f58600b2a60 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.468+0000 7fce47e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce48154780 0x7fce480ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.469+0000 7fce47e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce48154780 0x7fce480ceba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34282/0 (socket says 172.21.15.102:34282) 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.469+0000 7fce47e00640 1 -- 172.21.15.102:0/3702652748 learned_addr learned my addr 172.21.15.102:0/3702652748 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.472+0000 7fce47e00640 1 -- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce480cc1f0 msgr2=0x7fce4815c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.472+0000 7fce47e00640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce480cc1f0 0x7fce4815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.472+0000 7fce47e00640 1 -- 172.21.15.102:0/3702652748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce480c5050 msgr2=0x7fce481f8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.472+0000 7fce47e00640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce480c5050 0x7fce481f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.472+0000 7fce47e00640 1 -- 172.21.15.102:0/3702652748 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4815b690 con 0x7fce48154780 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.475+0000 7fce47e00640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce48154780 0x7fce480ceba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fce40009d30 tx=0x7fce4000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.479+0000 7fce3f400640 1 -- 172.21.15.102:0/3702652748 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40018020 con 0x7fce48154780 2024-03-18T16:22:16.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.479+0000 7fce3f400640 1 -- 172.21.15.102:0/3702652748 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce40014480 con 0x7fce48154780 2024-03-18T16:22:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.479+0000 7fce3f400640 1 -- 172.21.15.102:0/3702652748 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40014870 con 0x7fce48154780 2024-03-18T16:22:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.479+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4815b930 con 0x7fce48154780 2024-03-18T16:22:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.479+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4815bef0 con 0x7fce48154780 2024-03-18T16:22:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.481+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce48074fd0 con 0x7fce48154780 2024-03-18T16:22:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.502+0000 7fce3f400640 1 -- 172.21.15.102:0/3702652748 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fce4001b020 con 0x7fce48154780 2024-03-18T16:22:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.503+0000 7fce3f400640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce2806bc20 0x7fce2806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.503+0000 7fce47400640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce2806bc20 0x7fce2806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.504+0000 7fce47400640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce2806bc20 0x7fce2806e0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce38000f60 tx=0x7fce380078f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.504+0000 7fce3f400640 1 -- 172.21.15.102:0/3702652748 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fce40025e40 con 0x7fce48154780 2024-03-18T16:22:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.605+0000 7f9f93e00640 1 -- 172.21.15.102:0/3779793869 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9f8c0503f0 con 0x7f9f980d5090 2024-03-18T16:22:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.607+0000 7f9f93e00640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9f8006bc20 0x7f9f8006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.607+0000 7f9f9fe00640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9f8006bc20 0x7f9f8006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.608+0000 7f9f9fe00640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9f8006bc20 0x7f9f8006e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0075b80 tx=0x7f9fa0073c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.619+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe61015db00 con 0x7fe6101547a0 2024-03-18T16:22:16.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.630+0000 7f9f93e00640 1 -- 172.21.15.102:0/3779793869 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f9f8c07b6a0 con 0x7f9f980d5090 2024-03-18T16:22:16.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.630+0000 7f9f93e00640 1 -- 172.21.15.102:0/3779793869 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9f8c07d600 con 0x7f9f980d5090 2024-03-18T16:22:16.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.719+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5860074fd0 con 0x7f58600b2a60 2024-03-18T16:22:16.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.729+0000 7efe87400640 1 -- 172.21.15.102:0/2914274166 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efe88023020 con 0x7efe90154780 2024-03-18T16:22:16.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.756+0000 7fce3f400640 1 -- 172.21.15.102:0/3702652748 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fce40022030 con 0x7fce48154780 2024-03-18T16:22:16.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.768+0000 7f5856a00640 1 -- 172.21.15.102:0/661566562 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f584c059c10 con 0x7f58600b2a60 2024-03-18T16:22:16.770 INFO:teuthology.orchestra.run.smithi102.stdout:60129542160 2024-03-18T16:22:16.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f584006bc20 msgr2=0x7f584006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f584006bc20 0x7f584006e0e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5850000f60 tx=0x7f58500078f0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58600b2a60 msgr2=0x7f58601f4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58600b2a60 0x7f58601f4d90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f584c00b440 tx=0x7f584c00b900 comp rx=0 tx=0).stop 2024-03-18T16:22:16.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 shutdown_connections 2024-03-18T16:22:16.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f584006bc20 0x7f584006e0e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58600d0060 0x7f586015d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58600c9d00 0x7f586015a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 --2- 172.21.15.102:0/661566562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f58600b2a60 0x7f58601f4d90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 >> 172.21.15.102:0/661566562 conn(0x7f586008bfc0 msgr2=0x7f586008c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 shutdown_connections 2024-03-18T16:22:16.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.772+0000 7f5865400640 1 -- 172.21.15.102:0/661566562 wait complete. 2024-03-18T16:22:16.802 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542162 got 60129542160 for osd.2 2024-03-18T16:22:16.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.839+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9f7c005850 con 0x7f9f980d5090 2024-03-18T16:22:16.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.851+0000 7f9f93e00640 1 -- 172.21.15.102:0/3779793869 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9f8c0816a0 con 0x7f9f980d5090 2024-03-18T16:22:16.853 INFO:teuthology.orchestra.run.smithi102.stdout:64424509458 2024-03-18T16:22:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9f8006bc20 msgr2=0x7f9f8006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9f8006bc20 0x7f9f8006e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0075b80 tx=0x7f9fa0073c90 comp rx=0 tx=0).stop 2024-03-18T16:22:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f980d5090 msgr2=0x7f9f980d62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f980d5090 0x7f9f980d62e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c000c00 tx=0x7f9f8c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 shutdown_connections 2024-03-18T16:22:16.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9f8006bc20 0x7f9f8006e0e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f980de3b0 0x7f9f980c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f980d5090 0x7f9f980d62e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 --2- 172.21.15.102:0/3779793869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9f98045720 0x7f9f98036440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.853+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 >> 172.21.15.102:0/3779793869 conn(0x7f9f980505d0 msgr2=0x7f9f980d3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.855+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 shutdown_connections 2024-03-18T16:22:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.855+0000 7f9fa5a00640 1 -- 172.21.15.102:0/3779793869 wait complete. 2024-03-18T16:22:16.872 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509459 got 64424509458 for osd.4 2024-03-18T16:22:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.938+0000 7fcdf5400640 1 -- 172.21.15.102:0/2538213402 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8035020 con 0x7fcdf80c8740 2024-03-18T16:22:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.938+0000 7fcdf5400640 1 -- 172.21.15.102:0/2538213402 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde8031870 con 0x7fcdf80c8740 2024-03-18T16:22:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.939+0000 7fcdfcc00640 1 -- 172.21.15.102:0/2538213402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdf80c8740 msgr2=0x7fcdf8072300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.939+0000 7fcdfcc00640 1 --2- 172.21.15.102:0/2538213402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdf80c8740 0x7fcdf8072300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8072a00 tx=0x7fcde802b910 comp rx=0 tx=0).stop 2024-03-18T16:22:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.939+0000 7fcdfcc00640 1 -- 172.21.15.102:0/2538213402 shutdown_connections 2024-03-18T16:22:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.939+0000 7fcdfcc00640 1 --2- 172.21.15.102:0/2538213402 >> v2:172.21.15.102:3300/0 conn(0x7fcdf8158760 0x7fcdf815b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.939+0000 7fcdfcc00640 1 --2- 172.21.15.102:0/2538213402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdf80c8740 0x7fcdf8072300 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.939+0000 7fcdfcc00640 1 -- 172.21.15.102:0/2538213402 >> 172.21.15.102:0/2538213402 conn(0x7fcdf80700d0 msgr2=0x7fcdf806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.940+0000 7fcdfcc00640 1 -- 172.21.15.102:0/2538213402 shutdown_connections 2024-03-18T16:22:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.940+0000 7fcdfcc00640 1 -- 172.21.15.102:0/2538213402 wait complete. 2024-03-18T16:22:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.941+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7efe900cb850 con 0x7efe90154780 2024-03-18T16:22:16.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.942+0000 7fcdfcc00640 1 Processor -- start 2024-03-18T16:22:16.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.942+0000 7fcdfcc00640 1 -- start start 2024-03-18T16:22:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.944+0000 7fcdfcc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf80c8740 0x7fcdf8146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.944+0000 7fcdfcc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdf80c8b20 0x7fcdf8144710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.944+0000 7fcdfcc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdf8158760 0x7fcdf8144c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.944+0000 7fcdfcc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf815aab0 con 0x7fcdf80c8740 2024-03-18T16:22:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.944+0000 7fcdfcc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf815b7f0 con 0x7fcdf80c8b20 2024-03-18T16:22:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.944+0000 7fcdfcc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf815ad90 con 0x7fcdf8158760 2024-03-18T16:22:16.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.945+0000 7fcdf6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdf8158760 0x7fcdf8144c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.945+0000 7fcdf5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf80c8740 0x7fcdf8146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.945+0000 7fcdf5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf80c8740 0x7fcdf8146660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51980/0 (socket says 172.21.15.102:51980) 2024-03-18T16:22:16.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.945+0000 7fcdf5e00640 1 -- 172.21.15.102:0/1850933710 learned_addr learned my addr 172.21.15.102:0/1850933710 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.949+0000 7fcdf5e00640 1 -- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdf8158760 msgr2=0x7fcdf8144c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.949+0000 7fcdf5e00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdf8158760 0x7fcdf8144c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.949+0000 7fcdf5e00640 1 -- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdf80c8b20 msgr2=0x7fcdf8144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.949+0000 7fcdf5e00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdf80c8b20 0x7fcdf8144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.949+0000 7fcdf5e00640 1 -- 172.21.15.102:0/1850933710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf8147030 con 0x7fcdf80c8740 2024-03-18T16:22:16.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.952+0000 7fcdf5e00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf80c8740 0x7fcdf8146660 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcde802b950 tx=0x7fcde80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:16.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.956+0000 7fcdee000640 1 -- 172.21.15.102:0/1850933710 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8040070 con 0x7fcdf80c8740 2024-03-18T16:22:16.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.956+0000 7fcdee000640 1 -- 172.21.15.102:0/1850933710 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde8003a40 con 0x7fcdf80c8740 2024-03-18T16:22:16.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.956+0000 7fcdee000640 1 -- 172.21.15.102:0/1850933710 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8003e30 con 0x7fcdf80c8740 2024-03-18T16:22:16.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.957+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf81f33a0 con 0x7fcdf80c8740 2024-03-18T16:22:16.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.957+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf81f3960 con 0x7fcdf80c8740 2024-03-18T16:22:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.958+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdf8074fd0 con 0x7fcdf80c8740 2024-03-18T16:22:16.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.984+0000 7fcdee000640 1 -- 172.21.15.102:0/1850933710 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcde8038020 con 0x7fcdf80c8740 2024-03-18T16:22:16.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.985+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c001920 con 0x7f0748004d90 2024-03-18T16:22:16.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.986+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f075c035070 con 0x7f0748004d90 2024-03-18T16:22:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.986+0000 7f0762a00640 1 -- 172.21.15.102:0/372642409 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075c003b90 con 0x7f0748004d90 2024-03-18T16:22:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.986+0000 7fcdee000640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcdd406bc20 0x7fcdd406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.986+0000 7fcdefe00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcdd406bc20 0x7fcdd406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:16.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.987+0000 7f076ae00640 1 -- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0748004d90 msgr2=0x7f0748005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.987+0000 7f076ae00640 1 --2- 172.21.15.102:0/372642409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0748004d90 0x7f0748005210 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f075c001d50 tx=0x7f0748005830 comp rx=0 tx=0).stop 2024-03-18T16:22:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.987+0000 7f0768e00640 1 -- 172.21.15.102:0/372642409 reap_dead start 2024-03-18T16:22:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.987+0000 7f076ae00640 1 -- 172.21.15.102:0/372642409 shutdown_connections 2024-03-18T16:22:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.987+0000 7f076ae00640 1 -- 172.21.15.102:0/372642409 >> 172.21.15.102:0/372642409 conn(0x7f0764067160 msgr2=0x7f0764067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:17.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.987+0000 7fcdefe00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcdd406bc20 0x7fcdd406e0e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8073570 tx=0x7fcde0006ba0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:17.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.987+0000 7f076ae00640 1 -- 172.21.15.102:0/372642409 shutdown_connections 2024-03-18T16:22:17.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.987+0000 7f076ae00640 1 -- 172.21.15.102:0/372642409 wait complete. 2024-03-18T16:22:17.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.987+0000 7fcdee000640 1 -- 172.21.15.102:0/1850933710 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fcde8079800 con 0x7fcdf80c8740 2024-03-18T16:22:17.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.990+0000 7f076ae00640 1 Processor -- start 2024-03-18T16:22:17.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.990+0000 7f076ae00640 1 -- start start 2024-03-18T16:22:17.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.991+0000 7f076ae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748004d90 0x7f0764155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:17.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.991+0000 7f076ae00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c9300 0x7f07640b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:17.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.991+0000 7f076ae00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0764154e00 0x7f07641595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:17.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.991+0000 7f076ae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f076415dd40 con 0x7f0748004d90 2024-03-18T16:22:17.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.991+0000 7f076ae00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07640cc600 con 0x7f0764154e00 2024-03-18T16:22:17.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.991+0000 7f076ae00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f076415b370 con 0x7f07640c9300 2024-03-18T16:22:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.992+0000 7f0763400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c9300 0x7f07640b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.993+0000 7f0763400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c9300 0x7f07640b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34304/0 (socket says 172.21.15.102:34304) 2024-03-18T16:22:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.993+0000 7f0763400640 1 -- 172.21.15.102:0/486922802 learned_addr learned my addr 172.21.15.102:0/486922802 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.998+0000 7f0763400640 1 -- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0764154e00 msgr2=0x7f07641595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.998+0000 7f0763400640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0764154e00 0x7f07641595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.998+0000 7f0763400640 1 -- 172.21.15.102:0/486922802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748004d90 msgr2=0x7f0764155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.998+0000 7f0763400640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748004d90 0x7f0764155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:16.998+0000 7f0763400640 1 -- 172.21.15.102:0/486922802 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07641f2ff0 con 0x7f07640c9300 2024-03-18T16:22:17.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.002+0000 7f0763400640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c9300 0x7f07640b50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f075000b480 tx=0x7f075000b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:17.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.006+0000 7efe87400640 1 -- 172.21.15.102:0/2914274166 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efe880597b0 con 0x7efe90154780 2024-03-18T16:22:17.159 INFO:teuthology.orchestra.run.smithi102.stdout:64424509456 2024-03-18T16:22:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.006+0000 7f075be00640 1 -- 172.21.15.102:0/486922802 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750015020 con 0x7f07640c9300 2024-03-18T16:22:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.007+0000 7f075be00640 1 -- 172.21.15.102:0/486922802 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0750011830 con 0x7f07640c9300 2024-03-18T16:22:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.007+0000 7f075be00640 1 -- 172.21.15.102:0/486922802 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750011c20 con 0x7f07640c9300 2024-03-18T16:22:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.007+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07640c9d20 con 0x7f07640c9300 2024-03-18T16:22:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.007+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07641f3290 con 0x7f07640c9300 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe7006bc20 msgr2=0x7efe7006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe7006bc20 0x7efe7006e0e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efe80036000 tx=0x7efe80030000 comp rx=0 tx=0).stop 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 msgr2=0x7efe900cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 0x7efe900cc8b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efe8800b9d0 tx=0x7efe8800be90 comp rx=0 tx=0).stop 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 shutdown_connections 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe7006bc20 0x7efe7006e0e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe90154780 0x7efe900cc8b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe900cc1f0 0x7efe9015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 --2- 172.21.15.102:0/2914274166 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe900c5050 0x7efe901e5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 >> 172.21.15.102:0/2914274166 conn(0x7efe90067160 msgr2=0x7efe90151f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.008+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0764074fd0 con 0x7f07640c9300 2024-03-18T16:22:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.009+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 shutdown_connections 2024-03-18T16:22:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.009+0000 7efe96600640 1 -- 172.21.15.102:0/2914274166 wait complete. 2024-03-18T16:22:17.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509456 for osd.3 2024-03-18T16:22:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.027+0000 7f0f16000640 1 -- 172.21.15.102:0/425205957 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0f0c045020 con 0x7f0f040039b0 2024-03-18T16:22:17.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.027+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fce48072710 con 0x7fce48154780 2024-03-18T16:22:17.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.028+0000 7f0f16000640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0f0006bc50 0x7f0f0006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:17.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.029+0000 7f0f1d400640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0f0006bc50 0x7f0f0006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:17.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.030+0000 7f0f1d400640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0f0006bc50 0x7f0f0006e110 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0f200c6ba0 tx=0x7f0f10006d10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:17.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.031+0000 7f075be00640 1 -- 172.21.15.102:0/486922802 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0750018020 con 0x7f07640c9300 2024-03-18T16:22:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.031+0000 7f075be00640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f074406bc20 0x7f074406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.031+0000 7f0f16000640 1 -- 172.21.15.102:0/425205957 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f0f0c0790b0 con 0x7f0f040039b0 2024-03-18T16:22:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.032+0000 7f0763e00640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f074406bc20 0x7f074406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.032+0000 7f0763e00640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f074406bc20 0x7f074406e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0754007ae0 tx=0x7f0754001660 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:17.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.043+0000 7f0f16000640 1 -- 172.21.15.102:0/425205957 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0f0c04c030 con 0x7f0f040039b0 2024-03-18T16:22:17.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.052+0000 7fe60a000640 1 -- 172.21.15.102:0/1169430718 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe60c07be60 con 0x7fe6101547a0 2024-03-18T16:22:17.166 INFO:teuthology.orchestra.run.smithi102.stdout:64424509456 2024-03-18T16:22:17.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f406bdd0 msgr2=0x7fe5f406e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f406bdd0 0x7fe5f406e290 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe600006fd0 tx=0x7fe600008040 comp rx=0 tx=0).stop 2024-03-18T16:22:17.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6101547a0 msgr2=0x7fe6101586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6101547a0 0x7fe6101586c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe60c02c5b0 tx=0x7fe60c004a40 comp rx=0 tx=0).stop 2024-03-18T16:22:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 shutdown_connections 2024-03-18T16:22:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f406bdd0 0x7fe5f406e290 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6101547a0 0x7fe6101586c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe6100c98d0 0x7fe610158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 --2- 172.21.15.102:0/1169430718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe610071e20 0x7fe610156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 >> 172.21.15.102:0/1169430718 conn(0x7fe6100700d0 msgr2=0x7fe610153690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 shutdown_connections 2024-03-18T16:22:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.055+0000 7fe618800640 1 -- 172.21.15.102:0/1169430718 wait complete. 2024-03-18T16:22:17.168 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509456 for osd.0 2024-03-18T16:22:17.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.249+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0f20152f60 con 0x7f0f040039b0 2024-03-18T16:22:17.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.253+0000 7f0f16000640 1 -- 172.21.15.102:0/425205957 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0f0c07b710 con 0x7f0f040039b0 2024-03-18T16:22:17.255 INFO:teuthology.orchestra.run.smithi102.stdout:64424509458 2024-03-18T16:22:17.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0f0006bc50 msgr2=0x7f0f0006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0f0006bc50 0x7f0f0006e110 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0f200c6ba0 tx=0x7f0f10006d10 comp rx=0 tx=0).stop 2024-03-18T16:22:17.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f040039b0 msgr2=0x7f0f200c8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f040039b0 0x7f0f200c8b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009980 tx=0x7f0f0c0047d0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 shutdown_connections 2024-03-18T16:22:17.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0f0006bc50 0x7f0f0006e110 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0f2015b560 0x7f0f201f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f2015d3e0 0x7f0f201f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 --2- 172.21.15.102:0/425205957 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f040039b0 0x7f0f200c8b60 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c009980 tx=0x7f0f0c0047d0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 >> 172.21.15.102:0/425205957 conn(0x7f0f200b16a0 msgr2=0x7f0f2015a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:17.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 shutdown_connections 2024-03-18T16:22:17.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.255+0000 7f0f1f200640 1 -- 172.21.15.102:0/425205957 wait complete. 2024-03-18T16:22:17.273 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509459 got 64424509458 for osd.1 2024-03-18T16:22:17.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.278+0000 7f075be00640 1 -- 172.21.15.102:0/486922802 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f0750056f40 con 0x7f07640c9300 2024-03-18T16:22:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.290+0000 7f075be00640 1 -- 172.21.15.102:0/486922802 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f075001f030 con 0x7f07640c9300 2024-03-18T16:22:17.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.308+0000 7fce3f400640 1 -- 172.21.15.102:0/3702652748 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fce40010040 con 0x7fce48154780 2024-03-18T16:22:17.310 INFO:teuthology.orchestra.run.smithi102.stdout:55834574865 2024-03-18T16:22:17.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce2806bc20 msgr2=0x7fce2806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce2806bc20 0x7fce2806e0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce38000f60 tx=0x7fce380078f0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce48154780 msgr2=0x7fce480ceba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce48154780 0x7fce480ceba0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fce40009d30 tx=0x7fce4000c490 comp rx=0 tx=0).stop 2024-03-18T16:22:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 shutdown_connections 2024-03-18T16:22:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fce2806bc20 0x7fce2806e0e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fce48154780 0x7fce480ceba0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce480cc1f0 0x7fce4815c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 --2- 172.21.15.102:0/3702652748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce480c5050 0x7fce481f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 >> 172.21.15.102:0/3702652748 conn(0x7fce48067160 msgr2=0x7fce480caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 shutdown_connections 2024-03-18T16:22:17.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.310+0000 7fce4e800640 1 -- 172.21.15.102:0/3702652748 wait complete. 2024-03-18T16:22:17.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574866 got 55834574865 for osd.6 2024-03-18T16:22:17.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.405+0000 7fcdee000640 1 -- 172.21.15.102:0/1850933710 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcde807b760 con 0x7fcdf80c8740 2024-03-18T16:22:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.489+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f07640cb850 con 0x7f07640c9300 2024-03-18T16:22:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.494+0000 7f075be00640 1 -- 172.21.15.102:0/486922802 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f075000f2e0 con 0x7f07640c9300 2024-03-18T16:22:17.496 INFO:teuthology.orchestra.run.smithi102.stdout:55834574865 2024-03-18T16:22:17.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f074406bc20 msgr2=0x7f074406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f074406bc20 0x7f074406e0e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0754007ae0 tx=0x7f0754001660 comp rx=0 tx=0).stop 2024-03-18T16:22:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c9300 msgr2=0x7f07640b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c9300 0x7f07640b50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f075000b480 tx=0x7f075000b940 comp rx=0 tx=0).stop 2024-03-18T16:22:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 shutdown_connections 2024-03-18T16:22:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f074406bc20 0x7f074406e0e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0764154e00 0x7f07641595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07640c9300 0x7f07640b50e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 --2- 172.21.15.102:0/486922802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0748004d90 0x7f0764155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 >> 172.21.15.102:0/486922802 conn(0x7f0764067160 msgr2=0x7f07640c6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.497+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 shutdown_connections 2024-03-18T16:22:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.498+0000 7f076ae00640 1 -- 172.21.15.102:0/486922802 wait complete. 2024-03-18T16:22:17.517 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574866 got 55834574865 for osd.5 2024-03-18T16:22:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.769+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fcdf8148350 con 0x7fcdf80c8740 2024-03-18T16:22:17.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.774+0000 7fcdee000640 1 -- 172.21.15.102:0/1850933710 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcde807f800 con 0x7fcdf80c8740 2024-03-18T16:22:17.776 INFO:teuthology.orchestra.run.smithi102.stdout:64424509457 2024-03-18T16:22:17.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.777+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcdd406bc20 msgr2=0x7fcdd406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.777+0000 7fcdfcc00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcdd406bc20 0x7fcdd406e0e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8073570 tx=0x7fcde0006ba0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.777+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf80c8740 msgr2=0x7fcdf8146660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.777+0000 7fcdfcc00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf80c8740 0x7fcdf8146660 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcde802b950 tx=0x7fcde80046c0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.777+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 shutdown_connections 2024-03-18T16:22:17.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.777+0000 7fcdfcc00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcdd406bc20 0x7fcdd406e0e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.777+0000 7fcdfcc00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcdf8158760 0x7fcdf8144c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.777+0000 7fcdfcc00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcdf80c8b20 0x7fcdf8144710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.777+0000 7fcdfcc00640 1 --2- 172.21.15.102:0/1850933710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcdf80c8740 0x7fcdf8146660 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.778+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 >> 172.21.15.102:0/1850933710 conn(0x7fcdf80700d0 msgr2=0x7fcdf8159710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:17.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.778+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 shutdown_connections 2024-03-18T16:22:17.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.778+0000 7fcdfcc00640 1 -- 172.21.15.102:0/1850933710 wait complete. 2024-03-18T16:22:17.803 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509458 got 64424509457 for osd.7 2024-03-18T16:22:17.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-18T16:22:17.874 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-18T16:22:17.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.992+0000 7f5b89200640 1 Processor -- start 2024-03-18T16:22:17.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.992+0000 7f5b89200640 1 -- start start 2024-03-18T16:22:17.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.993+0000 7f5b89200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 0x7f5b8415a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:17.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.993+0000 7f5b89200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5b840cc1f0 0x7f5b840caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:17.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.993+0000 7f5b89200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b8415daa0 con 0x7f5b84154780 2024-03-18T16:22:17.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.993+0000 7f5b89200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5b8415e970 con 0x7f5b840cc1f0 2024-03-18T16:22:17.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.993+0000 7f5b89200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5b840cc600 con 0x7f5b840c5050 2024-03-18T16:22:17.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.994+0000 7f5b82800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5b840cc1f0 0x7f5b840caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:17.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.994+0000 7f5b81400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 0x7f5b8415a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:17.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.994+0000 7f5b82800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5b840cc1f0 0x7f5b840caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47654/0 (socket says 172.21.15.102:47654) 2024-03-18T16:22:17.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.994+0000 7f5b82800640 1 -- 172.21.15.102:0/3026498748 learned_addr learned my addr 172.21.15.102:0/3026498748 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:17.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.997+0000 7f5b82800640 1 -- 172.21.15.102:0/3026498748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 msgr2=0x7f5b8415a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:17.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.997+0000 7f5b82800640 1 --2- 172.21.15.102:0/3026498748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 0x7f5b8415a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:17.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.997+0000 7f5b82800640 1 -- 172.21.15.102:0/3026498748 >> v1:172.21.15.102:6789/0 conn(0x7f5b84154780 legacy=0x7f5b8415e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:17.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.997+0000 7f5b82800640 1 -- 172.21.15.102:0/3026498748 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8415aad0 con 0x7f5b840cc1f0 2024-03-18T16:22:18.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:17.999+0000 7f5b82800640 1 --2- 172.21.15.102:0/3026498748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 0x7f5b840caca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009e50 tx=0x7f5b7c02bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8d709e8783da17f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.003+0000 7f5b80a00640 1 -- 172.21.15.102:0/3026498748 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c02cb90 con 0x7f5b840cc1f0 2024-03-18T16:22:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.003+0000 7f5b80a00640 1 -- 172.21.15.102:0/3026498748 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b7c02cd30 con 0x7f5b840cc1f0 2024-03-18T16:22:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.003+0000 7f5b80a00640 1 -- 172.21.15.102:0/3026498748 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c034610 con 0x7f5b840cc1f0 2024-03-18T16:22:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.004+0000 7f5b89200640 1 -- 172.21.15.102:0/3026498748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 msgr2=0x7f5b840caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.004+0000 7f5b89200640 1 --2- 172.21.15.102:0/3026498748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 0x7f5b840caca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009e50 tx=0x7f5b7c02bff0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.004+0000 7f5b89200640 1 -- 172.21.15.102:0/3026498748 shutdown_connections 2024-03-18T16:22:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.004+0000 7f5b89200640 1 --2- 172.21.15.102:0/3026498748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 0x7f5b840caca0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.004+0000 7f5b89200640 1 --2- 172.21.15.102:0/3026498748 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 0x7f5b8415a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.004+0000 7f5b89200640 1 -- 172.21.15.102:0/3026498748 >> 172.21.15.102:0/3026498748 conn(0x7f5b84067160 msgr2=0x7f5b84067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:18.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.004+0000 7f5b89200640 1 -- 172.21.15.102:0/3026498748 shutdown_connections 2024-03-18T16:22:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.004+0000 7f5b89200640 1 -- 172.21.15.102:0/3026498748 wait complete. 2024-03-18T16:22:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.006+0000 7f5b89200640 1 Processor -- start 2024-03-18T16:22:18.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.006+0000 7f5b89200640 1 -- start start 2024-03-18T16:22:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.007+0000 7f5b89200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 0x7f5b840cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.007+0000 7f5b89200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 0x7f5b8415b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.007+0000 7f5b89200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b84154780 0x7f5b8415bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.007+0000 7f5b89200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8415dd40 con 0x7f5b84154780 2024-03-18T16:22:18.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.007+0000 7f5b89200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8415b090 con 0x7f5b840cc1f0 2024-03-18T16:22:18.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.007+0000 7f5b89200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5b8415b370 con 0x7f5b840c5050 2024-03-18T16:22:18.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.007+0000 7f5b81400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 0x7f5b8415b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.008+0000 7f5b81e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 0x7f5b840cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.008+0000 7f5b82800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b84154780 0x7f5b8415bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.008+0000 7f5b81400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 0x7f5b8415b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47670/0 (socket says 172.21.15.102:47670) 2024-03-18T16:22:18.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.008+0000 7f5b81400640 1 -- 172.21.15.102:0/2172910601 learned_addr learned my addr 172.21.15.102:0/2172910601 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.011+0000 7f5b81e00640 1 -- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 msgr2=0x7f5b8415b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.011+0000 7f5b81e00640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 0x7f5b8415b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.011+0000 7f5b81e00640 1 -- 172.21.15.102:0/2172910601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b84154780 msgr2=0x7f5b8415bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.011+0000 7f5b81e00640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b84154780 0x7f5b8415bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.011+0000 7f5b81e00640 1 -- 172.21.15.102:0/2172910601 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b84153c50 con 0x7f5b840c5050 2024-03-18T16:22:18.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.014+0000 7f5b81e00640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 0x7f5b840cee10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b8415dc70 tx=0x7f5b7400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.017+0000 7f5b7a000640 1 -- 172.21.15.102:0/2172910601 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74017070 con 0x7f5b840c5050 2024-03-18T16:22:18.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.017+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b84158db0 con 0x7f5b840c5050 2024-03-18T16:22:18.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.018+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b84159370 con 0x7f5b840c5050 2024-03-18T16:22:18.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.018+0000 7f5b7a000640 1 -- 172.21.15.102:0/2172910601 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b740052d0 con 0x7f5b840c5050 2024-03-18T16:22:18.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.018+0000 7f5b7a000640 1 -- 172.21.15.102:0/2172910601 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b74021060 con 0x7f5b840c5050 2024-03-18T16:22:18.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.019+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b5c005290 con 0x7f5b840c5050 2024-03-18T16:22:18.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.038+0000 7f5b7a000640 1 -- 172.21.15.102:0/2172910601 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5b7401a070 con 0x7f5b840c5050 2024-03-18T16:22:18.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.039+0000 7f5b7a000640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5b6406be00 0x7f5b6406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.040+0000 7f5b81400640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5b6406be00 0x7f5b6406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.041+0000 7f5b81400640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5b6406be00 0x7f5b6406e2c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5b8415a570 tx=0x7f5b70009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.041+0000 7f5b7a000640 1 -- 172.21.15.102:0/2172910601 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f5b74003610 con 0x7f5b840c5050 2024-03-18T16:22:18.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.077+0000 7facd4600640 1 Processor -- start 2024-03-18T16:22:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.077+0000 7facd4600640 1 -- start start 2024-03-18T16:22:18.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-18T16:22:18.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.079+0000 7facd4600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faccc0c8a90 0x7faccc0c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.079+0000 7facd4600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faccc15a870 0x7faccc159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.079+0000 7facd4600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faccc157280 con 0x7faccc0c86b0 2024-03-18T16:22:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.080+0000 7facd4600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faccc158770 con 0x7faccc15a870 2024-03-18T16:22:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.080+0000 7facd4600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faccc15adb0 con 0x7faccc0c8a90 2024-03-18T16:22:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.080+0000 7facd1c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7faccc0c86b0 0x7faccc157a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60066/0 (socket says 172.21.15.102:60066) 2024-03-18T16:22:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.080+0000 7facd1c00640 1 -- 172.21.15.102:0/884870180 learned_addr learned my addr 172.21.15.102:0/884870180 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.080+0000 7facd2600640 1 --2- 172.21.15.102:0/884870180 >> v2:172.21.15.154:3300/0 conn(0x7faccc15a870 0x7faccc159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.083+0000 7faccbe00640 1 -- 172.21.15.102:0/884870180 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 266592206 0 0) 0x7facc0001110 con 0x7faccc0c86b0 2024-03-18T16:22:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.083+0000 7faccbe00640 1 -- 172.21.15.102:0/884870180 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facb4005730 con 0x7faccc0c86b0 2024-03-18T16:22:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.084+0000 7facd2600640 1 -- 172.21.15.102:0/884870180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faccc0c8a90 msgr2=0x7faccc0c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.084+0000 7facd2600640 1 --2- 172.21.15.102:0/884870180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faccc0c8a90 0x7faccc0c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.084+0000 7facd2600640 1 -- 172.21.15.102:0/884870180 >> v1:172.21.15.102:6789/0 conn(0x7faccc0c86b0 legacy=0x7faccc157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.084+0000 7facd2600640 1 -- 172.21.15.102:0/884870180 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faccc15b070 con 0x7faccc15a870 2024-03-18T16:22:18.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.087+0000 7facd2600640 1 --2- 172.21.15.102:0/884870180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faccc15a870 0x7faccc159cf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7facc4009d90 tx=0x7facc402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84dc996eca4a3944 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.092+0000 7faccbe00640 1 -- 172.21.15.102:0/884870180 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc4039070 con 0x7faccc15a870 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.092+0000 7faccbe00640 1 -- 172.21.15.102:0/884870180 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc402cd00 con 0x7faccc15a870 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.092+0000 7faccbe00640 1 -- 172.21.15.102:0/884870180 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc4034660 con 0x7faccc15a870 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.093+0000 7facd4600640 1 -- 172.21.15.102:0/884870180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faccc15a870 msgr2=0x7faccc159cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.093+0000 7facd4600640 1 --2- 172.21.15.102:0/884870180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faccc15a870 0x7faccc159cf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7facc4009d90 tx=0x7facc402bf50 comp rx=0 tx=0).stop 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.093+0000 7facd4600640 1 -- 172.21.15.102:0/884870180 shutdown_connections 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.093+0000 7facd4600640 1 --2- 172.21.15.102:0/884870180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faccc15a870 0x7faccc159cf0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.093+0000 7facd4600640 1 --2- 172.21.15.102:0/884870180 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faccc0c8a90 0x7faccc0c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.093+0000 7facd4600640 1 -- 172.21.15.102:0/884870180 >> 172.21.15.102:0/884870180 conn(0x7faccc0700d0 msgr2=0x7faccc06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.093+0000 7facd4600640 1 -- 172.21.15.102:0/884870180 shutdown_connections 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.093+0000 7facd4600640 1 -- 172.21.15.102:0/884870180 wait complete. 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.095+0000 7facd4600640 1 Processor -- start 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.095+0000 7facd4600640 1 -- start start 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.096+0000 7facd4600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0c86b0 0x7faccc1f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.096+0000 7facd4600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faccc0c8a90 0x7faccc071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.096+0000 7facd4600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faccc15a870 0x7faccc1f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.096+0000 7facd4600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faccc157500 con 0x7faccc0c86b0 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.096+0000 7facd4600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faccc15b630 con 0x7faccc15a870 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.096+0000 7facd4600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faccc158130 con 0x7faccc0c8a90 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.097+0000 7facd1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0c86b0 0x7faccc1f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.097+0000 7facd2600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faccc15a870 0x7faccc1f1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.097+0000 7facd1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0c86b0 0x7faccc1f32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51998/0 (socket says 172.21.15.102:51998) 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.097+0000 7facd1c00640 1 -- 172.21.15.102:0/4039851687 learned_addr learned my addr 172.21.15.102:0/4039851687 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.100+0000 7facd1c00640 1 -- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faccc0c8a90 msgr2=0x7faccc071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.100+0000 7facd1c00640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faccc0c8a90 0x7faccc071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.100+0000 7facd1c00640 1 -- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faccc15a870 msgr2=0x7faccc1f1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.100+0000 7facd1c00640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faccc15a870 0x7faccc1f1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.100+0000 7facd1c00640 1 -- 172.21.15.102:0/4039851687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faccc1f3970 con 0x7faccc0c86b0 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.103+0000 7facd1c00640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0c86b0 0x7faccc1f32c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7facc000c730 tx=0x7facc000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.107+0000 7facc9600640 1 -- 172.21.15.102:0/4039851687 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0017070 con 0x7faccc0c86b0 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.107+0000 7facc9600640 1 -- 172.21.15.102:0/4039851687 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc0005ad0 con 0x7faccc0c86b0 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.107+0000 7facc9600640 1 -- 172.21.15.102:0/4039851687 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0007e40 con 0x7faccc0c86b0 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.107+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faccc1f3c50 con 0x7faccc0c86b0 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.107+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faccc1f1360 con 0x7faccc0c86b0 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.109+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faccc1f4170 con 0x7faccc0c86b0 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.128+0000 7facc9600640 1 -- 172.21.15.102:0/4039851687 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7facc001a070 con 0x7faccc0c86b0 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.129+0000 7facc9600640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7facac06bbd0 0x7facac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.130+0000 7facd1200640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7facac06bbd0 0x7facac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.131+0000 7facd1200640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7facac06bbd0 0x7facac06e090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faccc0c5660 tx=0x7facb8006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.133+0000 7facc9600640 1 -- 172.21.15.102:0/4039851687 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7facc00567a0 con 0x7faccc0c86b0 2024-03-18T16:22:18.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-18T16:22:18.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-18T16:22:18.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.298+0000 7f5b7a000640 1 -- 172.21.15.102:0/2172910601 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5b740586b0 con 0x7f5b840c5050 2024-03-18T16:22:18.330 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-18T16:22:18.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.457+0000 7f0387200640 1 Processor -- start 2024-03-18T16:22:18.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.457+0000 7f0387200640 1 -- start start 2024-03-18T16:22:18.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.460+0000 7f0387200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f03880caab0 0x7f03880cff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.460+0000 7f0387200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f03880cf3a0 0x7f038815a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.461+0000 7f0387200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03880cae90 con 0x7f038815a9b0 2024-03-18T16:22:18.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.461+0000 7f0385400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f03880cf3a0 0x7f038815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.461+0000 7f0387200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f038815e890 con 0x7f03880caab0 2024-03-18T16:22:18.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.462+0000 7f0385400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f03880cf3a0 0x7f038815a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47692/0 (socket says 172.21.15.102:47692) 2024-03-18T16:22:18.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.462+0000 7f0385400640 1 -- 172.21.15.102:0/3658965624 learned_addr learned my addr 172.21.15.102:0/3658965624 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.462+0000 7f0387200640 1 -- 172.21.15.102:0/3658965624 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f03880d08c0 con 0x7f03880cf3a0 2024-03-18T16:22:18.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.465+0000 7f0385400640 1 -- 172.21.15.102:0/3658965624 >> v2:172.21.15.102:3300/0 conn(0x7f03880caab0 msgr2=0x7f03880cff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:18.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.465+0000 7f0385400640 1 --2- 172.21.15.102:0/3658965624 >> v2:172.21.15.102:3300/0 conn(0x7f03880caab0 0x7f03880cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.465+0000 7f0385400640 1 -- 172.21.15.102:0/3658965624 >> v1:172.21.15.102:6789/0 conn(0x7f038815a9b0 legacy=0x7f038806d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:18.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.465+0000 7f0385400640 1 -- 172.21.15.102:0/3658965624 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03880cf130 con 0x7f03880cf3a0 2024-03-18T16:22:18.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.475+0000 7f0385400640 1 --2- 172.21.15.102:0/3658965624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03880cf3a0 0x7f038815a470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0388154db0 tx=0x7f037802b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eace510cf320d7bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.481+0000 7f0384a00640 1 -- 172.21.15.102:0/3658965624 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378035020 con 0x7f03880cf3a0 2024-03-18T16:22:18.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.481+0000 7f0384a00640 1 -- 172.21.15.102:0/3658965624 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0378031800 con 0x7f03880cf3a0 2024-03-18T16:22:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.481+0000 7f0384a00640 1 -- 172.21.15.102:0/3658965624 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0378031c10 con 0x7f03880cf3a0 2024-03-18T16:22:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.483+0000 7f0387200640 1 -- 172.21.15.102:0/3658965624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03880cf3a0 msgr2=0x7f038815a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.483+0000 7f0387200640 1 --2- 172.21.15.102:0/3658965624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03880cf3a0 0x7f038815a470 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0388154db0 tx=0x7f037802b8a0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.483+0000 7f0387200640 1 -- 172.21.15.102:0/3658965624 shutdown_connections 2024-03-18T16:22:18.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.483+0000 7f0387200640 1 --2- 172.21.15.102:0/3658965624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03880cf3a0 0x7f038815a470 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.484+0000 7f0387200640 1 --2- 172.21.15.102:0/3658965624 >> v2:172.21.15.102:3300/0 conn(0x7f03880caab0 0x7f03880cff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.484+0000 7f0387200640 1 -- 172.21.15.102:0/3658965624 >> 172.21.15.102:0/3658965624 conn(0x7f0388067160 msgr2=0x7f0388067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.484+0000 7f0387200640 1 -- 172.21.15.102:0/3658965624 shutdown_connections 2024-03-18T16:22:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.484+0000 7f0387200640 1 -- 172.21.15.102:0/3658965624 wait complete. 2024-03-18T16:22:18.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.488+0000 7f0387200640 1 Processor -- start 2024-03-18T16:22:18.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.488+0000 7f0387200640 1 -- start start 2024-03-18T16:22:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.490+0000 7f0387200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03880caab0 0x7f03881f8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.491+0000 7f0387200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03880cf3a0 0x7f03880b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.491+0000 7f0387200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f038815a9b0 0x7f03881e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.491+0000 7f0387200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f038815ddb0 con 0x7f03880caab0 2024-03-18T16:22:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.491+0000 7f0387200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f038815db10 con 0x7f03880cf3a0 2024-03-18T16:22:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.491+0000 7f0387200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f038815e090 con 0x7f038815a9b0 2024-03-18T16:22:18.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.495+0000 7f0975800640 1 Processor -- start 2024-03-18T16:22:18.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.492+0000 7f0386800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f038815a9b0 0x7f03881e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.492+0000 7f0386800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f038815a9b0 0x7f03881e5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34364/0 (socket says 172.21.15.102:34364) 2024-03-18T16:22:18.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.492+0000 7f0386800640 1 -- 172.21.15.102:0/2433927156 learned_addr learned my addr 172.21.15.102:0/2433927156 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.493+0000 7f0385400640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03880cf3a0 0x7f03880b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.496+0000 7f0975800640 1 -- start start 2024-03-18T16:22:18.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.498+0000 7f0975800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09701508d0 0x7f09700730f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.498+0000 7f0975800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f09700c7bc0 0x7f09700c8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.498+0000 7f0975800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c8000 0x7f09700c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.498+0000 7f0975800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f097006d830 con 0x7f09700c7bc0 2024-03-18T16:22:18.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.496+0000 7f0386800640 1 -- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03880cf3a0 msgr2=0x7f03880b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.496+0000 7f0386800640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03880cf3a0 0x7f03880b1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.496+0000 7f0386800640 1 -- 172.21.15.102:0/2433927156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03880caab0 msgr2=0x7f03881f8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.496+0000 7f0386800640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03880caab0 0x7f03881f8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.496+0000 7f0386800640 1 -- 172.21.15.102:0/2433927156 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03881e38a0 con 0x7f038815a9b0 2024-03-18T16:22:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.499+0000 7f096ea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09701508d0 0x7f09700730f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.499+0000 7f0975800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f097006db10 con 0x7f09701508d0 2024-03-18T16:22:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.499+0000 7f0975800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f097006ddf0 con 0x7f09700c8000 2024-03-18T16:22:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.499+0000 7f096f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c8000 0x7f09700c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.500+0000 7f096ea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09701508d0 0x7f09700730f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47708/0 (socket says 172.21.15.102:47708) 2024-03-18T16:22:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.500+0000 7f096ea00640 1 -- 172.21.15.102:0/3045374829 learned_addr learned my addr 172.21.15.102:0/3045374829 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.501+0000 7f0386800640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f038815a9b0 0x7f03881e5f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f037400c390 tx=0x7f037400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.503+0000 7f096f400640 1 -- 172.21.15.102:0/3045374829 >> v2:172.21.15.154:3300/0 conn(0x7f09701508d0 msgr2=0x7f09700730f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:18.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.503+0000 7f096f400640 1 --2- 172.21.15.102:0/3045374829 >> v2:172.21.15.154:3300/0 conn(0x7f09701508d0 0x7f09700730f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.503+0000 7f096f400640 1 -- 172.21.15.102:0/3045374829 >> v2:172.21.15.102:3300/0 conn(0x7f09700c7bc0 msgr2=0x7f09700c8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:18.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.503+0000 7f096f400640 1 --2- 172.21.15.102:0/3045374829 >> v2:172.21.15.102:3300/0 conn(0x7f09700c7bc0 0x7f09700c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.503+0000 7f096f400640 1 -- 172.21.15.102:0/3045374829 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097006e0d0 con 0x7f09700c8000 2024-03-18T16:22:18.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.505+0000 7f096ea00640 1 --2- 172.21.15.102:0/3045374829 >> v2:172.21.15.154:3300/0 conn(0x7f09701508d0 0x7f09700730f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:22:18.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.509+0000 7f037e000640 1 -- 172.21.15.102:0/2433927156 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374017070 con 0x7f038815a9b0 2024-03-18T16:22:18.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.509+0000 7f096f400640 1 --2- 172.21.15.102:0/3045374829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c8000 0x7f09700c8b60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f09700721b0 tx=0x7f095c02b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70551be785e342e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.509+0000 7f0387200640 1 -- 172.21.15.102:0/2433927156 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03881f8db0 con 0x7f038815a9b0 2024-03-18T16:22:18.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.509+0000 7f0387200640 1 -- 172.21.15.102:0/2433927156 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03881f8070 con 0x7f038815a9b0 2024-03-18T16:22:18.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.512+0000 7f0373400640 1 -- 172.21.15.102:0/2433927156 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0360005290 con 0x7f038815a9b0 2024-03-18T16:22:18.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.512+0000 7f037e000640 1 -- 172.21.15.102:0/2433927156 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03740035f0 con 0x7f038815a9b0 2024-03-18T16:22:18.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.512+0000 7f037e000640 1 -- 172.21.15.102:0/2433927156 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03740039e0 con 0x7f038815a9b0 2024-03-18T16:22:18.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.514+0000 7f096d600640 1 -- 172.21.15.102:0/3045374829 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c035020 con 0x7f09700c8000 2024-03-18T16:22:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.514+0000 7f096d600640 1 -- 172.21.15.102:0/3045374829 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f095c031870 con 0x7f09700c8000 2024-03-18T16:22:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.515+0000 7f096d600640 1 -- 172.21.15.102:0/3045374829 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c031c80 con 0x7f09700c8000 2024-03-18T16:22:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.516+0000 7f0975800640 1 -- 172.21.15.102:0/3045374829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c8000 msgr2=0x7f09700c8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.516+0000 7f0975800640 1 --2- 172.21.15.102:0/3045374829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c8000 0x7f09700c8b60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f09700721b0 tx=0x7f095c02b910 comp rx=0 tx=0).stop 2024-03-18T16:22:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.516+0000 7f0975800640 1 -- 172.21.15.102:0/3045374829 shutdown_connections 2024-03-18T16:22:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.516+0000 7f0975800640 1 --2- 172.21.15.102:0/3045374829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c8000 0x7f09700c8b60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.516+0000 7f0975800640 1 --2- 172.21.15.102:0/3045374829 >> v2:172.21.15.102:3300/0 conn(0x7f09700c7bc0 0x7f09700c8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.516+0000 7f0975800640 1 --2- 172.21.15.102:0/3045374829 >> v2:172.21.15.154:3300/0 conn(0x7f09701508d0 0x7f09700730f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.517+0000 7f0975800640 1 -- 172.21.15.102:0/3045374829 >> 172.21.15.102:0/3045374829 conn(0x7f09700700d0 msgr2=0x7f097006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:18.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.517+0000 7f0975800640 1 -- 172.21.15.102:0/3045374829 shutdown_connections 2024-03-18T16:22:18.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.517+0000 7f0975800640 1 -- 172.21.15.102:0/3045374829 wait complete. 2024-03-18T16:22:18.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.520+0000 7f0975800640 1 Processor -- start 2024-03-18T16:22:18.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.521+0000 7f0975800640 1 -- start start 2024-03-18T16:22:18.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.523+0000 7f0975800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c7bc0 0x7f09701f0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.523+0000 7f0975800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09701508d0 0x7f09700cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.523+0000 7f0975800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09701f1030 0x7f09700cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.524+0000 7f0975800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f097006b9b0 con 0x7f09701508d0 2024-03-18T16:22:18.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.524+0000 7f0975800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0970073630 con 0x7f09701f1030 2024-03-18T16:22:18.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.524+0000 7f0975800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f097006bb30 con 0x7f09700c7bc0 2024-03-18T16:22:18.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.524+0000 7f096f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09701f1030 0x7f09700cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.525+0000 7f096f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09701f1030 0x7f09700cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47714/0 (socket says 172.21.15.102:47714) 2024-03-18T16:22:18.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.525+0000 7f096f400640 1 -- 172.21.15.102:0/14197978 learned_addr learned my addr 172.21.15.102:0/14197978 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.528+0000 7f096f400640 1 -- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c7bc0 msgr2=0x7f09701f0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.528+0000 7f096f400640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c7bc0 0x7f09701f0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.528+0000 7f096f400640 1 -- 172.21.15.102:0/14197978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09701508d0 msgr2=0x7f09700cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.528+0000 7f096f400640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09701508d0 0x7f09700cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.528+0000 7f096f400640 1 -- 172.21.15.102:0/14197978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0970147ce0 con 0x7f09701f1030 2024-03-18T16:22:18.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.531+0000 7f096f400640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09701f1030 0x7f09700cb870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f095c009890 tx=0x7f095c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-18T16:22:18.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.534+0000 7f0966a00640 1 -- 172.21.15.102:0/14197978 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c041070 con 0x7f09701f1030 2024-03-18T16:22:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.534+0000 7f0966a00640 1 -- 172.21.15.102:0/14197978 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f095c03ab50 con 0x7f09701f1030 2024-03-18T16:22:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.534+0000 7f0966a00640 1 -- 172.21.15.102:0/14197978 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c035040 con 0x7f09701f1030 2024-03-18T16:22:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.535+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097014fd40 con 0x7f09701f1030 2024-03-18T16:22:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.535+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097014f780 con 0x7f09701f1030 2024-03-18T16:22:18.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.540+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0970074fd0 con 0x7f09701f1030 2024-03-18T16:22:18.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.545+0000 7facc9600640 1 -- 172.21.15.102:0/4039851687 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faccc1f4170 con 0x7faccc0c86b0 2024-03-18T16:22:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.560+0000 7f0966a00640 1 -- 172.21.15.102:0/14197978 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f095c038020 con 0x7f09701f1030 2024-03-18T16:22:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.562+0000 7f0966a00640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f095006bc00 0x7f095006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.562+0000 7f0966a00640 1 -- 172.21.15.102:0/14197978 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f095c0ad590 con 0x7f09701f1030 2024-03-18T16:22:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.562+0000 7f096ea00640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f095006bc00 0x7f095006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.565+0000 7f096ea00640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f095006bc00 0x7f095006e0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0960009630 tx=0x7f0960006e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.618+0000 7fbc2b200640 1 Processor -- start 2024-03-18T16:22:18.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.618+0000 7fbc2b200640 1 -- start start 2024-03-18T16:22:18.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.620+0000 7fbc2b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbc240c7c50 0x7fbc240c9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.620+0000 7fbc2b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2406dd90 0x7fbc2406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.621+0000 7fbc2b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2406db70 con 0x7fbc24158970 2024-03-18T16:22:18.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.621+0000 7fbc2b200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbc240c6960 con 0x7fbc240c7c50 2024-03-18T16:22:18.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.621+0000 7fbc2b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc240c77c0 con 0x7fbc2406dd90 2024-03-18T16:22:18.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.623+0000 7fbc23400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbc240c7c50 0x7fbc240c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.623+0000 7fbc23400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbc240c7c50 0x7fbc240c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52052/0 (socket says 172.21.15.102:52052) 2024-03-18T16:22:18.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.623+0000 7fbc23400640 1 -- 172.21.15.102:0/4029017270 learned_addr learned my addr 172.21.15.102:0/4029017270 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.629+0000 7fbc23400640 1 -- 172.21.15.102:0/4029017270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2406dd90 msgr2=0x7fbc2406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:18.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.629+0000 7fbc23400640 1 --2- 172.21.15.102:0/4029017270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2406dd90 0x7fbc2406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.629+0000 7fbc23400640 1 -- 172.21.15.102:0/4029017270 >> v1:172.21.15.102:6789/0 conn(0x7fbc24158970 legacy=0x7fbc2415f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.629+0000 7fbc23400640 1 -- 172.21.15.102:0/4029017270 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc240c6bb0 con 0x7fbc240c7c50 2024-03-18T16:22:18.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.638+0000 7fbc23400640 1 --2- 172.21.15.102:0/4029017270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc240c7c50 0x7fbc240c9690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009870 tx=0x7fbc1402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c1b78474e99bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.643+0000 7fbc22a00640 1 -- 172.21.15.102:0/4029017270 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1402f850 con 0x7fbc240c7c50 2024-03-18T16:22:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.649+0000 7fbc22a00640 1 -- 172.21.15.102:0/4029017270 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1402f9f0 con 0x7fbc240c7c50 2024-03-18T16:22:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.649+0000 7fbc22a00640 1 -- 172.21.15.102:0/4029017270 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1402fe00 con 0x7fbc240c7c50 2024-03-18T16:22:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.650+0000 7fbc2b200640 1 -- 172.21.15.102:0/4029017270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc240c7c50 msgr2=0x7fbc240c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.650+0000 7fbc2b200640 1 --2- 172.21.15.102:0/4029017270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc240c7c50 0x7fbc240c9690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009870 tx=0x7fbc1402ba30 comp rx=0 tx=0).stop 2024-03-18T16:22:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.653+0000 7fbc2b200640 1 -- 172.21.15.102:0/4029017270 shutdown_connections 2024-03-18T16:22:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.653+0000 7fbc2b200640 1 --2- 172.21.15.102:0/4029017270 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc2406dd90 0x7fbc2406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.653+0000 7fbc2b200640 1 --2- 172.21.15.102:0/4029017270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc240c7c50 0x7fbc240c9690 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.653+0000 7fbc2b200640 1 -- 172.21.15.102:0/4029017270 >> 172.21.15.102:0/4029017270 conn(0x7fbc240c8390 msgr2=0x7fbc240c8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.653+0000 7fbc2b200640 1 -- 172.21.15.102:0/4029017270 shutdown_connections 2024-03-18T16:22:18.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.653+0000 7fbc2b200640 1 -- 172.21.15.102:0/4029017270 wait complete. 2024-03-18T16:22:18.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.656+0000 7fbc2b200640 1 Processor -- start 2024-03-18T16:22:18.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.657+0000 7fbc2b200640 1 -- start start 2024-03-18T16:22:18.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.659+0000 7fbc2b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc2406dd90 0x7fbc241f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.659+0000 7fbc2b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc240c7c50 0x7fbc240c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.659+0000 7fbc2b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc24158970 0x7fbc241602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.660+0000 7fbc2b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbc240c7450 con 0x7fbc24158970 2024-03-18T16:22:18.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.660+0000 7fbc2b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc240c7170 con 0x7fbc2406dd90 2024-03-18T16:22:18.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.660+0000 7fbc2b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc24158d50 con 0x7fbc240c7c50 2024-03-18T16:22:18.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.662+0000 7fbc29200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc24158970 0x7fbc241602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.663+0000 7fbc29200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc24158970 0x7fbc241602d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52062/0 (socket says 172.21.15.102:52062) 2024-03-18T16:22:18.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.663+0000 7fbc29200640 1 -- 172.21.15.102:0/1845492288 learned_addr learned my addr 172.21.15.102:0/1845492288 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.670+0000 7fbc29200640 1 -- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc240c7c50 msgr2=0x7fbc240c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.670+0000 7fbc29200640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc240c7c50 0x7fbc240c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.670+0000 7fbc29200640 1 -- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc2406dd90 msgr2=0x7fbc241f4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.670+0000 7fbc29200640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc2406dd90 0x7fbc241f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.670+0000 7fbc29200640 1 -- 172.21.15.102:0/1845492288 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc24161440 con 0x7fbc24158970 2024-03-18T16:22:18.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.673+0000 7f8ce6e00640 1 Processor -- start 2024-03-18T16:22:18.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.674+0000 7f8ce6e00640 1 -- start start 2024-03-18T16:22:18.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.674+0000 7fbc29200640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc24158970 0x7fbc241602d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009fd0 tx=0x7fbc1000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.676+0000 7f8ce6e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8ce00cad80 0x7f8ce006dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.677+0000 7f8ce6e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00cb160 0x7f8ce01501f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.677+0000 7f8ce6e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ce0156360 con 0x7f8ce00c8e80 2024-03-18T16:22:18.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.677+0000 7f8ce6e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8ce0156620 con 0x7f8ce00cad80 2024-03-18T16:22:18.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.678+0000 7f8ce6e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8ce0156900 con 0x7f8ce00cb160 2024-03-18T16:22:18.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.683+0000 7f8cdfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8ce00cad80 0x7f8ce006dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.684+0000 7fbc1be00640 1 -- 172.21.15.102:0/1845492288 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc10011830 con 0x7fbc24158970 2024-03-18T16:22:18.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.684+0000 7fbc1be00640 1 -- 172.21.15.102:0/1845492288 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc10016070 con 0x7fbc24158970 2024-03-18T16:22:18.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.684+0000 7fbc1be00640 1 -- 172.21.15.102:0/1845492288 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc10011830 con 0x7fbc24158970 2024-03-18T16:22:18.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.684+0000 7f8cdfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8ce00cad80 0x7f8ce006dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52068/0 (socket says 172.21.15.102:52068) 2024-03-18T16:22:18.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.684+0000 7f8cdfe00640 1 -- 172.21.15.102:0/1674300229 learned_addr learned my addr 172.21.15.102:0/1674300229 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.689+0000 7fbc2b200640 1 -- 172.21.15.102:0/1845492288 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc24161720 con 0x7fbc24158970 2024-03-18T16:22:18.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.689+0000 7fbc2b200640 1 -- 172.21.15.102:0/1845492288 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2415fad0 con 0x7fbc24158970 2024-03-18T16:22:18.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.690+0000 7fbc19600640 1 -- 172.21.15.102:0/1845492288 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf8005290 con 0x7fbc24158970 2024-03-18T16:22:18.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.691+0000 7f8cdfe00640 1 -- 172.21.15.102:0/1674300229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00cb160 msgr2=0x7f8ce01501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:18.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.691+0000 7f8cdfe00640 1 --2- 172.21.15.102:0/1674300229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00cb160 0x7f8ce01501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.691+0000 7f8cdfe00640 1 -- 172.21.15.102:0/1674300229 >> v1:172.21.15.154:6789/0 conn(0x7f8ce00c8e80 legacy=0x7f8ce00c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:18.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.691+0000 7f8cdfe00640 1 -- 172.21.15.102:0/1674300229 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce0156be0 con 0x7f8ce00cad80 2024-03-18T16:22:18.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.697+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5b5c005850 con 0x7f5b840c5050 2024-03-18T16:22:18.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.735+0000 7fbc1be00640 1 -- 172.21.15.102:0/1845492288 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbc100119d0 con 0x7fbc24158970 2024-03-18T16:22:18.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.736+0000 7fbc1be00640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbc0406bdd0 0x7fbc0406e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.736+0000 7fbc23e00640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbc0406bdd0 0x7fbc0406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.737+0000 7fbc23e00640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbc0406bdd0 0x7fbc0406e290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0012e0 tx=0x7fbc1c007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.789+0000 7f4524800640 1 Processor -- start 2024-03-18T16:22:18.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.789+0000 7f4524800640 1 -- start start 2024-03-18T16:22:18.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.791+0000 7f4524800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f451c1524c0 0x7f451c072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.791+0000 7f4524800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f451c15b8e0 0x7f451c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.791+0000 7f4524800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f451c15c200 con 0x7f451c158ee0 2024-03-18T16:22:18.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.791+0000 7f4524800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f451c159300 con 0x7f451c1524c0 2024-03-18T16:22:18.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.791+0000 7f4524800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f451c15c4e0 con 0x7f451c15b8e0 2024-03-18T16:22:18.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-18T16:22:18.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.819+0000 7f037e000640 1 -- 172.21.15.102:0/2433927156 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f037401a070 con 0x7f038815a9b0 2024-03-18T16:22:18.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.821+0000 7f037e000640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f036806bc20 0x7f036806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.821+0000 7f0385e00640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f036806bc20 0x7f036806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.822+0000 7f037e000640 1 -- 172.21.15.102:0/2433927156 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f0374056720 con 0x7f038815a9b0 2024-03-18T16:22:18.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.822+0000 7f0385e00640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f036806bc20 0x7f036806e0e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f038000ab90 tx=0x7f0380009250 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.843+0000 7f0966a00640 1 -- 172.21.15.102:0/14197978 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0970147ce0 con 0x7f09701f1030 2024-03-18T16:22:18.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.844+0000 7f4522800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f451c15b8e0 0x7f451c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.845+0000 7f4521400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f451c158ee0 0x7f451c06e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58570/0 (socket says 172.21.15.102:58570) 2024-03-18T16:22:18.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.845+0000 7f4521400640 1 -- 172.21.15.102:0/643309755 learned_addr learned my addr 172.21.15.102:0/643309755 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:18.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.854+0000 7f4522800640 1 -- 172.21.15.102:0/643309755 >> v2:172.21.15.102:3300/0 conn(0x7f451c1524c0 msgr2=0x7f451c072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:18.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.854+0000 7f4522800640 1 --2- 172.21.15.102:0/643309755 >> v2:172.21.15.102:3300/0 conn(0x7f451c1524c0 0x7f451c072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.854+0000 7f4522800640 1 -- 172.21.15.102:0/643309755 >> v1:172.21.15.154:6789/0 conn(0x7f451c158ee0 legacy=0x7f451c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.854+0000 7f4522800640 1 -- 172.21.15.102:0/643309755 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f451c15c7c0 con 0x7f451c15b8e0 2024-03-18T16:22:18.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.859+0000 7f8cdfe00640 1 --2- 172.21.15.102:0/1674300229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00cad80 0x7f8ce006dfe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000bb10 tx=0x7f8cd002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ec251bcf406328 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.860+0000 7f4522800640 1 --2- 172.21.15.102:0/643309755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c15b8e0 0x7f451c0c51e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f451c06c090 tx=0x7f451802bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=601b89585f3f6643 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.863+0000 7f4520a00640 1 -- 172.21.15.102:0/643309755 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518039070 con 0x7f451c15b8e0 2024-03-18T16:22:18.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.866+0000 7f4520a00640 1 -- 172.21.15.102:0/643309755 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f451802cc70 con 0x7f451c15b8e0 2024-03-18T16:22:18.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.867+0000 7f4524800640 1 -- 172.21.15.102:0/643309755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c15b8e0 msgr2=0x7f451c0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.867+0000 7f4524800640 1 --2- 172.21.15.102:0/643309755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c15b8e0 0x7f451c0c51e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f451c06c090 tx=0x7f451802bec0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.867+0000 7f4524800640 1 -- 172.21.15.102:0/643309755 shutdown_connections 2024-03-18T16:22:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.868+0000 7f4524800640 1 --2- 172.21.15.102:0/643309755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c15b8e0 0x7f451c0c51e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.868+0000 7f4524800640 1 --2- 172.21.15.102:0/643309755 >> v2:172.21.15.102:3300/0 conn(0x7f451c1524c0 0x7f451c072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.868+0000 7f4524800640 1 -- 172.21.15.102:0/643309755 >> 172.21.15.102:0/643309755 conn(0x7f451c0700d0 msgr2=0x7f451c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:18.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.868+0000 7f4524800640 1 -- 172.21.15.102:0/643309755 shutdown_connections 2024-03-18T16:22:18.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.868+0000 7f4524800640 1 -- 172.21.15.102:0/643309755 wait complete. 2024-03-18T16:22:18.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.873+0000 7f4524800640 1 Processor -- start 2024-03-18T16:22:18.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.874+0000 7f4524800640 1 -- start start 2024-03-18T16:22:18.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.878+0000 7f037e000640 1 -- 172.21.15.102:0/2433927156 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0374021020 con 0x7f038815a9b0 2024-03-18T16:22:18.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.880+0000 7f5b7a000640 1 -- 172.21.15.102:0/2172910601 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5b7405c750 con 0x7f5b840c5050 2024-03-18T16:22:18.885 INFO:teuthology.orchestra.run.smithi102.stdout:60129542160 2024-03-18T16:22:18.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.884+0000 7fbc1be00640 1 -- 172.21.15.102:0/1845492288 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fbc10057330 con 0x7fbc24158970 2024-03-18T16:22:18.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5b6406be00 msgr2=0x7f5b6406e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5b6406be00 0x7f5b6406e2c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5b8415a570 tx=0x7f5b70009190 comp rx=0 tx=0).stop 2024-03-18T16:22:18.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f4524800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f451c1524c0 0x7f451c0c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 msgr2=0x7f5b840cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 0x7f5b840cee10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b8415dc70 tx=0x7f5b7400c820 comp rx=0 tx=0).stop 2024-03-18T16:22:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 shutdown_connections 2024-03-18T16:22:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5b6406be00 0x7f5b6406e2c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b84154780 0x7f5b8415bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b840cc1f0 0x7f5b8415b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 --2- 172.21.15.102:0/2172910601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5b840c5050 0x7f5b840cee10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 >> 172.21.15.102:0/2172910601 conn(0x7f5b84067160 msgr2=0x7f5b840c5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:18.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f4524800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c158ee0 0x7f451c152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.886+0000 7f4524800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f451c15b8e0 0x7f451c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:18.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.887+0000 7f4524800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f451c1589c0 con 0x7f451c15b8e0 2024-03-18T16:22:18.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.887+0000 7f4524800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f451c1586e0 con 0x7f451c158ee0 2024-03-18T16:22:18.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.887+0000 7f4524800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f451c1528a0 con 0x7f451c1524c0 2024-03-18T16:22:18.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.887+0000 7f4521400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c158ee0 0x7f451c152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.887+0000 7f4521400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c158ee0 0x7f451c152b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47752/0 (socket says 172.21.15.102:47752) 2024-03-18T16:22:18.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.887+0000 7f4521400640 1 -- 172.21.15.102:0/1879292372 learned_addr learned my addr 172.21.15.102:0/1879292372 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:18.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.883+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7faccc074fd0 con 0x7faccc0c86b0 2024-03-18T16:22:18.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.888+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 shutdown_connections 2024-03-18T16:22:18.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.888+0000 7f5b89200640 1 -- 172.21.15.102:0/2172910601 wait complete. 2024-03-18T16:22:18.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.889+0000 7f4521e00640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f451c1524c0 0x7f451c0c27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.895+0000 7f4521400640 1 -- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f451c1524c0 msgr2=0x7f451c0c27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.895+0000 7f4521400640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f451c1524c0 0x7f451c0c27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.895+0000 7f4521400640 1 -- 172.21.15.102:0/1879292372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f451c15b8e0 msgr2=0x7f451c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.895+0000 7f4521400640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f451c15b8e0 0x7f451c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:18.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.895+0000 7f4521400640 1 -- 172.21.15.102:0/1879292372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f451c154af0 con 0x7f451c158ee0 2024-03-18T16:22:18.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.897+0000 7f4521400640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c158ee0 0x7f451c152b40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f450c00c400 tx=0x7f450c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:18.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.899+0000 7f4516000640 1 -- 172.21.15.102:0/1879292372 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c017070 con 0x7f451c158ee0 2024-03-18T16:22:18.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.899+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f451c154db0 con 0x7f451c158ee0 2024-03-18T16:22:18.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.900+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f451c155370 con 0x7f451c158ee0 2024-03-18T16:22:18.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.905+0000 7f4516000640 1 -- 172.21.15.102:0/1879292372 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f450c005ce0 con 0x7f451c158ee0 2024-03-18T16:22:18.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.905+0000 7f4516000640 1 -- 172.21.15.102:0/1879292372 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c021060 con 0x7f451c158ee0 2024-03-18T16:22:18.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:18.905+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f451c074fd0 con 0x7f451c158ee0 2024-03-18T16:22:18.920 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542162 got 60129542160 for osd.2 2024-03-18T16:22:19.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.096+0000 7f488d600640 1 Processor -- start 2024-03-18T16:22:19.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.097+0000 7f488d600640 1 -- start start 2024-03-18T16:22:19.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.099+0000 7f488d600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4888154630 0x7f48880cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.099+0000 7f488d600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f488815a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.099+0000 7f488d600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48880cf5c0 con 0x7f48880ca510 2024-03-18T16:22:19.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.099+0000 7f488d600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f488806d640 con 0x7f4888154630 2024-03-18T16:22:19.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.099+0000 7f488d600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f488806d920 con 0x7f48880ca960 2024-03-18T16:22:19.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.100+0000 7f488cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f488815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:19.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.100+0000 7f488cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f488815a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34414/0 (socket says 172.21.15.102:34414) 2024-03-18T16:22:19.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.100+0000 7f488cc00640 1 -- 172.21.15.102:0/1392451144 learned_addr learned my addr 172.21.15.102:0/1392451144 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.104+0000 7f488cc00640 1 -- 172.21.15.102:0/1392451144 >> v2:172.21.15.102:3300/0 conn(0x7f4888154630 msgr2=0x7f48880cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.104+0000 7f488cc00640 1 --2- 172.21.15.102:0/1392451144 >> v2:172.21.15.102:3300/0 conn(0x7f4888154630 0x7f48880cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.104+0000 7f488cc00640 1 -- 172.21.15.102:0/1392451144 >> v1:172.21.15.102:6789/0 conn(0x7f48880ca510 legacy=0x7f48880cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:19.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.104+0000 7f488cc00640 1 -- 172.21.15.102:0/1392451144 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488806dc00 con 0x7f48880ca960 2024-03-18T16:22:19.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.108+0000 7f488cc00640 1 --2- 172.21.15.102:0/1392451144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f488815a520 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4880009e50 tx=0x7f488002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a7b56c3e50f1620 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:19.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.135+0000 7f4886a00640 1 -- 172.21.15.102:0/1392451144 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488002cb00 con 0x7f48880ca960 2024-03-18T16:22:19.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.136+0000 7f4886a00640 1 -- 172.21.15.102:0/1392451144 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488002cca0 con 0x7f48880ca960 2024-03-18T16:22:19.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.136+0000 7f4886a00640 1 -- 172.21.15.102:0/1392451144 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4880034610 con 0x7f48880ca960 2024-03-18T16:22:19.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.137+0000 7f488d600640 1 -- 172.21.15.102:0/1392451144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 msgr2=0x7f488815a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.137+0000 7f488d600640 1 --2- 172.21.15.102:0/1392451144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f488815a520 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4880009e50 tx=0x7f488002bf60 comp rx=0 tx=0).stop 2024-03-18T16:22:19.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.138+0000 7f488d600640 1 -- 172.21.15.102:0/1392451144 shutdown_connections 2024-03-18T16:22:19.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.138+0000 7f488d600640 1 --2- 172.21.15.102:0/1392451144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f488815a520 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.138+0000 7f488d600640 1 --2- 172.21.15.102:0/1392451144 >> v2:172.21.15.102:3300/0 conn(0x7f4888154630 0x7f48880cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.138+0000 7f488d600640 1 -- 172.21.15.102:0/1392451144 >> 172.21.15.102:0/1392451144 conn(0x7f4888067160 msgr2=0x7f4888067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:19.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.138+0000 7f488d600640 1 -- 172.21.15.102:0/1392451144 shutdown_connections 2024-03-18T16:22:19.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.138+0000 7f488d600640 1 -- 172.21.15.102:0/1392451144 wait complete. 2024-03-18T16:22:19.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.141+0000 7f488d600640 1 Processor -- start 2024-03-18T16:22:19.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.141+0000 7f488d600640 1 -- start start 2024-03-18T16:22:19.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.142+0000 7f488d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48880ca510 0x7f48881f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.142+0000 7f488d600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f48880b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.142+0000 7f488d600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4888154630 0x7f48880b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.142+0000 7f488d600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4888154b90 con 0x7f48880ca510 2024-03-18T16:22:19.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.142+0000 7f488d600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4888154a10 con 0x7f4888154630 2024-03-18T16:22:19.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.142+0000 7f488d600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f48880d0540 con 0x7f48880ca960 2024-03-18T16:22:19.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.143+0000 7f4887400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f48880b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:19.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.144+0000 7f4887400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f48880b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34428/0 (socket says 172.21.15.102:34428) 2024-03-18T16:22:19.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.144+0000 7f4887400640 1 -- 172.21.15.102:0/3751459348 learned_addr learned my addr 172.21.15.102:0/3751459348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:19.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.149+0000 7f4887400640 1 -- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4888154630 msgr2=0x7f48880b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.149+0000 7f4887400640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4888154630 0x7f48880b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.149+0000 7f4887400640 1 -- 172.21.15.102:0/3751459348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48880ca510 msgr2=0x7f48881f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.149+0000 7f4887400640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48880ca510 0x7f48881f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.149+0000 7f4887400640 1 -- 172.21.15.102:0/3751459348 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488815dc40 con 0x7f48880ca960 2024-03-18T16:22:19.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.152+0000 7f4887400640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f48880b32a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f487400c440 tx=0x7f487400c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:19.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.156+0000 7f487fe00640 1 -- 172.21.15.102:0/3751459348 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4874017070 con 0x7f48880ca960 2024-03-18T16:22:19.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.156+0000 7f487fe00640 1 -- 172.21.15.102:0/3751459348 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48740035f0 con 0x7f48880ca960 2024-03-18T16:22:19.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.156+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488815dee0 con 0x7f48880ca960 2024-03-18T16:22:19.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.156+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48880b2aa0 con 0x7f48880ca960 2024-03-18T16:22:19.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.156+0000 7f487fe00640 1 -- 172.21.15.102:0/3751459348 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48740039e0 con 0x7f48880ca960 2024-03-18T16:22:19.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.158+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4860005290 con 0x7f48880ca960 2024-03-18T16:22:19.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.178+0000 7f4516000640 1 -- 172.21.15.102:0/1879292372 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f450c01a070 con 0x7f451c158ee0 2024-03-18T16:22:19.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.181+0000 7f4516000640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f450006bc00 0x7f450006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.181+0000 7f4516000640 1 -- 172.21.15.102:0/1879292372 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f450c08ac20 con 0x7f451c158ee0 2024-03-18T16:22:19.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.181+0000 7f4521e00640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f450006bc00 0x7f450006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:19.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.182+0000 7f4521e00640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f450006bc00 0x7f450006e0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f451000b810 tx=0x7f4510008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:19.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.191+0000 7f487fe00640 1 -- 172.21.15.102:0/3751459348 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f487401a070 con 0x7f48880ca960 2024-03-18T16:22:19.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.192+0000 7f487fe00640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f486806bc20 0x7f486806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.193+0000 7f487fe00640 1 -- 172.21.15.102:0/3751459348 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f48740566f0 con 0x7f48880ca960 2024-03-18T16:22:19.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.193+0000 7f4516000640 1 -- 172.21.15.102:0/1879292372 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f450c058e30 con 0x7f451c158ee0 2024-03-18T16:22:19.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.193+0000 7f4887e00640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f486806bc20 0x7f486806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:19.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.195+0000 7f4887e00640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f486806bc20 0x7f486806e0e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4878009670 tx=0x7f4878006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:19.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.226+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f097014b1f0 con 0x7f09701f1030 2024-03-18T16:22:19.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.229+0000 7f0966a00640 1 -- 172.21.15.102:0/14197978 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f095c07bc60 con 0x7f09701f1030 2024-03-18T16:22:19.238 INFO:teuthology.orchestra.run.smithi102.stdout:64424509456 2024-03-18T16:22:19.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f095006bc00 msgr2=0x7f095006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f095006bc00 0x7f095006e0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0960009630 tx=0x7f0960006e90 comp rx=0 tx=0).stop 2024-03-18T16:22:19.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09701f1030 msgr2=0x7f09700cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09701f1030 0x7f09700cb870 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f095c009890 tx=0x7f095c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 shutdown_connections 2024-03-18T16:22:19.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f095006bc00 0x7f095006e0c0 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0960009630 tx=0x7f0960006e90 comp rx=0 tx=0).stop 2024-03-18T16:22:19.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09701f1030 0x7f09700cb870 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09701508d0 0x7f09700cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 --2- 172.21.15.102:0/14197978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f09700c7bc0 0x7f09701f0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.232+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 >> 172.21.15.102:0/14197978 conn(0x7f09700700d0 msgr2=0x7f097014b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:19.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.233+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 shutdown_connections 2024-03-18T16:22:19.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.233+0000 7f0975800640 1 -- 172.21.15.102:0/14197978 wait complete. 2024-03-18T16:22:19.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.235+0000 7fbc1be00640 1 -- 172.21.15.102:0/1845492288 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbc10059990 con 0x7fbc24158970 2024-03-18T16:22:19.261 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509456 for osd.0 2024-03-18T16:22:19.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.287+0000 7f0373400640 1 -- 172.21.15.102:0/2433927156 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0360005850 con 0x7f038815a9b0 2024-03-18T16:22:19.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.360+0000 7facc9600640 1 -- 172.21.15.102:0/4039851687 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7facc0021020 con 0x7faccc0c86b0 2024-03-18T16:22:19.363 INFO:teuthology.orchestra.run.smithi102.stdout:64424509458 2024-03-18T16:22:19.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7facac06bbd0 msgr2=0x7facac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7facac06bbd0 0x7facac06e090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faccc0c5660 tx=0x7facb8006c90 comp rx=0 tx=0).stop 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0c86b0 msgr2=0x7faccc1f32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0c86b0 0x7faccc1f32c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7facc000c730 tx=0x7facc000cbf0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 shutdown_connections 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7facac06bbd0 0x7facac06e090 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faccc15a870 0x7faccc1f1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faccc0c8a90 0x7faccc071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 --2- 172.21.15.102:0/4039851687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faccc0c86b0 0x7faccc1f32c0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7facc000c730 tx=0x7facc000cbf0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 >> 172.21.15.102:0/4039851687 conn(0x7faccc0700d0 msgr2=0x7faccc0c61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:19.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 shutdown_connections 2024-03-18T16:22:19.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.364+0000 7facd4600640 1 -- 172.21.15.102:0/4039851687 wait complete. 2024-03-18T16:22:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.379+0000 7f8cdea00640 1 -- 172.21.15.102:0/1674300229 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd003b070 con 0x7f8ce00cad80 2024-03-18T16:22:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.379+0000 7f8cdea00640 1 -- 172.21.15.102:0/1674300229 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cd0036440 con 0x7f8ce00cad80 2024-03-18T16:22:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.379+0000 7f8cdea00640 1 -- 172.21.15.102:0/1674300229 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0036810 con 0x7f8ce00cad80 2024-03-18T16:22:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.380+0000 7f8ce6e00640 1 -- 172.21.15.102:0/1674300229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00cad80 msgr2=0x7f8ce006dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.380+0000 7f8ce6e00640 1 --2- 172.21.15.102:0/1674300229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00cad80 0x7f8ce006dfe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000bb10 tx=0x7f8cd002e0c0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.381+0000 7f8ce6e00640 1 -- 172.21.15.102:0/1674300229 shutdown_connections 2024-03-18T16:22:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.381+0000 7f8ce6e00640 1 --2- 172.21.15.102:0/1674300229 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00cb160 0x7f8ce01501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.381+0000 7f8ce6e00640 1 --2- 172.21.15.102:0/1674300229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00cad80 0x7f8ce006dfe0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.381+0000 7f8ce6e00640 1 -- 172.21.15.102:0/1674300229 >> 172.21.15.102:0/1674300229 conn(0x7f8ce00c8390 msgr2=0x7f8ce00c8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.382+0000 7f8ce6e00640 1 -- 172.21.15.102:0/1674300229 shutdown_connections 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.382+0000 7f8ce6e00640 1 -- 172.21.15.102:0/1674300229 wait complete. 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.385+0000 7f8ce6e00640 1 Processor -- start 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.385+0000 7f8ce6e00640 1 -- start start 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.387+0000 7f8ce6e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00c8e80 0x7f8ce01fd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.387+0000 7f8ce6e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce00cad80 0x7f8ce01495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.387+0000 7f8ce6e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00cb160 0x7f8ce01f9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.387+0000 7f8ce6e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce006e520 con 0x7f8ce00c8e80 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.387+0000 7f8ce6e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce01571a0 con 0x7f8ce00cad80 2024-03-18T16:22:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.387+0000 7f8ce6e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8ce00c5100 con 0x7f8ce00cb160 2024-03-18T16:22:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.389+0000 7f8cdfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00c8e80 0x7f8ce01fd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.389+0000 7f8cdfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00c8e80 0x7f8ce01fd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52100/0 (socket says 172.21.15.102:52100) 2024-03-18T16:22:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.389+0000 7f8cdfe00640 1 -- 172.21.15.102:0/1493629986 learned_addr learned my addr 172.21.15.102:0/1493629986 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:19.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509459 got 64424509458 for osd.4 2024-03-18T16:22:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.394+0000 7f8cdfe00640 1 -- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00cb160 msgr2=0x7f8ce01f9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.394+0000 7f8cdfe00640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00cb160 0x7f8ce01f9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.394+0000 7f8cdfe00640 1 -- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce00cad80 msgr2=0x7f8ce01495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.394+0000 7f8cdfe00640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce00cad80 0x7f8ce01495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.394+0000 7f8cdfe00640 1 -- 172.21.15.102:0/1493629986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce01fda40 con 0x7f8ce00c8e80 2024-03-18T16:22:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.398+0000 7f8cdfe00640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00c8e80 0x7f8ce01fd500 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000bc30 tx=0x7f8cd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.404+0000 7f8cd7e00640 1 -- 172.21.15.102:0/1493629986 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd003b040 con 0x7f8ce00c8e80 2024-03-18T16:22:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.404+0000 7f8cd7e00640 1 -- 172.21.15.102:0/1493629986 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cd0044070 con 0x7f8ce00c8e80 2024-03-18T16:22:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.404+0000 7f8cd7e00640 1 -- 172.21.15.102:0/1493629986 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0034b50 con 0x7f8ce00c8e80 2024-03-18T16:22:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.405+0000 7f8ce6e00640 1 -- 172.21.15.102:0/1493629986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ce014a430 con 0x7f8ce00c8e80 2024-03-18T16:22:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.405+0000 7f8ce6e00640 1 -- 172.21.15.102:0/1493629986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ce014a9b0 con 0x7f8ce00c8e80 2024-03-18T16:22:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.407+0000 7f8cd5600640 1 -- 172.21.15.102:0/1493629986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb4005290 con 0x7f8ce00c8e80 2024-03-18T16:22:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.437+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f451c0684e0 con 0x7f451c158ee0 2024-03-18T16:22:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.441+0000 7f4516000640 1 -- 172.21.15.102:0/1879292372 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f450c05ced0 con 0x7f451c158ee0 2024-03-18T16:22:19.453 INFO:teuthology.orchestra.run.smithi102.stdout:55834574866 2024-03-18T16:22:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.443+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f450006bc00 msgr2=0x7f450006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.443+0000 7f4524800640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f450006bc00 0x7f450006e0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f451000b810 tx=0x7f4510008040 comp rx=0 tx=0).stop 2024-03-18T16:22:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.443+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c158ee0 msgr2=0x7f451c152b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.443+0000 7f4524800640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c158ee0 0x7f451c152b40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f450c00c400 tx=0x7f450c00c8c0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.443+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 shutdown_connections 2024-03-18T16:22:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.443+0000 7f4524800640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f450006bc00 0x7f450006e0c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.443+0000 7f4524800640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f451c15b8e0 0x7f451c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.443+0000 7f4524800640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f451c158ee0 0x7f451c152b40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.444+0000 7f4524800640 1 --2- 172.21.15.102:0/1879292372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f451c1524c0 0x7f451c0c27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.444+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 >> 172.21.15.102:0/1879292372 conn(0x7f451c0700d0 msgr2=0x7f451c06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.444+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 shutdown_connections 2024-03-18T16:22:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.444+0000 7f4524800640 1 -- 172.21.15.102:0/1879292372 wait complete. 2024-03-18T16:22:19.461 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574866 got 55834574866 for osd.5 2024-03-18T16:22:19.461 DEBUG:teuthology.parallel:result is None 2024-03-18T16:22:19.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.472+0000 7f037e000640 1 -- 172.21.15.102:0/2433927156 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0374058d80 con 0x7f038815a9b0 2024-03-18T16:22:19.514 INFO:teuthology.orchestra.run.smithi102.stdout:64424509456 2024-03-18T16:22:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.474+0000 7f487fe00640 1 -- 172.21.15.102:0/3751459348 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4874021020 con 0x7f48880ca960 2024-03-18T16:22:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.475+0000 7f0373400640 1 -- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f036806bc20 msgr2=0x7f036806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.475+0000 7f0373400640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f036806bc20 0x7f036806e0e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f038000ab90 tx=0x7f0380009250 comp rx=0 tx=0).stop 2024-03-18T16:22:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.475+0000 7f0373400640 1 -- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f038815a9b0 msgr2=0x7f03881e5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.475+0000 7f0373400640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f038815a9b0 0x7f03881e5f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f037400c390 tx=0x7f037400c850 comp rx=0 tx=0).stop 2024-03-18T16:22:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.476+0000 7f0373400640 1 -- 172.21.15.102:0/2433927156 shutdown_connections 2024-03-18T16:22:19.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.476+0000 7f0373400640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f036806bc20 0x7f036806e0e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.476+0000 7f0373400640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f038815a9b0 0x7f03881e5f90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.476+0000 7f0373400640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03880cf3a0 0x7f03880b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.476+0000 7f0373400640 1 --2- 172.21.15.102:0/2433927156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03880caab0 0x7f03881f8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.476+0000 7f0373400640 1 -- 172.21.15.102:0/2433927156 >> 172.21.15.102:0/2433927156 conn(0x7f0388067160 msgr2=0x7f03881e6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.476+0000 7f0373400640 1 -- 172.21.15.102:0/2433927156 shutdown_connections 2024-03-18T16:22:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.476+0000 7f0373400640 1 -- 172.21.15.102:0/2433927156 wait complete. 2024-03-18T16:22:19.517 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509456 for osd.3 2024-03-18T16:22:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.503+0000 7f8cd7e00640 1 -- 172.21.15.102:0/1493629986 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8cd0034cf0 con 0x7f8ce00c8e80 2024-03-18T16:22:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.505+0000 7f8cd7e00640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8cc006bc50 0x7f8cc006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.505+0000 7f8cdf400640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8cc006bc50 0x7f8cc006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.506+0000 7f8cdf400640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8cc006bc50 0x7f8cc006e110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8ce00c4a60 tx=0x7f8ccc006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.508+0000 7f8cd7e00640 1 -- 172.21.15.102:0/1493629986 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f8cd0079e00 con 0x7f8ce00c8e80 2024-03-18T16:22:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.536+0000 7fbc19600640 1 -- 172.21.15.102:0/1845492288 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbbf8005850 con 0x7fbc24158970 2024-03-18T16:22:19.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.709+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f4860005850 con 0x7f48880ca960 2024-03-18T16:22:19.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.714+0000 7f487fe00640 1 -- 172.21.15.102:0/3751459348 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4874058d50 con 0x7f48880ca960 2024-03-18T16:22:19.716 INFO:teuthology.orchestra.run.smithi102.stdout:64424509458 2024-03-18T16:22:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f486806bc20 msgr2=0x7f486806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f486806bc20 0x7f486806e0e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4878009670 tx=0x7f4878006bf0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 msgr2=0x7f48880b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f48880b32a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f487400c440 tx=0x7f487400c900 comp rx=0 tx=0).stop 2024-03-18T16:22:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 shutdown_connections 2024-03-18T16:22:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f486806bc20 0x7f486806e0e0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4878009670 tx=0x7f4878006bf0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4888154630 0x7f48880b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f48880ca960 0x7f48880b32a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 --2- 172.21.15.102:0/3751459348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48880ca510 0x7f48881f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 >> 172.21.15.102:0/3751459348 conn(0x7f4888067160 msgr2=0x7f488814f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 shutdown_connections 2024-03-18T16:22:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.716+0000 7f488d600640 1 -- 172.21.15.102:0/3751459348 wait complete. 2024-03-18T16:22:19.734 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509458 got 64424509458 for osd.7 2024-03-18T16:22:19.734 DEBUG:teuthology.parallel:result is None 2024-03-18T16:22:19.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.846+0000 7f8cd7e00640 1 -- 172.21.15.102:0/1493629986 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8cd00353a0 con 0x7f8ce00c8e80 2024-03-18T16:22:19.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-18T16:22:19.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.937+0000 7fbc1be00640 1 -- 172.21.15.102:0/1845492288 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbc10059290 con 0x7fbc24158970 2024-03-18T16:22:19.981 INFO:teuthology.orchestra.run.smithi102.stdout:64424509459 2024-03-18T16:22:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.940+0000 7fbc19600640 1 -- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbc0406bdd0 msgr2=0x7fbc0406e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.940+0000 7fbc19600640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbc0406bdd0 0x7fbc0406e290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c0012e0 tx=0x7fbc1c007d50 comp rx=0 tx=0).stop 2024-03-18T16:22:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.940+0000 7fbc19600640 1 -- 172.21.15.102:0/1845492288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc24158970 msgr2=0x7fbc241602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.940+0000 7fbc19600640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc24158970 0x7fbc241602d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009fd0 tx=0x7fbc1000b8e0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.941+0000 7fbc19600640 1 -- 172.21.15.102:0/1845492288 shutdown_connections 2024-03-18T16:22:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.941+0000 7fbc19600640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbc0406bdd0 0x7fbc0406e290 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.941+0000 7fbc19600640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc24158970 0x7fbc241602d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.941+0000 7fbc19600640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc240c7c50 0x7fbc240c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.941+0000 7fbc19600640 1 --2- 172.21.15.102:0/1845492288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc2406dd90 0x7fbc241f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:19.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.941+0000 7fbc19600640 1 -- 172.21.15.102:0/1845492288 >> 172.21.15.102:0/1845492288 conn(0x7fbc240c8390 msgr2=0x7fbc24157030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:19.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.941+0000 7fbc19600640 1 -- 172.21.15.102:0/1845492288 shutdown_connections 2024-03-18T16:22:19.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:19.941+0000 7fbc19600640 1 -- 172.21.15.102:0/1845492288 wait complete. 2024-03-18T16:22:19.984 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509459 got 64424509459 for osd.1 2024-03-18T16:22:19.984 DEBUG:teuthology.parallel:result is None 2024-03-18T16:22:20.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.058+0000 7f8cd5600640 1 -- 172.21.15.102:0/1493629986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f8cb4005850 con 0x7f8ce00c8e80 2024-03-18T16:22:20.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.117+0000 7f8cd7e00640 1 -- 172.21.15.102:0/1493629986 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8cd007c460 con 0x7f8ce00c8e80 2024-03-18T16:22:20.120 INFO:teuthology.orchestra.run.smithi102.stdout:55834574866 2024-03-18T16:22:20.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 -- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8cc006bc50 msgr2=0x7f8cc006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8cc006bc50 0x7f8cc006e110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8ce00c4a60 tx=0x7f8ccc006bf0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 -- 172.21.15.102:0/1493629986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00c8e80 msgr2=0x7f8ce01fd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00c8e80 0x7f8ce01fd500 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8cd000bc30 tx=0x7f8cd00046c0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 -- 172.21.15.102:0/1493629986 shutdown_connections 2024-03-18T16:22:20.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8cc006bc50 0x7f8cc006e110 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ce00cb160 0x7f8ce01f9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ce00cad80 0x7f8ce01495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 --2- 172.21.15.102:0/1493629986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ce00c8e80 0x7f8ce01fd500 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 -- 172.21.15.102:0/1493629986 >> 172.21.15.102:0/1493629986 conn(0x7f8ce00c8390 msgr2=0x7f8ce014ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:20.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 -- 172.21.15.102:0/1493629986 shutdown_connections 2024-03-18T16:22:20.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.121+0000 7f8cd5600640 1 -- 172.21.15.102:0/1493629986 wait complete. 2024-03-18T16:22:20.148 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574866 got 55834574866 for osd.6 2024-03-18T16:22:20.205 DEBUG:teuthology.parallel:result is None 2024-03-18T16:22:20.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.212+0000 7f7ef1000640 1 Processor -- start 2024-03-18T16:22:20.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.212+0000 7f7ef1000640 1 -- start start 2024-03-18T16:22:20.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.213+0000 7f7ef1000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7eec15ae70 0x7f7eec0ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.213+0000 7f7ef1000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7eec0cc4f0 0x7f7eec0cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.213+0000 7f7ef1000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec154ab0 con 0x7f7eec0c9be0 2024-03-18T16:22:20.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.213+0000 7f7ef1000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7eec06d2f0 con 0x7f7eec15ae70 2024-03-18T16:22:20.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.213+0000 7f7ef1000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7eec06d5d0 con 0x7f7eec0cc4f0 2024-03-18T16:22:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.214+0000 7f7eebe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7eec0cc4f0 0x7f7eec0cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.214+0000 7f7eebe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7eec0cc4f0 0x7f7eec0cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47770/0 (socket says 172.21.15.102:47770) 2024-03-18T16:22:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.214+0000 7f7eebe00640 1 -- 172.21.15.102:0/2696393310 learned_addr learned my addr 172.21.15.102:0/2696393310 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:20.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.215+0000 7f7eeb400640 1 --2- 172.21.15.102:0/2696393310 >> v2:172.21.15.102:3300/0 conn(0x7f7eec15ae70 0x7f7eec0ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.217+0000 7f7eebe00640 1 -- 172.21.15.102:0/2696393310 >> v2:172.21.15.102:3300/0 conn(0x7f7eec15ae70 msgr2=0x7f7eec0ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:20.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.217+0000 7f7eebe00640 1 --2- 172.21.15.102:0/2696393310 >> v2:172.21.15.102:3300/0 conn(0x7f7eec15ae70 0x7f7eec0ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.217+0000 7f7eebe00640 1 -- 172.21.15.102:0/2696393310 >> v1:172.21.15.102:6789/0 conn(0x7f7eec0c9be0 legacy=0x7f7eec0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.217+0000 7f7eebe00640 1 -- 172.21.15.102:0/2696393310 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eec15b250 con 0x7f7eec0cc4f0 2024-03-18T16:22:20.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.219+0000 7f7eebe00640 1 --2- 172.21.15.102:0/2696393310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec0cc4f0 0x7f7eec0cc8f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0c6a30 tx=0x7f7ee402bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=591e027c9df41125 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.223+0000 7f7eea000640 1 -- 172.21.15.102:0/2696393310 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee402cb10 con 0x7f7eec0cc4f0 2024-03-18T16:22:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.223+0000 7f7eea000640 1 -- 172.21.15.102:0/2696393310 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee402ccb0 con 0x7f7eec0cc4f0 2024-03-18T16:22:20.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.223+0000 7f7eea000640 1 -- 172.21.15.102:0/2696393310 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee402ce60 con 0x7f7eec0cc4f0 2024-03-18T16:22:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.223+0000 7f7ef1000640 1 -- 172.21.15.102:0/2696393310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec0cc4f0 msgr2=0x7f7eec0cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.223+0000 7f7ef1000640 1 --2- 172.21.15.102:0/2696393310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec0cc4f0 0x7f7eec0cc8f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7eec0c6a30 tx=0x7f7ee402bf70 comp rx=0 tx=0).stop 2024-03-18T16:22:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.224+0000 7f7ef1000640 1 -- 172.21.15.102:0/2696393310 shutdown_connections 2024-03-18T16:22:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.224+0000 7f7ef1000640 1 --2- 172.21.15.102:0/2696393310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec0cc4f0 0x7f7eec0cc8f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.224+0000 7f7ef1000640 1 --2- 172.21.15.102:0/2696393310 >> v2:172.21.15.102:3300/0 conn(0x7f7eec15ae70 0x7f7eec0ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.224+0000 7f7ef1000640 1 -- 172.21.15.102:0/2696393310 >> 172.21.15.102:0/2696393310 conn(0x7f7eec0673a0 msgr2=0x7f7eec0677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:20.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.224+0000 7f7ef1000640 1 -- 172.21.15.102:0/2696393310 shutdown_connections 2024-03-18T16:22:20.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.224+0000 7f7ef1000640 1 -- 172.21.15.102:0/2696393310 wait complete. 2024-03-18T16:22:20.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.225+0000 7f7ef1000640 1 Processor -- start 2024-03-18T16:22:20.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.226+0000 7f7ef1000640 1 -- start start 2024-03-18T16:22:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.226+0000 7f7ef1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7eec0c9be0 0x7f7eec0ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.226+0000 7f7ef1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eec0cc4f0 0x7f7eec1f3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.226+0000 7f7ef1000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec15ae70 0x7f7eec1f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.226+0000 7f7ef1000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec06de70 con 0x7f7eec0c9be0 2024-03-18T16:22:20.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.226+0000 7f7ef1000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7eec06db90 con 0x7f7eec15ae70 2024-03-18T16:22:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.226+0000 7f7ef1000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7eec0cce30 con 0x7f7eec0cc4f0 2024-03-18T16:22:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.227+0000 7f7eebe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec15ae70 0x7f7eec1f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.227+0000 7f7eeb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7eec0c9be0 0x7f7eec0ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.227+0000 7f7eeaa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eec0cc4f0 0x7f7eec1f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.227+0000 7f7eebe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec15ae70 0x7f7eec1f3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47774/0 (socket says 172.21.15.102:47774) 2024-03-18T16:22:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.227+0000 7f7eebe00640 1 -- 172.21.15.102:0/34325679 learned_addr learned my addr 172.21.15.102:0/34325679 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:20.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.231+0000 7f7eeb400640 1 -- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eec0cc4f0 msgr2=0x7f7eec1f3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.231+0000 7f7eeb400640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eec0cc4f0 0x7f7eec1f3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.231+0000 7f7eeb400640 1 -- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec15ae70 msgr2=0x7f7eec1f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.231+0000 7f7eeb400640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec15ae70 0x7f7eec1f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.231+0000 7f7eeb400640 1 -- 172.21.15.102:0/34325679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eec155600 con 0x7f7eec0c9be0 2024-03-18T16:22:20.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.235+0000 7f7eeb400640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7eec0c9be0 0x7f7eec0ce700 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00d620 tx=0x7f7edc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.237+0000 7f7ee3400640 1 -- 172.21.15.102:0/34325679 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00b620 con 0x7f7eec0c9be0 2024-03-18T16:22:20.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.237+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eec0c55d0 con 0x7f7eec0c9be0 2024-03-18T16:22:20.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.237+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eec155b80 con 0x7f7eec0c9be0 2024-03-18T16:22:20.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.239+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7eec074fd0 con 0x7f7eec0c9be0 2024-03-18T16:22:20.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.240+0000 7f7ee3400640 1 -- 172.21.15.102:0/34325679 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7edc019070 con 0x7f7eec0c9be0 2024-03-18T16:22:20.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.240+0000 7f7ee3400640 1 -- 172.21.15.102:0/34325679 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00b9c0 con 0x7f7eec0c9be0 2024-03-18T16:22:20.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-18T16:22:20.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-18T16:22:20.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.514+0000 7f7ee3400640 1 -- 172.21.15.102:0/34325679 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7edc00bb60 con 0x7f7eec0c9be0 2024-03-18T16:22:20.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.516+0000 7f7ee3400640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ecc06bc00 0x7f7ecc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-18T16:22:20.520 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:22:20.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.516+0000 7f7eeaa00640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ecc06bc00 0x7f7ecc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.517+0000 7f7eeaa00640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ecc06bc00 0x7f7ecc06e0c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7eec1e7590 tx=0x7f7ed8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:22:20.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.541+0000 7f7ee3400640 1 -- 172.21.15.102:0/34325679 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f7edc057d20 con 0x7f7eec0c9be0 2024-03-18T16:22:20.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.542+0000 7f7ee3400640 1 -- 172.21.15.102:0/34325679 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7edc089e20 con 0x7f7eec0c9be0 2024-03-18T16:22:20.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.664+0000 7f3ed6200640 1 Processor -- start 2024-03-18T16:22:20.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.665+0000 7f3ed6200640 1 -- start start 2024-03-18T16:22:20.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.667+0000 7f3ed6200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed00cf690 0x7f3ed015a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.667+0000 7f3ed6200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3ed006dfa0 0x7f3ed00ce700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.668+0000 7f3ed6200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ed00cb1e0 con 0x7f3ed0154950 2024-03-18T16:22:20.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.668+0000 7f3ed6200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3ed015acb0 con 0x7f3ed006dfa0 2024-03-18T16:22:20.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.668+0000 7f3ed6200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3ed015af90 con 0x7f3ed00cf690 2024-03-18T16:22:20.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.668+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed00cf690 0x7f3ed015a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.669+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed00cf690 0x7f3ed015a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34440/0 (socket says 172.21.15.102:34440) 2024-03-18T16:22:20.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.669+0000 7f3ecfe00640 1 -- 172.21.15.102:0/18964465 learned_addr learned my addr 172.21.15.102:0/18964465 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:20.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.672+0000 7f3ed5800640 1 --2- 172.21.15.102:0/18964465 >> v2:172.21.15.154:3300/0 conn(0x7f3ed006dfa0 0x7f3ed00ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.672+0000 7f3ecfe00640 1 -- 172.21.15.102:0/18964465 >> v2:172.21.15.154:3300/0 conn(0x7f3ed006dfa0 msgr2=0x7f3ed00ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.672+0000 7f3ecfe00640 1 --2- 172.21.15.102:0/18964465 >> v2:172.21.15.154:3300/0 conn(0x7f3ed006dfa0 0x7f3ed00ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.672+0000 7f3ecfe00640 1 -- 172.21.15.102:0/18964465 >> v1:172.21.15.154:6789/0 conn(0x7f3ed0154950 legacy=0x7f3ed00cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.672+0000 7f3ecfe00640 1 -- 172.21.15.102:0/18964465 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed015b270 con 0x7f3ed00cf690 2024-03-18T16:22:20.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.675+0000 7f3ecfe00640 1 --2- 172.21.15.102:0/18964465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed00cf690 0x7f3ed015a620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009870 tx=0x7f3ebc02b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9c27f9d398947c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.680+0000 7f3ecf400640 1 -- 172.21.15.102:0/18964465 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc034070 con 0x7f3ed00cf690 2024-03-18T16:22:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.680+0000 7f3ecf400640 1 -- 172.21.15.102:0/18964465 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ebc030860 con 0x7f3ed00cf690 2024-03-18T16:22:20.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.680+0000 7f3ecf400640 1 -- 172.21.15.102:0/18964465 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc030c70 con 0x7f3ed00cf690 2024-03-18T16:22:20.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.681+0000 7f3ed6200640 1 -- 172.21.15.102:0/18964465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed00cf690 msgr2=0x7f3ed015a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.681+0000 7f3ed6200640 1 --2- 172.21.15.102:0/18964465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed00cf690 0x7f3ed015a620 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc009870 tx=0x7f3ebc02b9a0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.682+0000 7f3ed6200640 1 -- 172.21.15.102:0/18964465 shutdown_connections 2024-03-18T16:22:20.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.682+0000 7f3ed6200640 1 --2- 172.21.15.102:0/18964465 >> v2:172.21.15.154:3300/0 conn(0x7f3ed006dfa0 0x7f3ed00ce700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.682+0000 7f3ed6200640 1 --2- 172.21.15.102:0/18964465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed00cf690 0x7f3ed015a620 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.682+0000 7f3ed6200640 1 -- 172.21.15.102:0/18964465 >> 172.21.15.102:0/18964465 conn(0x7f3ed00b1720 msgr2=0x7f3ed00b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:20.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.683+0000 7f3ed6200640 1 -- 172.21.15.102:0/18964465 shutdown_connections 2024-03-18T16:22:20.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.683+0000 7f3ed6200640 1 -- 172.21.15.102:0/18964465 wait complete. 2024-03-18T16:22:20.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.686+0000 7f3ed6200640 1 Processor -- start 2024-03-18T16:22:20.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.686+0000 7f3ed6200640 1 -- start start 2024-03-18T16:22:20.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.688+0000 7f3ed6200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed006dfa0 0x7f3ed01573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.689+0000 7f3ed6200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed00cf690 0x7f3ed01ec770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.689+0000 7f3ed6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed0154950 0x7f3ed01eccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.690+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed00cf690 0x7f3ed01ec770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.690+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed00cf690 0x7f3ed01ec770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47790/0 (socket says 172.21.15.102:47790) 2024-03-18T16:22:20.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.690+0000 7f3ed4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed006dfa0 0x7f3ed01573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.690+0000 7f3ecfe00640 1 -- 172.21.15.102:0/1902957281 learned_addr learned my addr 172.21.15.102:0/1902957281 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:20.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.690+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed00ce1e0 con 0x7f3ed0154950 2024-03-18T16:22:20.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.690+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed00cdf00 con 0x7f3ed00cf690 2024-03-18T16:22:20.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.690+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3ed0154d30 con 0x7f3ed006dfa0 2024-03-18T16:22:20.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.694+0000 7f3ecfe00640 1 -- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed006dfa0 msgr2=0x7f3ed01573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.694+0000 7f3ecfe00640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed006dfa0 0x7f3ed01573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.694+0000 7f3ecfe00640 1 -- 172.21.15.102:0/1902957281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed0154950 msgr2=0x7f3ed01eccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.694+0000 7f3ecfe00640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed0154950 0x7f3ed01eccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.695+0000 7f3ecfe00640 1 -- 172.21.15.102:0/1902957281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed01ef720 con 0x7f3ed00cf690 2024-03-18T16:22:20.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.697+0000 7f3ecfe00640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed00cf690 0x7f3ed01ec770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc02b9e0 tx=0x7f3ebc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.715+0000 7f3eccc00640 1 -- 172.21.15.102:0/1902957281 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc048070 con 0x7f3ed00cf690 2024-03-18T16:22:20.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.718+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed01ef9e0 con 0x7f3ed00cf690 2024-03-18T16:22:20.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.718+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed01effa0 con 0x7f3ed00cf690 2024-03-18T16:22:20.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.718+0000 7f3eccc00640 1 -- 172.21.15.102:0/1902957281 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ebc039d90 con 0x7f3ed00cf690 2024-03-18T16:22:20.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.718+0000 7f3eccc00640 1 -- 172.21.15.102:0/1902957281 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ebc034040 con 0x7f3ed00cf690 2024-03-18T16:22:20.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.719+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ed0073830 con 0x7f3ed00cf690 2024-03-18T16:22:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.741+0000 7f3eccc00640 1 -- 172.21.15.102:0/1902957281 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3ebc037070 con 0x7f3ed00cf690 2024-03-18T16:22:20.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.742+0000 7f3eccc00640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3eb006bc00 0x7f3eb006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.743+0000 7f3ed4e00640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3eb006bc00 0x7f3eb006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.744+0000 7f3ed4e00640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3eb006bc00 0x7f3eb006e0c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4006fd0 tx=0x7f3ec4009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.755+0000 7f64b7200640 1 Processor -- start 2024-03-18T16:22:20.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.756+0000 7f64b7200640 1 -- start start 2024-03-18T16:22:20.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.758+0000 7f64b7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b0154780 0x7f64b015a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.758+0000 7f64b7200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64b00c5050 0x7f64b00cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.758+0000 7f64b7200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b015daa0 con 0x7f64b015e260 2024-03-18T16:22:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.759+0000 7f64b7200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f64b015aad0 con 0x7f64b00c5050 2024-03-18T16:22:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.759+0000 7f64b7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f64b015adb0 con 0x7f64b0154780 2024-03-18T16:22:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.759+0000 7f64afe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b0154780 0x7f64b015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.760+0000 7f64afe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b0154780 0x7f64b015a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34450/0 (socket says 172.21.15.102:34450) 2024-03-18T16:22:20.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.760+0000 7f64afe00640 1 -- 172.21.15.102:0/1006347468 learned_addr learned my addr 172.21.15.102:0/1006347468 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.763+0000 7f64afe00640 1 -- 172.21.15.102:0/1006347468 >> v2:172.21.15.102:3300/0 conn(0x7f64b00c5050 msgr2=0x7f64b00cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.763+0000 7f64afe00640 1 --2- 172.21.15.102:0/1006347468 >> v2:172.21.15.102:3300/0 conn(0x7f64b00c5050 0x7f64b00cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.763+0000 7f64afe00640 1 -- 172.21.15.102:0/1006347468 >> v1:172.21.15.102:6789/0 conn(0x7f64b015e260 legacy=0x7f64b00caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.763+0000 7f64afe00640 1 -- 172.21.15.102:0/1006347468 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b015b090 con 0x7f64b0154780 2024-03-18T16:22:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.766+0000 7f64afe00640 1 --2- 172.21.15.102:0/1006347468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b0154780 0x7f64b015a490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f64b0067eb0 tx=0x7f64a002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6edec8480d1ead0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.770+0000 7f64aea00640 1 -- 172.21.15.102:0/1006347468 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0034070 con 0x7f64b0154780 2024-03-18T16:22:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.770+0000 7f64aea00640 1 -- 172.21.15.102:0/1006347468 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64a0030840 con 0x7f64b0154780 2024-03-18T16:22:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.770+0000 7f64aea00640 1 -- 172.21.15.102:0/1006347468 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0030c10 con 0x7f64b0154780 2024-03-18T16:22:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.772+0000 7f64b7200640 1 -- 172.21.15.102:0/1006347468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b0154780 msgr2=0x7f64b015a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.772+0000 7f64b7200640 1 --2- 172.21.15.102:0/1006347468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b0154780 0x7f64b015a490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f64b0067eb0 tx=0x7f64a002b980 comp rx=0 tx=0).stop 2024-03-18T16:22:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.772+0000 7f64b7200640 1 -- 172.21.15.102:0/1006347468 shutdown_connections 2024-03-18T16:22:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.772+0000 7f64b7200640 1 --2- 172.21.15.102:0/1006347468 >> v2:172.21.15.102:3300/0 conn(0x7f64b00c5050 0x7f64b00cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.772+0000 7f64b7200640 1 --2- 172.21.15.102:0/1006347468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b0154780 0x7f64b015a490 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.772+0000 7f64b7200640 1 -- 172.21.15.102:0/1006347468 >> 172.21.15.102:0/1006347468 conn(0x7f64b0067160 msgr2=0x7f64b0067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.772+0000 7f64b7200640 1 -- 172.21.15.102:0/1006347468 shutdown_connections 2024-03-18T16:22:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.772+0000 7f64b7200640 1 -- 172.21.15.102:0/1006347468 wait complete. 2024-03-18T16:22:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.775+0000 7f64b7200640 1 Processor -- start 2024-03-18T16:22:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.775+0000 7f64b7200640 1 -- start start 2024-03-18T16:22:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.777+0000 7f64b7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64b00c5050 0x7f64b01e7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.777+0000 7f64b7200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b0154780 0x7f64b00b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.777+0000 7f64b7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b015e260 0x7f64b00b4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.777+0000 7f64b7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f64b0154b60 con 0x7f64b00c5050 2024-03-18T16:22:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.777+0000 7f64b7200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f64b015b370 con 0x7f64b0154780 2024-03-18T16:22:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.777+0000 7f64b7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f64b00c55a0 con 0x7f64b015e260 2024-03-18T16:22:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.778+0000 7f64b5200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b015e260 0x7f64b00b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.778+0000 7f64b5200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b015e260 0x7f64b00b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34464/0 (socket says 172.21.15.102:34464) 2024-03-18T16:22:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.778+0000 7f64b5200640 1 -- 172.21.15.102:0/443429487 learned_addr learned my addr 172.21.15.102:0/443429487 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:20.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.783+0000 7f64b5200640 1 -- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b0154780 msgr2=0x7f64b00b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.783+0000 7f64b5200640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b0154780 0x7f64b00b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.783+0000 7f64b5200640 1 -- 172.21.15.102:0/443429487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64b00c5050 msgr2=0x7f64b01e7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.783+0000 7f64b5200640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64b00c5050 0x7f64b01e7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.783+0000 7f64b5200640 1 -- 172.21.15.102:0/443429487 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b015b790 con 0x7f64b015e260 2024-03-18T16:22:20.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.787+0000 7f64b5200640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b015e260 0x7f64b00b4650 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f64a800c7e0 tx=0x7f64a800cca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.791+0000 7f64a7e00640 1 -- 172.21.15.102:0/443429487 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a8017070 con 0x7f64b015e260 2024-03-18T16:22:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.791+0000 7f64a7e00640 1 -- 172.21.15.102:0/443429487 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64a8003da0 con 0x7f64b015e260 2024-03-18T16:22:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.791+0000 7f64a7e00640 1 -- 172.21.15.102:0/443429487 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a80052b0 con 0x7f64b015e260 2024-03-18T16:22:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.791+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b015d1d0 con 0x7f64b015e260 2024-03-18T16:22:20.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.791+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b015cca0 con 0x7f64b015e260 2024-03-18T16:22:20.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.793+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b015d890 con 0x7f64b015e260 2024-03-18T16:22:20.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.824+0000 7f64a7e00640 1 -- 172.21.15.102:0/443429487 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f64a801a070 con 0x7f64b015e260 2024-03-18T16:22:20.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.826+0000 7f64a7e00640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f649006bc50 0x7f649006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.826+0000 7f64afe00640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f649006bc50 0x7f649006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.827+0000 7f64afe00640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f649006bc50 0x7f649006e110 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64a0036000 tx=0x7f64a003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.827+0000 7f843be00640 1 Processor -- start 2024-03-18T16:22:20.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.827+0000 7f843be00640 1 -- start start 2024-03-18T16:22:20.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.829+0000 7f843be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f843c15ae70 0x7f843c0ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.829+0000 7f843be00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f843c0c9be0 0x7f843c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.829+0000 7f843be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f843c154ab0 con 0x7f843c0d05a0 2024-03-18T16:22:20.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.829+0000 7f843be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f843c06d2f0 con 0x7f843c15ae70 2024-03-18T16:22:20.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.829+0000 7f843be00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f843c06d5d0 con 0x7f843c0c9be0 2024-03-18T16:22:20.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.829+0000 7f64a7e00640 1 -- 172.21.15.102:0/443429487 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f64a8056a60 con 0x7f64b015e260 2024-03-18T16:22:20.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.830+0000 7f843aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f843c15ae70 0x7f843c0ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.831+0000 7f843aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f843c15ae70 0x7f843c0ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52166/0 (socket says 172.21.15.102:52166) 2024-03-18T16:22:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.831+0000 7f843aa00640 1 -- 172.21.15.102:0/2931316607 learned_addr learned my addr 172.21.15.102:0/2931316607 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:20.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.835+0000 7f843aa00640 1 -- 172.21.15.102:0/2931316607 >> v2:172.21.15.154:3300/0 conn(0x7f843c0c9be0 msgr2=0x7f843c0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:20.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.835+0000 7f843aa00640 1 --2- 172.21.15.102:0/2931316607 >> v2:172.21.15.154:3300/0 conn(0x7f843c0c9be0 0x7f843c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.835+0000 7f843aa00640 1 -- 172.21.15.102:0/2931316607 >> v1:172.21.15.102:6789/0 conn(0x7f843c0d05a0 legacy=0x7f843c0cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.835+0000 7f843aa00640 1 -- 172.21.15.102:0/2931316607 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843c15b250 con 0x7f843c15ae70 2024-03-18T16:22:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.840+0000 7f843aa00640 1 --2- 172.21.15.102:0/2931316607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c15ae70 0x7f843c0ca490 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f843c157f40 tx=0x7f842c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70aff7db1eb47223 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.848+0000 7f8439600640 1 -- 172.21.15.102:0/2931316607 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c034070 con 0x7f843c15ae70 2024-03-18T16:22:20.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.849+0000 7f8439600640 1 -- 172.21.15.102:0/2931316607 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f842c030860 con 0x7f843c15ae70 2024-03-18T16:22:20.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.849+0000 7f8439600640 1 -- 172.21.15.102:0/2931316607 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c030c30 con 0x7f843c15ae70 2024-03-18T16:22:20.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.849+0000 7f843be00640 1 -- 172.21.15.102:0/2931316607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c15ae70 msgr2=0x7f843c0ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.850+0000 7f843be00640 1 --2- 172.21.15.102:0/2931316607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c15ae70 0x7f843c0ca490 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f843c157f40 tx=0x7f842c02b9a0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.850+0000 7f843be00640 1 -- 172.21.15.102:0/2931316607 shutdown_connections 2024-03-18T16:22:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.850+0000 7f843be00640 1 --2- 172.21.15.102:0/2931316607 >> v2:172.21.15.154:3300/0 conn(0x7f843c0c9be0 0x7f843c0d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.850+0000 7f843be00640 1 --2- 172.21.15.102:0/2931316607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c15ae70 0x7f843c0ca490 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.850+0000 7f843be00640 1 -- 172.21.15.102:0/2931316607 >> 172.21.15.102:0/2931316607 conn(0x7f843c0673a0 msgr2=0x7f843c0677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.850+0000 7f843be00640 1 -- 172.21.15.102:0/2931316607 shutdown_connections 2024-03-18T16:22:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.850+0000 7f843be00640 1 -- 172.21.15.102:0/2931316607 wait complete. 2024-03-18T16:22:20.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.853+0000 7f843be00640 1 Processor -- start 2024-03-18T16:22:20.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.853+0000 7f843be00640 1 -- start start 2024-03-18T16:22:20.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.855+0000 7f843be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f843c0c9be0 0x7f843c1f7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.855+0000 7f843be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c0d05a0 0x7f843c155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.855+0000 7f843be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f843c15ae70 0x7f843c14acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.855+0000 7f843be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f843c06de70 con 0x7f843c0d05a0 2024-03-18T16:22:20.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.855+0000 7f843be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f843c06db90 con 0x7f843c0c9be0 2024-03-18T16:22:20.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.855+0000 7f843be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f843c06e150 con 0x7f843c15ae70 2024-03-18T16:22:20.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.857+0000 7f843a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c0d05a0 0x7f843c155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.857+0000 7f843a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c0d05a0 0x7f843c155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55522/0 (socket says 172.21.15.102:55522) 2024-03-18T16:22:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.857+0000 7f843a000640 1 -- 172.21.15.102:0/1619749552 learned_addr learned my addr 172.21.15.102:0/1619749552 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.862+0000 7f843a000640 1 -- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f843c15ae70 msgr2=0x7f843c14acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.862+0000 7f843a000640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f843c15ae70 0x7f843c14acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.862+0000 7f843a000640 1 -- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f843c0c9be0 msgr2=0x7f843c1f7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.862+0000 7f843a000640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f843c0c9be0 0x7f843c1f7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.862+0000 7f843a000640 1 -- 172.21.15.102:0/1619749552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843c1f8730 con 0x7f843c0d05a0 2024-03-18T16:22:20.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.866+0000 7f843a000640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c0d05a0 0x7f843c155b90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f842800b410 tx=0x7f842800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.871+0000 7f8433400640 1 -- 172.21.15.102:0/1619749552 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428015020 con 0x7f843c0d05a0 2024-03-18T16:22:20.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.872+0000 7f8433400640 1 -- 172.21.15.102:0/1619749552 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8428011850 con 0x7f843c0d05a0 2024-03-18T16:22:20.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.872+0000 7f8433400640 1 -- 172.21.15.102:0/1619749552 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428011c40 con 0x7f843c0d05a0 2024-03-18T16:22:20.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.872+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f843c1f89d0 con 0x7f843c0d05a0 2024-03-18T16:22:20.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.872+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f843c1578d0 con 0x7f843c0d05a0 2024-03-18T16:22:20.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.873+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f843c074fd0 con 0x7f843c0d05a0 2024-03-18T16:22:20.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.877+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7eec0c9180 con 0x7f7eec0c9be0 2024-03-18T16:22:20.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.914+0000 7f8433400640 1 -- 172.21.15.102:0/1619749552 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8428018020 con 0x7f843c0d05a0 2024-03-18T16:22:20.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.915+0000 7f8433400640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f841c06bc50 0x7f841c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:20.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.915+0000 7f843aa00640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f841c06bc50 0x7f841c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:20.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.916+0000 7f843aa00640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f841c06bc50 0x7f841c06e110 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f842c036000 tx=0x7f842c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:20.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.978+0000 7f3eccc00640 1 -- 172.21.15.102:0/1902957281 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f3ebc079370 con 0x7f3ed00cf690 2024-03-18T16:22:20.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:20.988+0000 7f3eccc00640 1 -- 172.21.15.102:0/1902957281 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3ed0073830 con 0x7f3ed00cf690 2024-03-18T16:22:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.089+0000 7f64a7e00640 1 -- 172.21.15.102:0/443429487 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f64b015d890 con 0x7f64b015e260 2024-03-18T16:22:21.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.364+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3ed004f5b0 con 0x7f3ed00cf690 2024-03-18T16:22:21.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.366+0000 7f8433400640 1 -- 172.21.15.102:0/1619749552 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f8428011de0 con 0x7f843c0d05a0 2024-03-18T16:22:21.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.367+0000 7f8433400640 1 -- 172.21.15.102:0/1619749552 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f842801f020 con 0x7f843c0d05a0 2024-03-18T16:22:21.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.367+0000 7f7ee3400640 1 -- 172.21.15.102:0/34325679 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7edc059c80 con 0x7f7eec0c9be0 2024-03-18T16:22:21.369 INFO:teuthology.orchestra.run.smithi102.stdout:60129542162 2024-03-18T16:22:21.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.369+0000 7f3eccc00640 1 -- 172.21.15.102:0/1902957281 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3ebc051020 con 0x7f3ed00cf690 2024-03-18T16:22:21.371 INFO:teuthology.orchestra.run.smithi102.stdout:64424509458 2024-03-18T16:22:21.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.370+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ecc06bc00 msgr2=0x7f7ecc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.370+0000 7f7ef1000640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ecc06bc00 0x7f7ecc06e0c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7eec1e7590 tx=0x7f7ed8009040 comp rx=0 tx=0).stop 2024-03-18T16:22:21.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.370+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7eec0c9be0 msgr2=0x7f7eec0ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.370+0000 7f7ef1000640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7eec0c9be0 0x7f7eec0ce700 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00d620 tx=0x7f7edc00dae0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.371+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 shutdown_connections 2024-03-18T16:22:21.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.371+0000 7f7ef1000640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ecc06bc00 0x7f7ecc06e0c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.371+0000 7f7ef1000640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7eec15ae70 0x7f7eec1f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.371+0000 7f7ef1000640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7eec0cc4f0 0x7f7eec1f3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.371+0000 7f7ef1000640 1 --2- 172.21.15.102:0/34325679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7eec0c9be0 0x7f7eec0ce700 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.371+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 >> 172.21.15.102:0/34325679 conn(0x7f7eec0673a0 msgr2=0x7f7eec0af470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:21.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.371+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 shutdown_connections 2024-03-18T16:22:21.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.371+0000 7f7ef1000640 1 -- 172.21.15.102:0/34325679 wait complete. 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3eb006bc00 msgr2=0x7f3eb006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3eb006bc00 0x7f3eb006e0c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4006fd0 tx=0x7f3ec4009060 comp rx=0 tx=0).stop 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed00cf690 msgr2=0x7f3ed01ec770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed00cf690 0x7f3ed01ec770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc02b9e0 tx=0x7f3ebc004730 comp rx=0 tx=0).stop 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 shutdown_connections 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3eb006bc00 0x7f3eb006e0c0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4006fd0 tx=0x7f3ec4009060 comp rx=0 tx=0).stop 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ed0154950 0x7f3ed01eccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ed00cf690 0x7f3ed01ec770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 --2- 172.21.15.102:0/1902957281 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3ed006dfa0 0x7f3ed01573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 >> 172.21.15.102:0/1902957281 conn(0x7f3ed00b1720 msgr2=0x7f3ed00b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 shutdown_connections 2024-03-18T16:22:21.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.373+0000 7f3ed6200640 1 -- 172.21.15.102:0/1902957281 wait complete. 2024-03-18T16:22:21.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542162 got 60129542162 for osd.2 2024-03-18T16:22:21.403 DEBUG:teuthology.parallel:result is None 2024-03-18T16:22:21.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509458 for osd.0 2024-03-18T16:22:21.404 DEBUG:teuthology.parallel:result is None 2024-03-18T16:22:21.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.489+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f64b0074fd0 con 0x7f64b015e260 2024-03-18T16:22:21.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.526+0000 7f64a7e00640 1 -- 172.21.15.102:0/443429487 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f64a8021020 con 0x7f64b015e260 2024-03-18T16:22:21.534 INFO:teuthology.orchestra.run.smithi102.stdout:64424509460 2024-03-18T16:22:21.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f649006bc50 msgr2=0x7f649006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f649006bc50 0x7f649006e110 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64a0036000 tx=0x7f64a003b040 comp rx=0 tx=0).stop 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b015e260 msgr2=0x7f64b00b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b015e260 0x7f64b00b4650 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f64a800c7e0 tx=0x7f64a800cca0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 shutdown_connections 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f649006bc50 0x7f649006e110 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64b015e260 0x7f64b00b4650 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b0154780 0x7f64b00b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 --2- 172.21.15.102:0/443429487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64b00c5050 0x7f64b01e7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 >> 172.21.15.102:0/443429487 conn(0x7f64b0067160 msgr2=0x7f64b00caa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 shutdown_connections 2024-03-18T16:22:21.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.530+0000 7f64b7200640 1 -- 172.21.15.102:0/443429487 wait complete. 2024-03-18T16:22:21.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509459 got 64424509460 for osd.4 2024-03-18T16:22:21.555 DEBUG:teuthology.parallel:result is None 2024-03-18T16:22:21.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.673+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f843c0b5a90 con 0x7f843c0d05a0 2024-03-18T16:22:21.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.680+0000 7f8433400640 1 -- 172.21.15.102:0/1619749552 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f842800f2e0 con 0x7f843c0d05a0 2024-03-18T16:22:21.682 INFO:teuthology.orchestra.run.smithi102.stdout:64424509458 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.682+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f841c06bc50 msgr2=0x7f841c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.682+0000 7f843be00640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f841c06bc50 0x7f841c06e110 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f842c036000 tx=0x7f842c03b040 comp rx=0 tx=0).stop 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.682+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c0d05a0 msgr2=0x7f843c155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.682+0000 7f843be00640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c0d05a0 0x7f843c155b90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f842800b410 tx=0x7f842800b8d0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.683+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 shutdown_connections 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.683+0000 7f843be00640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f841c06bc50 0x7f841c06e110 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.683+0000 7f843be00640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f843c15ae70 0x7f843c14acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.683+0000 7f843be00640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f843c0d05a0 0x7f843c155b90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.683+0000 7f843be00640 1 --2- 172.21.15.102:0/1619749552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f843c0c9be0 0x7f843c1f7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.683+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 >> 172.21.15.102:0/1619749552 conn(0x7f843c0673a0 msgr2=0x7f843c159c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:21.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.683+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 shutdown_connections 2024-03-18T16:22:21.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.683+0000 7f843be00640 1 -- 172.21.15.102:0/1619749552 wait complete. 2024-03-18T16:22:21.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509457 got 64424509458 for osd.3 2024-03-18T16:22:21.709 DEBUG:teuthology.parallel:result is None 2024-03-18T16:22:21.710 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-18T16:22:21.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:22:21.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.878+0000 7fe495a00640 1 Processor -- start 2024-03-18T16:22:21.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.878+0000 7fe495a00640 1 -- start start 2024-03-18T16:22:21.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.879+0000 7fe495a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe49015a310 0x7fe49006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:21.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.879+0000 7fe495a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe49015bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:21.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.879+0000 7fe495a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4900ca500 con 0x7fe49015a6f0 2024-03-18T16:22:21.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.879+0000 7fe495a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe4900cd410 con 0x7fe49015a310 2024-03-18T16:22:21.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.879+0000 7fe495a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe49015c490 con 0x7fe49015baa0 2024-03-18T16:22:21.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.880+0000 7fe495000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe49015bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:21.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.881+0000 7fe48fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe49015a310 0x7fe49006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:21.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.881+0000 7fe495000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe49015bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:47992/0 (socket says 172.21.15.102:47992) 2024-03-18T16:22:21.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.881+0000 7fe495000640 1 -- 172.21.15.102:0/2297586837 learned_addr learned my addr 172.21.15.102:0/2297586837 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:21.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.884+0000 7fe495000640 1 -- 172.21.15.102:0/2297586837 >> v2:172.21.15.102:3300/0 conn(0x7fe49015a310 msgr2=0x7fe49006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:21.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.884+0000 7fe495000640 1 --2- 172.21.15.102:0/2297586837 >> v2:172.21.15.102:3300/0 conn(0x7fe49015a310 0x7fe49006d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.884+0000 7fe495000640 1 -- 172.21.15.102:0/2297586837 >> v1:172.21.15.102:6789/0 conn(0x7fe49015a6f0 legacy=0x7fe49015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.884+0000 7fe495000640 1 -- 172.21.15.102:0/2297586837 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4900ca7a0 con 0x7fe49015baa0 2024-03-18T16:22:21.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.887+0000 7fe495000640 1 --2- 172.21.15.102:0/2297586837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe49015bf50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe488009e50 tx=0x7fe48802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55fb6e3104124bf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:21.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.891+0000 7fe48ea00640 1 -- 172.21.15.102:0/2297586837 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48802cb00 con 0x7fe49015baa0 2024-03-18T16:22:21.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.891+0000 7fe48ea00640 1 -- 172.21.15.102:0/2297586837 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe48802cca0 con 0x7fe49015baa0 2024-03-18T16:22:21.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.891+0000 7fe48ea00640 1 -- 172.21.15.102:0/2297586837 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe488034610 con 0x7fe49015baa0 2024-03-18T16:22:21.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.892+0000 7fe495a00640 1 -- 172.21.15.102:0/2297586837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 msgr2=0x7fe49015bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.892+0000 7fe495a00640 1 --2- 172.21.15.102:0/2297586837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe49015bf50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe488009e50 tx=0x7fe48802bf60 comp rx=0 tx=0).stop 2024-03-18T16:22:21.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.892+0000 7fe495a00640 1 -- 172.21.15.102:0/2297586837 shutdown_connections 2024-03-18T16:22:21.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.892+0000 7fe495a00640 1 --2- 172.21.15.102:0/2297586837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe49015bf50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.892+0000 7fe495a00640 1 --2- 172.21.15.102:0/2297586837 >> v2:172.21.15.102:3300/0 conn(0x7fe49015a310 0x7fe49006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.892+0000 7fe495a00640 1 -- 172.21.15.102:0/2297586837 >> 172.21.15.102:0/2297586837 conn(0x7fe49008c1a0 msgr2=0x7fe490066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:21.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.892+0000 7fe495a00640 1 -- 172.21.15.102:0/2297586837 shutdown_connections 2024-03-18T16:22:21.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.892+0000 7fe495a00640 1 -- 172.21.15.102:0/2297586837 wait complete. 2024-03-18T16:22:21.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.895+0000 7fe495a00640 1 Processor -- start 2024-03-18T16:22:21.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.895+0000 7fe495a00640 1 -- start start 2024-03-18T16:22:21.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.896+0000 7fe495a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe49015a310 0x7fe4901f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:21.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.897+0000 7fe495a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49015a6f0 0x7fe4901f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:21.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.897+0000 7fe495a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe4900c4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:21.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.897+0000 7fe495a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4900cd130 con 0x7fe49015a310 2024-03-18T16:22:21.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.897+0000 7fe495a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4900cce50 con 0x7fe49015a6f0 2024-03-18T16:22:21.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.897+0000 7fe495a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe49006d920 con 0x7fe49015baa0 2024-03-18T16:22:21.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.897+0000 7fe48fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe49015a310 0x7fe4901f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:21.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.897+0000 7fe495000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe4900c4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:21.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.897+0000 7fe48f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49015a6f0 0x7fe4901f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:21.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.898+0000 7fe495000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe4900c4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48008/0 (socket says 172.21.15.102:48008) 2024-03-18T16:22:21.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.898+0000 7fe48f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49015a6f0 0x7fe4901f37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54722/0 (socket says 172.21.15.102:54722) 2024-03-18T16:22:21.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.898+0000 7fe495000640 1 -- 172.21.15.102:0/3621514428 learned_addr learned my addr 172.21.15.102:0/3621514428 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:21.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.901+0000 7fe48fe00640 1 -- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 msgr2=0x7fe4900c4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.901+0000 7fe48fe00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe4900c4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.901+0000 7fe48fe00640 1 -- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49015a6f0 msgr2=0x7fe4901f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:21.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.901+0000 7fe48fe00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49015a6f0 0x7fe4901f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:21.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.901+0000 7fe48fe00640 1 -- 172.21.15.102:0/3621514428 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4901f3e00 con 0x7fe49015a310 2024-03-18T16:22:21.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.901+0000 7fe495000640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe4900c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:22:21.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.901+0000 7fe48f400640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49015a6f0 0x7fe4901f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:22:21.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.904+0000 7fe48fe00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe49015a310 0x7fe4901f01f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe48000e870 tx=0x7fe48000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:21.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.910+0000 7fe487e00640 1 -- 172.21.15.102:0/3621514428 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480017070 con 0x7fe49015a310 2024-03-18T16:22:21.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.910+0000 7fe487e00640 1 -- 172.21.15.102:0/3621514428 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe48000ee60 con 0x7fe49015a310 2024-03-18T16:22:21.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.910+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4901f40e0 con 0x7fe49015a310 2024-03-18T16:22:21.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.910+0000 7fe487e00640 1 -- 172.21.15.102:0/3621514428 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48000cb60 con 0x7fe49015a310 2024-03-18T16:22:21.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.910+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4901f46a0 con 0x7fe49015a310 2024-03-18T16:22:21.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:21.911+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe490074fd0 con 0x7fe49015a310 2024-03-18T16:22:22.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.184+0000 7fe487e00640 1 -- 172.21.15.102:0/3621514428 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe48001a070 con 0x7fe49015a310 2024-03-18T16:22:22.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.185+0000 7fe487e00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe47006bc50 0x7fe47006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:22.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.186+0000 7fe48f400640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe47006bc50 0x7fe47006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:22.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.186+0000 7fe48f400640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe47006bc50 0x7fe47006e110 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe4900c9e50 tx=0x7fe47c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:22.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.197+0000 7fe487e00640 1 -- 172.21.15.102:0/3621514428 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fe480057490 con 0x7fe49015a310 2024-03-18T16:22:22.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.197+0000 7fe487e00640 1 -- 172.21.15.102:0/3621514428 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe4800593f0 con 0x7fe49015a310 2024-03-18T16:22:22.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.392+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe4900c3240 con 0x7fe47006bc50 2024-03-18T16:22:22.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.401+0000 7fe487e00640 1 -- 172.21.15.102:0/3621514428 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332746 (secure 0 0 0) 0x7fe4900c3240 con 0x7fe47006bc50 2024-03-18T16:22:22.408 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:22.409 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:22:22.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe47006bc50 msgr2=0x7fe47006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:22.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe47006bc50 0x7fe47006e110 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe4900c9e50 tx=0x7fe47c009040 comp rx=0 tx=0).stop 2024-03-18T16:22:22.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe49015a310 msgr2=0x7fe4901f01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe49015a310 0x7fe4901f01f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe48000e870 tx=0x7fe48000ed30 comp rx=0 tx=0).stop 2024-03-18T16:22:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 shutdown_connections 2024-03-18T16:22:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe47006bc50 0x7fe47006e110 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe49015baa0 0x7fe4900c4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49015a6f0 0x7fe4901f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 --2- 172.21.15.102:0/3621514428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe49015a310 0x7fe4901f01f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 >> 172.21.15.102:0/3621514428 conn(0x7fe49008c1a0 msgr2=0x7fe49014ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 shutdown_connections 2024-03-18T16:22:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.407+0000 7fe495a00640 1 -- 172.21.15.102:0/3621514428 wait complete. 2024-03-18T16:22:22.426 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-18T16:22:20.826317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599870,"num_objects":88,"num_object_clones":0,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":345,"num_write_kb":2474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218872,"kb_used_data":2744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755848,"statfs":{"total":773094113280,"available":772869988352,"internally_reserved":0,"allocated":2809856,"data_stored":2234524,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3546,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":120,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003064"},"pg_stats":[{"pgid":"2.1f","version":"29'11","reported_seq":41,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.667859+0000","last_change":"2024-03-18T16:21:21.402212+0000","last_active":"2024-03-18T16:22:15.667859+0000","last_peered":"2024-03-18T16:22:15.667859+0000","last_clean":"2024-03-18T16:22:15.667859+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:22:15.667859+0000","last_undegraded":"2024-03-18T16:22:15.667859+0000","last_fullsized":"2024-03-18T16:22:15.667859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:33.215915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841700+0000","last_change":"2024-03-18T16:21:23.045605+0000","last_active":"2024-03-18T16:21:54.841700+0000","last_peered":"2024-03-18T16:21:54.841700+0000","last_clean":"2024-03-18T16:21:54.841700+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:21:54.841700+0000","last_undegraded":"2024-03-18T16:21:54.841700+0000","last_fullsized":"2024-03-18T16:21:54.841700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.839646+0000","last_change":"2024-03-18T16:21:29.146958+0000","last_active":"2024-03-18T16:21:54.839646+0000","last_peered":"2024-03-18T16:21:54.839646+0000","last_clean":"2024-03-18T16:21:54.839646+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:21:54.839646+0000","last_undegraded":"2024-03-18T16:21:54.839646+0000","last_fullsized":"2024-03-18T16:21:54.839646+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:55.871927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842657+0000","last_change":"2024-03-18T16:21:21.413060+0000","last_active":"2024-03-18T16:21:54.842657+0000","last_peered":"2024-03-18T16:21:54.842657+0000","last_clean":"2024-03-18T16:21:54.842657+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:21:54.842657+0000","last_undegraded":"2024-03-18T16:21:54.842657+0000","last_fullsized":"2024-03-18T16:21:54.842657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:40.114937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278167+0000","last_change":"2024-03-18T16:21:23.050849+0000","last_active":"2024-03-18T16:21:55.278167+0000","last_peered":"2024-03-18T16:21:55.278167+0000","last_clean":"2024-03-18T16:21:55.278167+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:21:55.278167+0000","last_undegraded":"2024-03-18T16:21:55.278167+0000","last_fullsized":"2024-03-18T16:21:55.278167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147644+0000","last_change":"2024-03-18T16:21:29.065755+0000","last_active":"2024-03-18T16:21:55.147644+0000","last_peered":"2024-03-18T16:21:55.147644+0000","last_clean":"2024-03-18T16:21:55.147644+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:21:55.147644+0000","last_undegraded":"2024-03-18T16:21:55.147644+0000","last_fullsized":"2024-03-18T16:21:55.147644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:45.934586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'7","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.417336+0000","last_change":"2024-03-18T16:21:21.398188+0000","last_active":"2024-03-18T16:22:15.417336+0000","last_peered":"2024-03-18T16:22:15.417336+0000","last_clean":"2024-03-18T16:22:15.417336+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:22:15.417336+0000","last_undegraded":"2024-03-18T16:22:15.417336+0000","last_fullsized":"2024-03-18T16:22:15.417336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064521+0000","last_change":"2024-03-18T16:21:22.946288+0000","last_active":"2024-03-18T16:21:55.064521+0000","last_peered":"2024-03-18T16:21:55.064521+0000","last_clean":"2024-03-18T16:21:55.064521+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:21:55.064521+0000","last_undegraded":"2024-03-18T16:21:55.064521+0000","last_fullsized":"2024-03-18T16:21:55.064521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:08.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147915+0000","last_change":"2024-03-18T16:21:29.065891+0000","last_active":"2024-03-18T16:21:55.147915+0000","last_peered":"2024-03-18T16:21:55.147915+0000","last_clean":"2024-03-18T16:21:55.147915+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:21:55.147915+0000","last_undegraded":"2024-03-18T16:21:55.147915+0000","last_fullsized":"2024-03-18T16:21:55.147915+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265954+0000","last_change":"2024-03-18T16:21:21.397071+0000","last_active":"2024-03-18T16:21:55.265954+0000","last_peered":"2024-03-18T16:21:55.265954+0000","last_clean":"2024-03-18T16:21:55.265954+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:21:55.265954+0000","last_undegraded":"2024-03-18T16:21:55.265954+0000","last_fullsized":"2024-03-18T16:21:55.265954+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065035+0000","last_change":"2024-03-18T16:21:22.946468+0000","last_active":"2024-03-18T16:21:55.065035+0000","last_peered":"2024-03-18T16:21:55.065035+0000","last_clean":"2024-03-18T16:21:55.065035+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:21:55.065035+0000","last_undegraded":"2024-03-18T16:21:55.065035+0000","last_fullsized":"2024-03-18T16:21:55.065035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147794+0000","last_change":"2024-03-18T16:21:29.073724+0000","last_active":"2024-03-18T16:21:55.147794+0000","last_peered":"2024-03-18T16:21:55.147794+0000","last_clean":"2024-03-18T16:21:55.147794+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:21:55.147794+0000","last_undegraded":"2024-03-18T16:21:55.147794+0000","last_fullsized":"2024-03-18T16:21:55.147794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273576+0000","last_change":"2024-03-18T16:21:23.041926+0000","last_active":"2024-03-18T16:21:55.273576+0000","last_peered":"2024-03-18T16:21:55.273576+0000","last_clean":"2024-03-18T16:21:55.273576+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:21:55.273576+0000","last_undegraded":"2024-03-18T16:21:55.273576+0000","last_fullsized":"2024-03-18T16:21:55.273576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:03.532253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.821045+0000","last_change":"2024-03-18T16:21:21.394801+0000","last_active":"2024-03-18T16:22:05.821045+0000","last_peered":"2024-03-18T16:22:05.821045+0000","last_clean":"2024-03-18T16:22:05.821045+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:22:05.821045+0000","last_undegraded":"2024-03-18T16:22:05.821045+0000","last_fullsized":"2024-03-18T16:22:05.821045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148181+0000","last_change":"2024-03-18T16:21:29.073757+0000","last_active":"2024-03-18T16:21:55.148181+0000","last_peered":"2024-03-18T16:21:55.148181+0000","last_clean":"2024-03-18T16:21:55.148181+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:21:55.148181+0000","last_undegraded":"2024-03-18T16:21:55.148181+0000","last_fullsized":"2024-03-18T16:21:55.148181+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841346+0000","last_change":"2024-03-18T16:21:23.060776+0000","last_active":"2024-03-18T16:21:54.841346+0000","last_peered":"2024-03-18T16:21:54.841346+0000","last_clean":"2024-03-18T16:21:54.841346+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:21:54.841346+0000","last_undegraded":"2024-03-18T16:21:54.841346+0000","last_fullsized":"2024-03-18T16:21:54.841346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831874+0000","last_change":"2024-03-18T16:21:21.412733+0000","last_active":"2024-03-18T16:21:54.831874+0000","last_peered":"2024-03-18T16:21:54.831874+0000","last_clean":"2024-03-18T16:21:54.831874+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:21:54.831874+0000","last_undegraded":"2024-03-18T16:21:54.831874+0000","last_fullsized":"2024-03-18T16:21:54.831874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148007+0000","last_change":"2024-03-18T16:21:29.063980+0000","last_active":"2024-03-18T16:21:55.148007+0000","last_peered":"2024-03-18T16:21:55.148007+0000","last_clean":"2024-03-18T16:21:55.148007+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:21:55.148007+0000","last_undegraded":"2024-03-18T16:21:55.148007+0000","last_fullsized":"2024-03-18T16:21:55.148007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:39.469490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.845055+0000","last_change":"2024-03-18T16:21:54.838108+0000","last_active":"2024-03-18T16:21:54.845055+0000","last_peered":"2024-03-18T16:21:54.845055+0000","last_clean":"2024-03-18T16:21:54.845055+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:21:54.845055+0000","last_undegraded":"2024-03-18T16:21:54.845055+0000","last_fullsized":"2024-03-18T16:21:54.845055+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:40.182971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.418782+0000","last_change":"2024-03-18T16:21:21.407132+0000","last_active":"2024-03-18T16:22:15.418782+0000","last_peered":"2024-03-18T16:22:15.418782+0000","last_clean":"2024-03-18T16:22:15.418782+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:22:15.418782+0000","last_undegraded":"2024-03-18T16:22:15.418782+0000","last_fullsized":"2024-03-18T16:22:15.418782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.272088+0000","last_change":"2024-03-18T16:21:55.269661+0000","last_active":"2024-03-18T16:21:55.272088+0000","last_peered":"2024-03-18T16:21:55.272088+0000","last_clean":"2024-03-18T16:21:55.272088+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:21:55.272088+0000","last_undegraded":"2024-03-18T16:21:55.272088+0000","last_fullsized":"2024-03-18T16:21:55.272088+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:38.290937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148142+0000","last_change":"2024-03-18T16:21:22.840551+0000","last_active":"2024-03-18T16:21:55.148142+0000","last_peered":"2024-03-18T16:21:55.148142+0000","last_clean":"2024-03-18T16:21:55.148142+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:21:55.148142+0000","last_undegraded":"2024-03-18T16:21:55.148142+0000","last_fullsized":"2024-03-18T16:21:55.148142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:17.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.822682+0000","last_change":"2024-03-18T16:21:21.400349+0000","last_active":"2024-03-18T16:22:05.822682+0000","last_peered":"2024-03-18T16:22:05.822682+0000","last_clean":"2024-03-18T16:22:05.822682+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:22:05.822682+0000","last_undegraded":"2024-03-18T16:22:05.822682+0000","last_fullsized":"2024-03-18T16:22:05.822682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148262+0000","last_change":"2024-03-18T16:21:29.064624+0000","last_active":"2024-03-18T16:21:55.148262+0000","last_peered":"2024-03-18T16:21:55.148262+0000","last_clean":"2024-03-18T16:21:55.148262+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:21:55.148262+0000","last_undegraded":"2024-03-18T16:21:55.148262+0000","last_fullsized":"2024-03-18T16:21:55.148262+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278343+0000","last_change":"2024-03-18T16:21:23.055982+0000","last_active":"2024-03-18T16:21:55.278343+0000","last_peered":"2024-03-18T16:21:55.278343+0000","last_clean":"2024-03-18T16:21:55.278343+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:21:55.278343+0000","last_undegraded":"2024-03-18T16:21:55.278343+0000","last_fullsized":"2024-03-18T16:21:55.278343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.781767+0000","last_change":"2024-03-18T16:21:21.400843+0000","last_active":"2024-03-18T16:22:18.781767+0000","last_peered":"2024-03-18T16:22:18.781767+0000","last_clean":"2024-03-18T16:22:18.781767+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:22:18.781767+0000","last_undegraded":"2024-03-18T16:22:18.781767+0000","last_fullsized":"2024-03-18T16:22:18.781767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147884+0000","last_change":"2024-03-18T16:21:29.140888+0000","last_active":"2024-03-18T16:21:55.147884+0000","last_peered":"2024-03-18T16:21:55.147884+0000","last_clean":"2024-03-18T16:21:55.147884+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:21:55.147884+0000","last_undegraded":"2024-03-18T16:21:55.147884+0000","last_fullsized":"2024-03-18T16:21:55.147884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842117+0000","last_change":"2024-03-18T16:21:22.994455+0000","last_active":"2024-03-18T16:21:54.842117+0000","last_peered":"2024-03-18T16:21:54.842117+0000","last_clean":"2024-03-18T16:21:54.842117+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:21:54.842117+0000","last_undegraded":"2024-03-18T16:21:54.842117+0000","last_fullsized":"2024-03-18T16:21:54.842117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.421651+0000","last_change":"2024-03-18T16:21:21.402920+0000","last_active":"2024-03-18T16:22:15.421651+0000","last_peered":"2024-03-18T16:22:15.421651+0000","last_clean":"2024-03-18T16:22:15.421651+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:22:15.421651+0000","last_undegraded":"2024-03-18T16:22:15.421651+0000","last_fullsized":"2024-03-18T16:22:15.421651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.838803+0000","last_change":"2024-03-18T16:21:29.145235+0000","last_active":"2024-03-18T16:21:54.838803+0000","last_peered":"2024-03-18T16:21:54.838803+0000","last_clean":"2024-03-18T16:21:54.838803+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:21:54.838803+0000","last_undegraded":"2024-03-18T16:21:54.838803+0000","last_fullsized":"2024-03-18T16:21:54.838803+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:42.379951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.833875+0000","last_change":"2024-03-18T16:21:22.975172+0000","last_active":"2024-03-18T16:21:54.833875+0000","last_peered":"2024-03-18T16:21:54.833875+0000","last_clean":"2024-03-18T16:21:54.833875+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:21:54.833875+0000","last_undegraded":"2024-03-18T16:21:54.833875+0000","last_fullsized":"2024-03-18T16:21:54.833875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'3","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.422581+0000","last_change":"2024-03-18T16:21:21.399688+0000","last_active":"2024-03-18T16:22:15.422581+0000","last_peered":"2024-03-18T16:22:15.422581+0000","last_clean":"2024-03-18T16:22:15.422581+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:22:15.422581+0000","last_undegraded":"2024-03-18T16:22:15.422581+0000","last_fullsized":"2024-03-18T16:22:15.422581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:11.933810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277363+0000","last_change":"2024-03-18T16:21:29.141555+0000","last_active":"2024-03-18T16:21:55.277363+0000","last_peered":"2024-03-18T16:21:55.277363+0000","last_clean":"2024-03-18T16:21:55.277363+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:21:55.277363+0000","last_undegraded":"2024-03-18T16:21:55.277363+0000","last_fullsized":"2024-03-18T16:21:55.277363+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065093+0000","last_change":"2024-03-18T16:21:22.947980+0000","last_active":"2024-03-18T16:21:55.065093+0000","last_peered":"2024-03-18T16:21:55.065093+0000","last_clean":"2024-03-18T16:21:55.065093+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:21:55.065093+0000","last_undegraded":"2024-03-18T16:21:55.065093+0000","last_fullsized":"2024-03-18T16:21:55.065093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:31.739693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834250+0000","last_change":"2024-03-18T16:21:21.400087+0000","last_active":"2024-03-18T16:21:54.834250+0000","last_peered":"2024-03-18T16:21:54.834250+0000","last_clean":"2024-03-18T16:21:54.834250+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:21:54.834250+0000","last_undegraded":"2024-03-18T16:21:54.834250+0000","last_fullsized":"2024-03-18T16:21:54.834250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.830782+0000","last_change":"2024-03-18T16:21:29.076629+0000","last_active":"2024-03-18T16:21:54.830782+0000","last_peered":"2024-03-18T16:21:54.830782+0000","last_clean":"2024-03-18T16:21:54.830782+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:21:54.830782+0000","last_undegraded":"2024-03-18T16:21:54.830782+0000","last_fullsized":"2024-03-18T16:21:54.830782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148132+0000","last_change":"2024-03-18T16:21:23.055810+0000","last_active":"2024-03-18T16:21:55.148132+0000","last_peered":"2024-03-18T16:21:55.148132+0000","last_clean":"2024-03-18T16:21:55.148132+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:21:55.148132+0000","last_undegraded":"2024-03-18T16:21:55.148132+0000","last_fullsized":"2024-03-18T16:21:55.148132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:40.451255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.866023+0000","last_change":"2024-03-18T16:21:21.399134+0000","last_active":"2024-03-18T16:22:18.866023+0000","last_peered":"2024-03-18T16:22:18.866023+0000","last_clean":"2024-03-18T16:22:18.866023+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:22:18.866023+0000","last_undegraded":"2024-03-18T16:22:18.866023+0000","last_fullsized":"2024-03-18T16:22:18.866023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834980+0000","last_change":"2024-03-18T16:21:29.067567+0000","last_active":"2024-03-18T16:21:54.834980+0000","last_peered":"2024-03-18T16:21:54.834980+0000","last_clean":"2024-03-18T16:21:54.834980+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:21:54.834980+0000","last_undegraded":"2024-03-18T16:21:54.834980+0000","last_fullsized":"2024-03-18T16:21:54.834980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.832134+0000","last_change":"2024-03-18T16:21:22.977089+0000","last_active":"2024-03-18T16:21:54.832134+0000","last_peered":"2024-03-18T16:21:54.832134+0000","last_clean":"2024-03-18T16:21:54.832134+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:21:54.832134+0000","last_undegraded":"2024-03-18T16:21:54.832134+0000","last_fullsized":"2024-03-18T16:21:54.832134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:12.723995+0000","last_change":"2024-03-18T16:21:21.408450+0000","last_active":"2024-03-18T16:22:12.723995+0000","last_peered":"2024-03-18T16:22:12.723995+0000","last_clean":"2024-03-18T16:22:12.723995+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:22:12.723995+0000","last_undegraded":"2024-03-18T16:22:12.723995+0000","last_fullsized":"2024-03-18T16:22:12.723995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.835511+0000","last_change":"2024-03-18T16:21:29.076875+0000","last_active":"2024-03-18T16:21:54.835511+0000","last_peered":"2024-03-18T16:21:54.835511+0000","last_clean":"2024-03-18T16:21:54.835511+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:21:54.835511+0000","last_undegraded":"2024-03-18T16:21:54.835511+0000","last_fullsized":"2024-03-18T16:21:54.835511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277013+0000","last_change":"2024-03-18T16:21:23.054497+0000","last_active":"2024-03-18T16:21:55.277013+0000","last_peered":"2024-03-18T16:21:55.277013+0000","last_clean":"2024-03-18T16:21:55.277013+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:21:55.277013+0000","last_undegraded":"2024-03-18T16:21:55.277013+0000","last_fullsized":"2024-03-18T16:21:55.277013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:37.455180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'3","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:12.602372+0000","last_change":"2024-03-18T16:21:21.404754+0000","last_active":"2024-03-18T16:22:12.602372+0000","last_peered":"2024-03-18T16:22:12.602372+0000","last_clean":"2024-03-18T16:22:12.602372+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:22:12.602372+0000","last_undegraded":"2024-03-18T16:22:12.602372+0000","last_fullsized":"2024-03-18T16:22:12.602372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278071+0000","last_change":"2024-03-18T16:21:29.147316+0000","last_active":"2024-03-18T16:21:55.278071+0000","last_peered":"2024-03-18T16:21:55.278071+0000","last_clean":"2024-03-18T16:21:55.278071+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:21:55.278071+0000","last_undegraded":"2024-03-18T16:21:55.278071+0000","last_fullsized":"2024-03-18T16:21:55.278071+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842010+0000","last_change":"2024-03-18T16:21:22.947635+0000","last_active":"2024-03-18T16:21:54.842010+0000","last_peered":"2024-03-18T16:21:54.842010+0000","last_clean":"2024-03-18T16:21:54.842010+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:21:54.842010+0000","last_undegraded":"2024-03-18T16:21:54.842010+0000","last_fullsized":"2024-03-18T16:21:54.842010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.422726+0000","last_change":"2024-03-18T16:21:21.381306+0000","last_active":"2024-03-18T16:22:15.422726+0000","last_peered":"2024-03-18T16:22:15.422726+0000","last_clean":"2024-03-18T16:22:15.422726+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:22:15.422726+0000","last_undegraded":"2024-03-18T16:22:15.422726+0000","last_fullsized":"2024-03-18T16:22:15.422726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":186,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265740+0000","last_change":"2024-03-18T16:21:08.662767+0000","last_active":"2024-03-18T16:21:55.265740+0000","last_peered":"2024-03-18T16:21:55.265740+0000","last_clean":"2024-03-18T16:21:55.265740+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:21:55.265740+0000","last_undegraded":"2024-03-18T16:21:55.265740+0000","last_fullsized":"2024-03-18T16:21:55.265740+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_clean_scrub_stamp":"2024-03-18T16:21:05.847994+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:12:10.494707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273700+0000","last_change":"2024-03-18T16:21:29.145697+0000","last_active":"2024-03-18T16:21:55.273700+0000","last_peered":"2024-03-18T16:21:55.273700+0000","last_clean":"2024-03-18T16:21:55.273700+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:21:55.273700+0000","last_undegraded":"2024-03-18T16:21:55.273700+0000","last_fullsized":"2024-03-18T16:21:55.273700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:46.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266203+0000","last_change":"2024-03-18T16:21:22.995291+0000","last_active":"2024-03-18T16:21:55.266203+0000","last_peered":"2024-03-18T16:21:55.266203+0000","last_clean":"2024-03-18T16:21:55.266203+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:21:55.266203+0000","last_undegraded":"2024-03-18T16:21:55.266203+0000","last_fullsized":"2024-03-18T16:21:55.266203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:32.918246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.446285+0000","last_change":"2024-03-18T16:21:21.382137+0000","last_active":"2024-03-18T16:22:15.446285+0000","last_peered":"2024-03-18T16:22:15.446285+0000","last_clean":"2024-03-18T16:22:15.446285+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:22:15.446285+0000","last_undegraded":"2024-03-18T16:22:15.446285+0000","last_fullsized":"2024-03-18T16:22:15.446285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273492+0000","last_change":"2024-03-18T16:21:29.139432+0000","last_active":"2024-03-18T16:21:55.273492+0000","last_peered":"2024-03-18T16:21:55.273492+0000","last_clean":"2024-03-18T16:21:55.273492+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:21:55.273492+0000","last_undegraded":"2024-03-18T16:21:55.273492+0000","last_fullsized":"2024-03-18T16:21:55.273492+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:14.828125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064603+0000","last_change":"2024-03-18T16:21:22.949578+0000","last_active":"2024-03-18T16:21:55.064603+0000","last_peered":"2024-03-18T16:21:55.064603+0000","last_clean":"2024-03-18T16:21:55.064603+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:21:55.064603+0000","last_undegraded":"2024-03-18T16:21:55.064603+0000","last_fullsized":"2024-03-18T16:21:55.064603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:36:20.997720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.835288+0000","last_change":"2024-03-18T16:21:21.400859+0000","last_active":"2024-03-18T16:21:54.835288+0000","last_peered":"2024-03-18T16:21:54.835288+0000","last_clean":"2024-03-18T16:21:54.835288+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:21:54.835288+0000","last_undegraded":"2024-03-18T16:21:54.835288+0000","last_fullsized":"2024-03-18T16:21:54.835288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277608+0000","last_change":"2024-03-18T16:21:29.076813+0000","last_active":"2024-03-18T16:21:55.277608+0000","last_peered":"2024-03-18T16:21:55.277608+0000","last_clean":"2024-03-18T16:21:55.277608+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:21:55.277608+0000","last_undegraded":"2024-03-18T16:21:55.277608+0000","last_fullsized":"2024-03-18T16:21:55.277608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266390+0000","last_change":"2024-03-18T16:21:23.000009+0000","last_active":"2024-03-18T16:21:55.266390+0000","last_peered":"2024-03-18T16:21:55.266390+0000","last_clean":"2024-03-18T16:21:55.266390+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:21:55.266390+0000","last_undegraded":"2024-03-18T16:21:55.266390+0000","last_fullsized":"2024-03-18T16:21:55.266390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:39.383342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.888727+0000","last_change":"2024-03-18T16:21:21.409271+0000","last_active":"2024-03-18T16:22:18.888727+0000","last_peered":"2024-03-18T16:22:18.888727+0000","last_clean":"2024-03-18T16:22:18.888727+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:22:18.888727+0000","last_undegraded":"2024-03-18T16:22:18.888727+0000","last_fullsized":"2024-03-18T16:22:18.888727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064256+0000","last_change":"2024-03-18T16:21:29.139595+0000","last_active":"2024-03-18T16:21:55.064256+0000","last_peered":"2024-03-18T16:21:55.064256+0000","last_clean":"2024-03-18T16:21:55.064256+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:21:55.064256+0000","last_undegraded":"2024-03-18T16:21:55.064256+0000","last_fullsized":"2024-03-18T16:21:55.064256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:49.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266190+0000","last_change":"2024-03-18T16:21:23.000054+0000","last_active":"2024-03-18T16:21:55.266190+0000","last_peered":"2024-03-18T16:21:55.266190+0000","last_clean":"2024-03-18T16:21:55.266190+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:21:55.266190+0000","last_undegraded":"2024-03-18T16:21:55.266190+0000","last_fullsized":"2024-03-18T16:21:55.266190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:34:41.535042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148537+0000","last_change":"2024-03-18T16:21:21.399315+0000","last_active":"2024-03-18T16:21:55.148537+0000","last_peered":"2024-03-18T16:21:55.148537+0000","last_clean":"2024-03-18T16:21:55.148537+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:21:55.148537+0000","last_undegraded":"2024-03-18T16:21:55.148537+0000","last_fullsized":"2024-03-18T16:21:55.148537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:29.543106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147629+0000","last_change":"2024-03-18T16:21:29.065243+0000","last_active":"2024-03-18T16:21:55.147629+0000","last_peered":"2024-03-18T16:21:55.147629+0000","last_clean":"2024-03-18T16:21:55.147629+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:21:55.147629+0000","last_undegraded":"2024-03-18T16:21:55.147629+0000","last_fullsized":"2024-03-18T16:21:55.147629+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265799+0000","last_change":"2024-03-18T16:21:22.996080+0000","last_active":"2024-03-18T16:21:55.265799+0000","last_peered":"2024-03-18T16:21:55.265799+0000","last_clean":"2024-03-18T16:21:55.265799+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:21:55.265799+0000","last_undegraded":"2024-03-18T16:21:55.265799+0000","last_fullsized":"2024-03-18T16:21:55.265799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":47,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.463973+0000","last_change":"2024-03-18T16:21:21.404930+0000","last_active":"2024-03-18T16:22:15.463973+0000","last_peered":"2024-03-18T16:22:15.463973+0000","last_clean":"2024-03-18T16:22:15.463973+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:22:15.463973+0000","last_undegraded":"2024-03-18T16:22:15.463973+0000","last_fullsized":"2024-03-18T16:22:15.463973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:02.571905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278055+0000","last_change":"2024-03-18T16:21:29.079031+0000","last_active":"2024-03-18T16:21:55.278055+0000","last_peered":"2024-03-18T16:21:55.278055+0000","last_clean":"2024-03-18T16:21:55.278055+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:21:55.278055+0000","last_undegraded":"2024-03-18T16:21:55.278055+0000","last_fullsized":"2024-03-18T16:21:55.278055+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273519+0000","last_change":"2024-03-18T16:21:23.056832+0000","last_active":"2024-03-18T16:21:55.273519+0000","last_peered":"2024-03-18T16:21:55.273519+0000","last_clean":"2024-03-18T16:21:55.273519+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:21:55.273519+0000","last_undegraded":"2024-03-18T16:21:55.273519+0000","last_fullsized":"2024-03-18T16:21:55.273519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:56.124747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.396156+0000","last_change":"2024-03-18T16:21:21.404392+0000","last_active":"2024-03-18T16:22:15.396156+0000","last_peered":"2024-03-18T16:22:15.396156+0000","last_clean":"2024-03-18T16:22:15.396156+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:22:15.396156+0000","last_undegraded":"2024-03-18T16:22:15.396156+0000","last_fullsized":"2024-03-18T16:22:15.396156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841565+0000","last_change":"2024-03-18T16:21:29.139956+0000","last_active":"2024-03-18T16:21:54.841565+0000","last_peered":"2024-03-18T16:21:54.841565+0000","last_clean":"2024-03-18T16:21:54.841565+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:21:54.841565+0000","last_undegraded":"2024-03-18T16:21:54.841565+0000","last_fullsized":"2024-03-18T16:21:54.841565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:52.356673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265953+0000","last_change":"2024-03-18T16:21:22.995275+0000","last_active":"2024-03-18T16:21:55.265953+0000","last_peered":"2024-03-18T16:21:55.265953+0000","last_clean":"2024-03-18T16:21:55.265953+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:21:55.265953+0000","last_undegraded":"2024-03-18T16:21:55.265953+0000","last_fullsized":"2024-03-18T16:21:55.265953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:48.024985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.429963+0000","last_change":"2024-03-18T16:21:21.412044+0000","last_active":"2024-03-18T16:22:15.429963+0000","last_peered":"2024-03-18T16:22:15.429963+0000","last_clean":"2024-03-18T16:22:15.429963+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:22:15.429963+0000","last_undegraded":"2024-03-18T16:22:15.429963+0000","last_fullsized":"2024-03-18T16:22:15.429963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.270056+0000","last_change":"2024-03-18T16:21:55.270056+0000","last_active":"2024-03-18T16:21:55.270056+0000","last_peered":"2024-03-18T16:21:55.270056+0000","last_clean":"2024-03-18T16:21:55.270056+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:21:55.270056+0000","last_undegraded":"2024-03-18T16:21:55.270056+0000","last_fullsized":"2024-03-18T16:21:55.270056+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:50.466826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277605+0000","last_change":"2024-03-18T16:21:23.056116+0000","last_active":"2024-03-18T16:21:55.277605+0000","last_peered":"2024-03-18T16:21:55.277605+0000","last_clean":"2024-03-18T16:21:55.277605+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:21:55.277605+0000","last_undegraded":"2024-03-18T16:21:55.277605+0000","last_fullsized":"2024-03-18T16:21:55.277605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:18.455315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'7","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:12.722073+0000","last_change":"2024-03-18T16:21:21.400378+0000","last_active":"2024-03-18T16:22:12.722073+0000","last_peered":"2024-03-18T16:22:12.722073+0000","last_clean":"2024-03-18T16:22:12.722073+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:22:12.722073+0000","last_undegraded":"2024-03-18T16:22:12.722073+0000","last_fullsized":"2024-03-18T16:22:12.722073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064943+0000","last_change":"2024-03-18T16:21:29.141554+0000","last_active":"2024-03-18T16:21:55.064943+0000","last_peered":"2024-03-18T16:21:55.064943+0000","last_clean":"2024-03-18T16:21:55.064943+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:21:55.064943+0000","last_undegraded":"2024-03-18T16:21:55.064943+0000","last_fullsized":"2024-03-18T16:21:55.064943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:25.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266115+0000","last_change":"2024-03-18T16:21:22.996694+0000","last_active":"2024-03-18T16:21:55.266115+0000","last_peered":"2024-03-18T16:21:55.266115+0000","last_clean":"2024-03-18T16:21:55.266115+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:21:55.266115+0000","last_undegraded":"2024-03-18T16:21:55.266115+0000","last_fullsized":"2024-03-18T16:21:55.266115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:03.704090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":44,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.784931+0000","last_change":"2024-03-18T16:21:21.412149+0000","last_active":"2024-03-18T16:22:18.784931+0000","last_peered":"2024-03-18T16:22:18.784931+0000","last_clean":"2024-03-18T16:22:18.784931+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:22:18.784931+0000","last_undegraded":"2024-03-18T16:22:18.784931+0000","last_fullsized":"2024-03-18T16:22:18.784931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831282+0000","last_change":"2024-03-18T16:21:29.077350+0000","last_active":"2024-03-18T16:21:54.831282+0000","last_peered":"2024-03-18T16:21:54.831282+0000","last_clean":"2024-03-18T16:21:54.831282+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:21:54.831282+0000","last_undegraded":"2024-03-18T16:21:54.831282+0000","last_fullsized":"2024-03-18T16:21:54.831282+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148199+0000","last_change":"2024-03-18T16:21:22.992928+0000","last_active":"2024-03-18T16:21:55.148199+0000","last_peered":"2024-03-18T16:21:55.148199+0000","last_clean":"2024-03-18T16:21:55.148199+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:21:55.148199+0000","last_undegraded":"2024-03-18T16:21:55.148199+0000","last_fullsized":"2024-03-18T16:21:55.148199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:50.978267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.826412+0000","last_change":"2024-03-18T16:21:21.403749+0000","last_active":"2024-03-18T16:22:05.826412+0000","last_peered":"2024-03-18T16:22:05.826412+0000","last_clean":"2024-03-18T16:22:05.826412+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:22:05.826412+0000","last_undegraded":"2024-03-18T16:22:05.826412+0000","last_fullsized":"2024-03-18T16:22:05.826412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:29:30.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147816+0000","last_change":"2024-03-18T16:21:29.065378+0000","last_active":"2024-03-18T16:21:55.147816+0000","last_peered":"2024-03-18T16:21:55.147816+0000","last_clean":"2024-03-18T16:21:55.147816+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:21:55.147816+0000","last_undegraded":"2024-03-18T16:21:55.147816+0000","last_fullsized":"2024-03-18T16:21:55.147816+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:22:12.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278432+0000","last_change":"2024-03-18T16:21:23.054168+0000","last_active":"2024-03-18T16:21:55.278432+0000","last_peered":"2024-03-18T16:21:55.278432+0000","last_clean":"2024-03-18T16:21:55.278432+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:21:55.278432+0000","last_undegraded":"2024-03-18T16:21:55.278432+0000","last_fullsized":"2024-03-18T16:21:55.278432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.861925+0000","last_change":"2024-03-18T16:21:21.398307+0000","last_active":"2024-03-18T16:22:18.861925+0000","last_peered":"2024-03-18T16:22:18.861925+0000","last_clean":"2024-03-18T16:22:18.861925+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:22:18.861925+0000","last_undegraded":"2024-03-18T16:22:18.861925+0000","last_fullsized":"2024-03-18T16:22:18.861925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840228+0000","last_change":"2024-03-18T16:21:29.065855+0000","last_active":"2024-03-18T16:21:54.840228+0000","last_peered":"2024-03-18T16:21:54.840228+0000","last_clean":"2024-03-18T16:21:54.840228+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:21:54.840228+0000","last_undegraded":"2024-03-18T16:21:54.840228+0000","last_fullsized":"2024-03-18T16:21:54.840228+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:43.207302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064726+0000","last_change":"2024-03-18T16:21:22.993606+0000","last_active":"2024-03-18T16:21:55.064726+0000","last_peered":"2024-03-18T16:21:55.064726+0000","last_clean":"2024-03-18T16:21:55.064726+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:21:55.064726+0000","last_undegraded":"2024-03-18T16:21:55.064726+0000","last_fullsized":"2024-03-18T16:21:55.064726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:38:23.121609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.041784+0000","last_change":"2024-03-18T16:21:21.399588+0000","last_active":"2024-03-18T16:22:06.041784+0000","last_peered":"2024-03-18T16:22:06.041784+0000","last_clean":"2024-03-18T16:22:06.041784+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:22:06.041784+0000","last_undegraded":"2024-03-18T16:22:06.041784+0000","last_fullsized":"2024-03-18T16:22:06.041784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.843888+0000","last_change":"2024-03-18T16:21:29.144832+0000","last_active":"2024-03-18T16:21:54.843888+0000","last_peered":"2024-03-18T16:21:54.843888+0000","last_clean":"2024-03-18T16:21:54.843888+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:21:54.843888+0000","last_undegraded":"2024-03-18T16:21:54.843888+0000","last_fullsized":"2024-03-18T16:21:54.843888+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:10.103662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277724+0000","last_change":"2024-03-18T16:21:29.148215+0000","last_active":"2024-03-18T16:21:55.277724+0000","last_peered":"2024-03-18T16:21:55.277724+0000","last_clean":"2024-03-18T16:21:55.277724+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:21:55.277724+0000","last_undegraded":"2024-03-18T16:21:55.277724+0000","last_fullsized":"2024-03-18T16:21:55.277724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.829181+0000","last_change":"2024-03-18T16:21:22.980300+0000","last_active":"2024-03-18T16:21:54.829181+0000","last_peered":"2024-03-18T16:21:54.829181+0000","last_clean":"2024-03-18T16:21:54.829181+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:21:54.829181+0000","last_undegraded":"2024-03-18T16:21:54.829181+0000","last_fullsized":"2024-03-18T16:21:54.829181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.423345+0000","last_change":"2024-03-18T16:21:21.329294+0000","last_active":"2024-03-18T16:22:15.423345+0000","last_peered":"2024-03-18T16:22:15.423345+0000","last_clean":"2024-03-18T16:22:15.423345+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:22:15.423345+0000","last_undegraded":"2024-03-18T16:22:15.423345+0000","last_fullsized":"2024-03-18T16:22:15.423345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273909+0000","last_change":"2024-03-18T16:21:29.076706+0000","last_active":"2024-03-18T16:21:55.273909+0000","last_peered":"2024-03-18T16:21:55.273909+0000","last_clean":"2024-03-18T16:21:55.273909+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:21:55.273909+0000","last_undegraded":"2024-03-18T16:21:55.273909+0000","last_fullsized":"2024-03-18T16:21:55.273909+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:57.880669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840470+0000","last_change":"2024-03-18T16:21:23.052989+0000","last_active":"2024-03-18T16:21:54.840470+0000","last_peered":"2024-03-18T16:21:54.840470+0000","last_clean":"2024-03-18T16:21:54.840470+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:21:54.840470+0000","last_undegraded":"2024-03-18T16:21:54.840470+0000","last_fullsized":"2024-03-18T16:21:54.840470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:12.606644+0000","last_change":"2024-03-18T16:21:21.413020+0000","last_active":"2024-03-18T16:22:12.606644+0000","last_peered":"2024-03-18T16:22:12.606644+0000","last_clean":"2024-03-18T16:22:12.606644+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:22:12.606644+0000","last_undegraded":"2024-03-18T16:22:12.606644+0000","last_fullsized":"2024-03-18T16:22:12.606644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:49.237529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.400410+0000","last_change":"2024-03-18T16:21:21.397354+0000","last_active":"2024-03-18T16:22:15.400410+0000","last_peered":"2024-03-18T16:22:15.400410+0000","last_clean":"2024-03-18T16:22:15.400410+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:22:15.400410+0000","last_undegraded":"2024-03-18T16:22:15.400410+0000","last_fullsized":"2024-03-18T16:22:15.400410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278026+0000","last_change":"2024-03-18T16:21:23.059441+0000","last_active":"2024-03-18T16:21:55.278026+0000","last_peered":"2024-03-18T16:21:55.278026+0000","last_clean":"2024-03-18T16:21:55.278026+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:21:55.278026+0000","last_undegraded":"2024-03-18T16:21:55.278026+0000","last_fullsized":"2024-03-18T16:21:55.278026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827518+0000","last_change":"2024-03-18T16:21:29.058800+0000","last_active":"2024-03-18T16:21:54.827518+0000","last_peered":"2024-03-18T16:21:54.827518+0000","last_clean":"2024-03-18T16:21:54.827518+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:21:54.827518+0000","last_undegraded":"2024-03-18T16:21:54.827518+0000","last_fullsized":"2024-03-18T16:21:54.827518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:20.279579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.074918+0000","last_change":"2024-03-18T16:21:21.412536+0000","last_active":"2024-03-18T16:22:06.074918+0000","last_peered":"2024-03-18T16:22:06.074918+0000","last_clean":"2024-03-18T16:22:06.074918+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:22:06.074918+0000","last_undegraded":"2024-03-18T16:22:06.074918+0000","last_fullsized":"2024-03-18T16:22:06.074918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148039+0000","last_change":"2024-03-18T16:21:22.974963+0000","last_active":"2024-03-18T16:21:55.148039+0000","last_peered":"2024-03-18T16:21:55.148039+0000","last_clean":"2024-03-18T16:21:55.148039+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:21:55.148039+0000","last_undegraded":"2024-03-18T16:21:55.148039+0000","last_fullsized":"2024-03-18T16:21:55.148039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:51.820583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.151933+0000","last_change":"2024-03-18T16:21:55.151933+0000","last_active":"2024-03-18T16:21:55.151933+0000","last_peered":"2024-03-18T16:21:55.151933+0000","last_clean":"2024-03-18T16:21:55.151933+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:21:55.151933+0000","last_undegraded":"2024-03-18T16:21:55.151933+0000","last_fullsized":"2024-03-18T16:21:55.151933+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:38.438575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273462+0000","last_change":"2024-03-18T16:21:23.056857+0000","last_active":"2024-03-18T16:21:55.273462+0000","last_peered":"2024-03-18T16:21:55.273462+0000","last_clean":"2024-03-18T16:21:55.273462+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:21:55.273462+0000","last_undegraded":"2024-03-18T16:21:55.273462+0000","last_fullsized":"2024-03-18T16:21:55.273462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:13.728335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827238+0000","last_change":"2024-03-18T16:21:22.949289+0000","last_active":"2024-03-18T16:21:54.827238+0000","last_peered":"2024-03-18T16:21:54.827238+0000","last_clean":"2024-03-18T16:21:54.827238+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:21:54.827238+0000","last_undegraded":"2024-03-18T16:21:54.827238+0000","last_fullsized":"2024-03-18T16:21:54.827238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:51:51.357680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.848377+0000","last_change":"2024-03-18T16:21:54.845858+0000","last_active":"2024-03-18T16:21:54.848377+0000","last_peered":"2024-03-18T16:21:54.848377+0000","last_clean":"2024-03-18T16:21:54.848377+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:21:54.848377+0000","last_undegraded":"2024-03-18T16:21:54.848377+0000","last_fullsized":"2024-03-18T16:21:54.848377+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277673+0000","last_change":"2024-03-18T16:21:23.059438+0000","last_active":"2024-03-18T16:21:55.277673+0000","last_peered":"2024-03-18T16:21:55.277673+0000","last_clean":"2024-03-18T16:21:55.277673+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:21:55.277673+0000","last_undegraded":"2024-03-18T16:21:55.277673+0000","last_fullsized":"2024-03-18T16:21:55.277673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841882+0000","last_change":"2024-03-18T16:21:22.849115+0000","last_active":"2024-03-18T16:21:54.841882+0000","last_peered":"2024-03-18T16:21:54.841882+0000","last_clean":"2024-03-18T16:21:54.841882+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:21:54.841882+0000","last_undegraded":"2024-03-18T16:21:54.841882+0000","last_fullsized":"2024-03-18T16:21:54.841882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:57.469261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834154+0000","last_change":"2024-03-18T16:21:22.980253+0000","last_active":"2024-03-18T16:21:54.834154+0000","last_peered":"2024-03-18T16:21:54.834154+0000","last_clean":"2024-03-18T16:21:54.834154+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:21:54.834154+0000","last_undegraded":"2024-03-18T16:21:54.834154+0000","last_fullsized":"2024-03-18T16:21:54.834154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273836+0000","last_change":"2024-03-18T16:21:23.042188+0000","last_active":"2024-03-18T16:21:55.273836+0000","last_peered":"2024-03-18T16:21:55.273836+0000","last_clean":"2024-03-18T16:21:55.273836+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:21:55.273836+0000","last_undegraded":"2024-03-18T16:21:55.273836+0000","last_fullsized":"2024-03-18T16:21:55.273836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841847+0000","last_change":"2024-03-18T16:21:22.851677+0000","last_active":"2024-03-18T16:21:54.841847+0000","last_peered":"2024-03-18T16:21:54.841847+0000","last_clean":"2024-03-18T16:21:54.841847+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:21:54.841847+0000","last_undegraded":"2024-03-18T16:21:54.841847+0000","last_fullsized":"2024-03-18T16:21:54.841847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064494+0000","last_change":"2024-03-18T16:21:22.947869+0000","last_active":"2024-03-18T16:21:55.064494+0000","last_peered":"2024-03-18T16:21:55.064494+0000","last_clean":"2024-03-18T16:21:55.064494+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:21:55.064494+0000","last_undegraded":"2024-03-18T16:21:55.064494+0000","last_fullsized":"2024-03-18T16:21:55.064494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840151+0000","last_change":"2024-03-18T16:21:22.851663+0000","last_active":"2024-03-18T16:21:54.840151+0000","last_peered":"2024-03-18T16:21:54.840151+0000","last_clean":"2024-03-18T16:21:54.840151+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:21:54.840151+0000","last_undegraded":"2024-03-18T16:21:54.840151+0000","last_fullsized":"2024-03-18T16:21:54.840151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.271992+0000","last_change":"2024-03-18T16:21:55.268883+0000","last_active":"2024-03-18T16:21:55.271992+0000","last_peered":"2024-03-18T16:21:55.271992+0000","last_clean":"2024-03-18T16:21:55.271992+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:21:55.271992+0000","last_undegraded":"2024-03-18T16:21:55.271992+0000","last_fullsized":"2024-03-18T16:21:55.271992+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277397+0000","last_change":"2024-03-18T16:21:23.059733+0000","last_active":"2024-03-18T16:21:55.277397+0000","last_peered":"2024-03-18T16:21:55.277397+0000","last_clean":"2024-03-18T16:21:55.277397+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:21:55.277397+0000","last_undegraded":"2024-03-18T16:21:55.277397+0000","last_fullsized":"2024-03-18T16:21:55.277397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273660+0000","last_change":"2024-03-18T16:21:23.042457+0000","last_active":"2024-03-18T16:21:55.273660+0000","last_peered":"2024-03-18T16:21:55.273660+0000","last_clean":"2024-03-18T16:21:55.273660+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:21:55.273660+0000","last_undegraded":"2024-03-18T16:21:55.273660+0000","last_fullsized":"2024-03-18T16:21:55.273660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:09.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.845542+0000","last_change":"2024-03-18T16:21:54.842423+0000","last_active":"2024-03-18T16:21:54.845542+0000","last_peered":"2024-03-18T16:21:54.845542+0000","last_clean":"2024-03-18T16:21:54.845542+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:21:54.845542+0000","last_undegraded":"2024-03-18T16:21:54.845542+0000","last_fullsized":"2024-03-18T16:21:54.845542+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:36.656058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.828079+0000","last_change":"2024-03-18T16:21:23.045061+0000","last_active":"2024-03-18T16:21:54.828079+0000","last_peered":"2024-03-18T16:21:54.828079+0000","last_clean":"2024-03-18T16:21:54.828079+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:21:54.828079+0000","last_undegraded":"2024-03-18T16:21:54.828079+0000","last_fullsized":"2024-03-18T16:21:54.828079+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:54.119675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065127+0000","last_change":"2024-03-18T16:21:22.993632+0000","last_active":"2024-03-18T16:21:55.065127+0000","last_peered":"2024-03-18T16:21:55.065127+0000","last_clean":"2024-03-18T16:21:55.065127+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:21:55.065127+0000","last_undegraded":"2024-03-18T16:21:55.065127+0000","last_fullsized":"2024-03-18T16:21:55.065127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:35:42.249037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064410+0000","last_change":"2024-03-18T16:21:23.056575+0000","last_active":"2024-03-18T16:21:55.064410+0000","last_peered":"2024-03-18T16:21:55.064410+0000","last_clean":"2024-03-18T16:21:55.064410+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:21:55.064410+0000","last_undegraded":"2024-03-18T16:21:55.064410+0000","last_fullsized":"2024-03-18T16:21:55.064410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831640+0000","last_change":"2024-03-18T16:21:22.973283+0000","last_active":"2024-03-18T16:21:54.831640+0000","last_peered":"2024-03-18T16:21:54.831640+0000","last_clean":"2024-03-18T16:21:54.831640+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:21:54.831640+0000","last_undegraded":"2024-03-18T16:21:54.831640+0000","last_fullsized":"2024-03-18T16:21:54.831640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277550+0000","last_change":"2024-03-18T16:21:23.057150+0000","last_active":"2024-03-18T16:21:55.277550+0000","last_peered":"2024-03-18T16:21:55.277550+0000","last_clean":"2024-03-18T16:21:55.277550+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:21:55.277550+0000","last_undegraded":"2024-03-18T16:21:55.277550+0000","last_fullsized":"2024-03-18T16:21:55.277550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841077+0000","last_change":"2024-03-18T16:21:23.047968+0000","last_active":"2024-03-18T16:21:54.841077+0000","last_peered":"2024-03-18T16:21:54.841077+0000","last_clean":"2024-03-18T16:21:54.841077+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:21:54.841077+0000","last_undegraded":"2024-03-18T16:21:54.841077+0000","last_fullsized":"2024-03-18T16:21:54.841077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148114+0000","last_change":"2024-03-18T16:21:23.041794+0000","last_active":"2024-03-18T16:21:55.148114+0000","last_peered":"2024-03-18T16:21:55.148114+0000","last_clean":"2024-03-18T16:21:55.148114+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:21:55.148114+0000","last_undegraded":"2024-03-18T16:21:55.148114+0000","last_fullsized":"2024-03-18T16:21:55.148114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:13.065494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273737+0000","last_change":"2024-03-18T16:21:23.036940+0000","last_active":"2024-03-18T16:21:55.273737+0000","last_peered":"2024-03-18T16:21:55.273737+0000","last_clean":"2024-03-18T16:21:55.273737+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:21:55.273737+0000","last_undegraded":"2024-03-18T16:21:55.273737+0000","last_fullsized":"2024-03-18T16:21:55.273737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:45:38.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273810+0000","last_change":"2024-03-18T16:21:23.042140+0000","last_active":"2024-03-18T16:21:55.273810+0000","last_peered":"2024-03-18T16:21:55.273810+0000","last_clean":"2024-03-18T16:21:55.273810+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:21:55.273810+0000","last_undegraded":"2024-03-18T16:21:55.273810+0000","last_fullsized":"2024-03-18T16:21:55.273810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266289+0000","last_change":"2024-03-18T16:21:22.995117+0000","last_active":"2024-03-18T16:21:55.266289+0000","last_peered":"2024-03-18T16:21:55.266289+0000","last_clean":"2024-03-18T16:21:55.266289+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:21:55.266289+0000","last_undegraded":"2024-03-18T16:21:55.266289+0000","last_fullsized":"2024-03-18T16:21:55.266289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:57.544738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.839997+0000","last_change":"2024-03-18T16:21:22.948123+0000","last_active":"2024-03-18T16:21:54.839997+0000","last_peered":"2024-03-18T16:21:54.839997+0000","last_clean":"2024-03-18T16:21:54.839997+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:21:54.839997+0000","last_undegraded":"2024-03-18T16:21:54.839997+0000","last_fullsized":"2024-03-18T16:21:54.839997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265696+0000","last_change":"2024-03-18T16:21:23.049877+0000","last_active":"2024-03-18T16:21:55.265696+0000","last_peered":"2024-03-18T16:21:55.265696+0000","last_clean":"2024-03-18T16:21:55.265696+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:21:55.265696+0000","last_undegraded":"2024-03-18T16:21:55.265696+0000","last_fullsized":"2024-03-18T16:21:55.265696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064503+0000","last_change":"2024-03-18T16:21:23.049483+0000","last_active":"2024-03-18T16:21:55.064503+0000","last_peered":"2024-03-18T16:21:55.064503+0000","last_clean":"2024-03-18T16:21:55.064503+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:21:55.064503+0000","last_undegraded":"2024-03-18T16:21:55.064503+0000","last_fullsized":"2024-03-18T16:21:55.064503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265858+0000","last_change":"2024-03-18T16:21:22.995726+0000","last_active":"2024-03-18T16:21:55.265858+0000","last_peered":"2024-03-18T16:21:55.265858+0000","last_clean":"2024-03-18T16:21:55.265858+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:21:55.265858+0000","last_undegraded":"2024-03-18T16:21:55.265858+0000","last_fullsized":"2024-03-18T16:21:55.265858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064410+0000","last_change":"2024-03-18T16:21:22.949640+0000","last_active":"2024-03-18T16:21:55.064410+0000","last_peered":"2024-03-18T16:21:55.064410+0000","last_clean":"2024-03-18T16:21:55.064410+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:21:55.064410+0000","last_undegraded":"2024-03-18T16:21:55.064410+0000","last_fullsized":"2024-03-18T16:21:55.064410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:21.734293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277697+0000","last_change":"2024-03-18T16:21:23.050203+0000","last_active":"2024-03-18T16:21:55.277697+0000","last_peered":"2024-03-18T16:21:55.277697+0000","last_clean":"2024-03-18T16:21:55.277697+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:21:55.277697+0000","last_undegraded":"2024-03-18T16:21:55.277697+0000","last_fullsized":"2024-03-18T16:21:55.277697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:21.982408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834586+0000","last_change":"2024-03-18T16:21:23.053003+0000","last_active":"2024-03-18T16:21:54.834586+0000","last_peered":"2024-03-18T16:21:54.834586+0000","last_clean":"2024-03-18T16:21:54.834586+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:21:54.834586+0000","last_undegraded":"2024-03-18T16:21:54.834586+0000","last_fullsized":"2024-03-18T16:21:54.834586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266295+0000","last_change":"2024-03-18T16:21:23.041863+0000","last_active":"2024-03-18T16:21:55.266295+0000","last_peered":"2024-03-18T16:21:55.266295+0000","last_clean":"2024-03-18T16:21:55.266295+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:21:55.266295+0000","last_undegraded":"2024-03-18T16:21:55.266295+0000","last_fullsized":"2024-03-18T16:21:55.266295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:26.190759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9502,"num_objects":86,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":186,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":57,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":18976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509459,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":397312,"data_stored":721208,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509460,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":130832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509458,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":134146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":130446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509458,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":393216,"data_stored":720564,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":130964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":134134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":344064,"data_stored":132230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:22:22.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:22:22.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.640+0000 7f3aa9c00640 1 Processor -- start 2024-03-18T16:22:22.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.641+0000 7f3aa9c00640 1 -- start start 2024-03-18T16:22:22.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.642+0000 7f3aa9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa415ae70 0x7f3aa40ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:22.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.642+0000 7f3aa9c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3aa40cc4f0 0x7f3aa40cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:22.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.642+0000 7f3aa9c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa4154ab0 con 0x7f3aa40c9be0 2024-03-18T16:22:22.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.642+0000 7f3aa9c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3aa40cce80 con 0x7f3aa40cc4f0 2024-03-18T16:22:22.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.642+0000 7f3aa9c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa406d2f0 con 0x7f3aa415ae70 2024-03-18T16:22:22.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.642+0000 7f3aa3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa415ae70 0x7f3aa40ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:22.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.643+0000 7f3aa9200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3aa40cc4f0 0x7f3aa40cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:22.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.643+0000 7f3aa3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa415ae70 0x7f3aa40ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48024/0 (socket says 172.21.15.102:48024) 2024-03-18T16:22:22.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.643+0000 7f3aa3e00640 1 -- 172.21.15.102:0/2534945762 learned_addr learned my addr 172.21.15.102:0/2534945762 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:22.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.645+0000 7f3aa3e00640 1 -- 172.21.15.102:0/2534945762 >> v2:172.21.15.102:3300/0 conn(0x7f3aa40cc4f0 msgr2=0x7f3aa40cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:22.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.645+0000 7f3aa3e00640 1 --2- 172.21.15.102:0/2534945762 >> v2:172.21.15.102:3300/0 conn(0x7f3aa40cc4f0 0x7f3aa40cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:22.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.645+0000 7f3aa3e00640 1 -- 172.21.15.102:0/2534945762 >> v1:172.21.15.102:6789/0 conn(0x7f3aa40c9be0 legacy=0x7f3aa40d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:22.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.645+0000 7f3aa3e00640 1 -- 172.21.15.102:0/2534945762 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa415b250 con 0x7f3aa415ae70 2024-03-18T16:22:22.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.647+0000 7f3aa3e00640 1 --2- 172.21.15.102:0/2534945762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa415ae70 0x7f3aa40ca490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009870 tx=0x7f3a9402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fcb4f751a7035dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:22.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.651+0000 7f3aa2a00640 1 -- 172.21.15.102:0/2534945762 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94034070 con 0x7f3aa415ae70 2024-03-18T16:22:22.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.651+0000 7f3aa2a00640 1 -- 172.21.15.102:0/2534945762 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a94030840 con 0x7f3aa415ae70 2024-03-18T16:22:22.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.651+0000 7f3aa2a00640 1 -- 172.21.15.102:0/2534945762 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94030c10 con 0x7f3aa415ae70 2024-03-18T16:22:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.652+0000 7f3aa9c00640 1 -- 172.21.15.102:0/2534945762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa415ae70 msgr2=0x7f3aa40ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.652+0000 7f3aa9c00640 1 --2- 172.21.15.102:0/2534945762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa415ae70 0x7f3aa40ca490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009870 tx=0x7f3a9402b980 comp rx=0 tx=0).stop 2024-03-18T16:22:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.652+0000 7f3aa9c00640 1 -- 172.21.15.102:0/2534945762 shutdown_connections 2024-03-18T16:22:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.652+0000 7f3aa9c00640 1 --2- 172.21.15.102:0/2534945762 >> v2:172.21.15.102:3300/0 conn(0x7f3aa40cc4f0 0x7f3aa40cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.652+0000 7f3aa9c00640 1 --2- 172.21.15.102:0/2534945762 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa415ae70 0x7f3aa40ca490 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.652+0000 7f3aa9c00640 1 -- 172.21.15.102:0/2534945762 >> 172.21.15.102:0/2534945762 conn(0x7f3aa40673a0 msgr2=0x7f3aa40677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.652+0000 7f3aa9c00640 1 -- 172.21.15.102:0/2534945762 shutdown_connections 2024-03-18T16:22:22.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.652+0000 7f3aa9c00640 1 -- 172.21.15.102:0/2534945762 wait complete. 2024-03-18T16:22:22.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.655+0000 7f3aa9c00640 1 Processor -- start 2024-03-18T16:22:22.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.655+0000 7f3aa9c00640 1 -- start start 2024-03-18T16:22:22.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.656+0000 7f3aa9c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa40c9be0 0x7f3aa41f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:22.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.656+0000 7f3aa9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa40cc4f0 0x7f3aa41f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:22.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.656+0000 7f3aa9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3aa415ae70 0x7f3aa4153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:22.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.656+0000 7f3aa9c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa406db90 con 0x7f3aa415ae70 2024-03-18T16:22:22.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.656+0000 7f3aa9c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa406d8b0 con 0x7f3aa40c9be0 2024-03-18T16:22:22.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.656+0000 7f3aa9c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa406de70 con 0x7f3aa40cc4f0 2024-03-18T16:22:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.657+0000 7f3aa9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3aa415ae70 0x7f3aa4153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.657+0000 7f3aa3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa40c9be0 0x7f3aa41f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.657+0000 7f3aa3400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa40cc4f0 0x7f3aa41f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.657+0000 7f3aa3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa40c9be0 0x7f3aa41f17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54728/0 (socket says 172.21.15.102:54728) 2024-03-18T16:22:22.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.657+0000 7f3aa3e00640 1 -- 172.21.15.102:0/1170146104 learned_addr learned my addr 172.21.15.102:0/1170146104 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.661+0000 7f3aa3e00640 1 -- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa40cc4f0 msgr2=0x7f3aa41f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.661+0000 7f3aa3e00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa40cc4f0 0x7f3aa41f1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.661+0000 7f3aa3e00640 1 -- 172.21.15.102:0/1170146104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3aa415ae70 msgr2=0x7f3aa4153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.661+0000 7f3aa3e00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3aa415ae70 0x7f3aa4153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.661+0000 7f3aa3e00640 1 -- 172.21.15.102:0/1170146104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa40c3350 con 0x7f3aa40c9be0 2024-03-18T16:22:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.664+0000 7f3aa3e00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa40c9be0 0x7f3aa41f17a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3a9402b9c0 tx=0x7f3a940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:22.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.666+0000 7f3a9be00640 1 -- 172.21.15.102:0/1170146104 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a94034040 con 0x7f3aa40c9be0 2024-03-18T16:22:22.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.666+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa40c35f0 con 0x7f3aa40c9be0 2024-03-18T16:22:22.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.667+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa40c45f0 con 0x7f3aa40c9be0 2024-03-18T16:22:22.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.668+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa4074fd0 con 0x7f3aa40c9be0 2024-03-18T16:22:22.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.668+0000 7f3a9be00640 1 -- 172.21.15.102:0/1170146104 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a94030840 con 0x7f3aa40c9be0 2024-03-18T16:22:22.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.668+0000 7f3a9be00640 1 -- 172.21.15.102:0/1170146104 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9404d450 con 0x7f3aa40c9be0 2024-03-18T16:22:22.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.941+0000 7f3a9be00640 1 -- 172.21.15.102:0/1170146104 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3a9404b060 con 0x7f3aa40c9be0 2024-03-18T16:22:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.943+0000 7f3a9be00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a8406bbd0 0x7f3a8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.944+0000 7f3aa3400640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a8406bbd0 0x7f3a8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.944+0000 7f3aa3400640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a8406bbd0 0x7f3a8406e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40c5310 tx=0x7f3a90009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:22.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.947+0000 7f3a9be00640 1 -- 172.21.15.102:0/1170146104 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f3a94079300 con 0x7f3aa40c9be0 2024-03-18T16:22:22.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:22.972+0000 7f3a9be00640 1 -- 172.21.15.102:0/1170146104 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3a9403f070 con 0x7f3aa40c9be0 2024-03-18T16:22:23.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.186+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3aa40ce960 con 0x7f3a8406bbd0 2024-03-18T16:22:23.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.203+0000 7f3a9be00640 1 -- 172.21.15.102:0/1170146104 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332783 (secure 0 0 0) 0x7f3aa40ce960 con 0x7f3a8406bbd0 2024-03-18T16:22:23.210 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:23.212 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a8406bbd0 msgr2=0x7f3a8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a8406bbd0 0x7f3a8406e090 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3aa40c5310 tx=0x7f3a90009040 comp rx=0 tx=0).stop 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa40c9be0 msgr2=0x7f3aa41f17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa40c9be0 0x7f3aa41f17a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3a9402b9c0 tx=0x7f3a940046c0 comp rx=0 tx=0).stop 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 shutdown_connections 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a8406bbd0 0x7f3a8406e090 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3aa415ae70 0x7f3aa4153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3aa40cc4f0 0x7f3aa41f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 --2- 172.21.15.102:0/1170146104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa40c9be0 0x7f3aa41f17a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:23.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 >> 172.21.15.102:0/1170146104 conn(0x7f3aa40673a0 msgr2=0x7f3aa40cf140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:23.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 shutdown_connections 2024-03-18T16:22:23.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.209+0000 7f3aa9c00640 1 -- 172.21.15.102:0/1170146104 wait complete. 2024-03-18T16:22:23.226 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-03-18T16:22:22.826745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":96,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":362,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":294,"ondisk_log_size":294,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218896,"kb_used_data":2768,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755824,"statfs":{"total":773094113280,"available":772869963776,"internally_reserved":0,"allocated":2834432,"data_stored":2235488,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4218,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002986"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":43,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.785193+0000","last_change":"2024-03-18T16:21:21.402212+0000","last_active":"2024-03-18T16:22:18.785193+0000","last_peered":"2024-03-18T16:22:18.785193+0000","last_clean":"2024-03-18T16:22:18.785193+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:22:18.785193+0000","last_undegraded":"2024-03-18T16:22:18.785193+0000","last_fullsized":"2024-03-18T16:22:18.785193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:33.215915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841700+0000","last_change":"2024-03-18T16:21:23.045605+0000","last_active":"2024-03-18T16:21:54.841700+0000","last_peered":"2024-03-18T16:21:54.841700+0000","last_clean":"2024-03-18T16:21:54.841700+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:21:54.841700+0000","last_undegraded":"2024-03-18T16:21:54.841700+0000","last_fullsized":"2024-03-18T16:21:54.841700+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.839646+0000","last_change":"2024-03-18T16:21:29.146958+0000","last_active":"2024-03-18T16:21:54.839646+0000","last_peered":"2024-03-18T16:21:54.839646+0000","last_clean":"2024-03-18T16:21:54.839646+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:21:54.839646+0000","last_undegraded":"2024-03-18T16:21:54.839646+0000","last_fullsized":"2024-03-18T16:21:54.839646+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:55.871927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842657+0000","last_change":"2024-03-18T16:21:21.413060+0000","last_active":"2024-03-18T16:21:54.842657+0000","last_peered":"2024-03-18T16:21:54.842657+0000","last_clean":"2024-03-18T16:21:54.842657+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:21:54.842657+0000","last_undegraded":"2024-03-18T16:21:54.842657+0000","last_fullsized":"2024-03-18T16:21:54.842657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:40.114937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278167+0000","last_change":"2024-03-18T16:21:23.050849+0000","last_active":"2024-03-18T16:21:55.278167+0000","last_peered":"2024-03-18T16:21:55.278167+0000","last_clean":"2024-03-18T16:21:55.278167+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:21:55.278167+0000","last_undegraded":"2024-03-18T16:21:55.278167+0000","last_fullsized":"2024-03-18T16:21:55.278167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147644+0000","last_change":"2024-03-18T16:21:29.065755+0000","last_active":"2024-03-18T16:21:55.147644+0000","last_peered":"2024-03-18T16:21:55.147644+0000","last_clean":"2024-03-18T16:21:55.147644+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:21:55.147644+0000","last_undegraded":"2024-03-18T16:21:55.147644+0000","last_fullsized":"2024-03-18T16:21:55.147644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:45.934586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'11","reported_seq":41,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.778058+0000","last_change":"2024-03-18T16:21:21.398188+0000","last_active":"2024-03-18T16:22:18.778058+0000","last_peered":"2024-03-18T16:22:18.778058+0000","last_clean":"2024-03-18T16:22:18.778058+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:22:18.778058+0000","last_undegraded":"2024-03-18T16:22:18.778058+0000","last_fullsized":"2024-03-18T16:22:18.778058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064521+0000","last_change":"2024-03-18T16:21:22.946288+0000","last_active":"2024-03-18T16:21:55.064521+0000","last_peered":"2024-03-18T16:21:55.064521+0000","last_clean":"2024-03-18T16:21:55.064521+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:21:55.064521+0000","last_undegraded":"2024-03-18T16:21:55.064521+0000","last_fullsized":"2024-03-18T16:21:55.064521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:08.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147915+0000","last_change":"2024-03-18T16:21:29.065891+0000","last_active":"2024-03-18T16:21:55.147915+0000","last_peered":"2024-03-18T16:21:55.147915+0000","last_clean":"2024-03-18T16:21:55.147915+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:21:55.147915+0000","last_undegraded":"2024-03-18T16:21:55.147915+0000","last_fullsized":"2024-03-18T16:21:55.147915+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.779058+0000","last_change":"2024-03-18T16:21:21.397071+0000","last_active":"2024-03-18T16:22:18.779058+0000","last_peered":"2024-03-18T16:22:18.779058+0000","last_clean":"2024-03-18T16:22:18.779058+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:22:18.779058+0000","last_undegraded":"2024-03-18T16:22:18.779058+0000","last_fullsized":"2024-03-18T16:22:18.779058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065035+0000","last_change":"2024-03-18T16:21:22.946468+0000","last_active":"2024-03-18T16:21:55.065035+0000","last_peered":"2024-03-18T16:21:55.065035+0000","last_clean":"2024-03-18T16:21:55.065035+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:21:55.065035+0000","last_undegraded":"2024-03-18T16:21:55.065035+0000","last_fullsized":"2024-03-18T16:21:55.065035+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147794+0000","last_change":"2024-03-18T16:21:29.073724+0000","last_active":"2024-03-18T16:21:55.147794+0000","last_peered":"2024-03-18T16:21:55.147794+0000","last_clean":"2024-03-18T16:21:55.147794+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:21:55.147794+0000","last_undegraded":"2024-03-18T16:21:55.147794+0000","last_fullsized":"2024-03-18T16:21:55.147794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273576+0000","last_change":"2024-03-18T16:21:23.041926+0000","last_active":"2024-03-18T16:21:55.273576+0000","last_peered":"2024-03-18T16:21:55.273576+0000","last_clean":"2024-03-18T16:21:55.273576+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:21:55.273576+0000","last_undegraded":"2024-03-18T16:21:55.273576+0000","last_fullsized":"2024-03-18T16:21:55.273576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:03.532253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.821045+0000","last_change":"2024-03-18T16:21:21.394801+0000","last_active":"2024-03-18T16:22:05.821045+0000","last_peered":"2024-03-18T16:22:05.821045+0000","last_clean":"2024-03-18T16:22:05.821045+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:22:05.821045+0000","last_undegraded":"2024-03-18T16:22:05.821045+0000","last_fullsized":"2024-03-18T16:22:05.821045+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148181+0000","last_change":"2024-03-18T16:21:29.073757+0000","last_active":"2024-03-18T16:21:55.148181+0000","last_peered":"2024-03-18T16:21:55.148181+0000","last_clean":"2024-03-18T16:21:55.148181+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:21:55.148181+0000","last_undegraded":"2024-03-18T16:21:55.148181+0000","last_fullsized":"2024-03-18T16:21:55.148181+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841346+0000","last_change":"2024-03-18T16:21:23.060776+0000","last_active":"2024-03-18T16:21:54.841346+0000","last_peered":"2024-03-18T16:21:54.841346+0000","last_clean":"2024-03-18T16:21:54.841346+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:21:54.841346+0000","last_undegraded":"2024-03-18T16:21:54.841346+0000","last_fullsized":"2024-03-18T16:21:54.841346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831874+0000","last_change":"2024-03-18T16:21:21.412733+0000","last_active":"2024-03-18T16:21:54.831874+0000","last_peered":"2024-03-18T16:21:54.831874+0000","last_clean":"2024-03-18T16:21:54.831874+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:21:54.831874+0000","last_undegraded":"2024-03-18T16:21:54.831874+0000","last_fullsized":"2024-03-18T16:21:54.831874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148007+0000","last_change":"2024-03-18T16:21:29.063980+0000","last_active":"2024-03-18T16:21:55.148007+0000","last_peered":"2024-03-18T16:21:55.148007+0000","last_clean":"2024-03-18T16:21:55.148007+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:21:55.148007+0000","last_undegraded":"2024-03-18T16:21:55.148007+0000","last_fullsized":"2024-03-18T16:21:55.148007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:39.469490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.845055+0000","last_change":"2024-03-18T16:21:54.838108+0000","last_active":"2024-03-18T16:21:54.845055+0000","last_peered":"2024-03-18T16:21:54.845055+0000","last_clean":"2024-03-18T16:21:54.845055+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:21:54.845055+0000","last_undegraded":"2024-03-18T16:21:54.845055+0000","last_fullsized":"2024-03-18T16:21:54.845055+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:40.182971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.418782+0000","last_change":"2024-03-18T16:21:21.407132+0000","last_active":"2024-03-18T16:22:15.418782+0000","last_peered":"2024-03-18T16:22:15.418782+0000","last_clean":"2024-03-18T16:22:15.418782+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:22:15.418782+0000","last_undegraded":"2024-03-18T16:22:15.418782+0000","last_fullsized":"2024-03-18T16:22:15.418782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.272088+0000","last_change":"2024-03-18T16:21:55.269661+0000","last_active":"2024-03-18T16:21:55.272088+0000","last_peered":"2024-03-18T16:21:55.272088+0000","last_clean":"2024-03-18T16:21:55.272088+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:21:55.272088+0000","last_undegraded":"2024-03-18T16:21:55.272088+0000","last_fullsized":"2024-03-18T16:21:55.272088+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:38.290937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148142+0000","last_change":"2024-03-18T16:21:22.840551+0000","last_active":"2024-03-18T16:21:55.148142+0000","last_peered":"2024-03-18T16:21:55.148142+0000","last_clean":"2024-03-18T16:21:55.148142+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:21:55.148142+0000","last_undegraded":"2024-03-18T16:21:55.148142+0000","last_fullsized":"2024-03-18T16:21:55.148142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:17.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.822682+0000","last_change":"2024-03-18T16:21:21.400349+0000","last_active":"2024-03-18T16:22:05.822682+0000","last_peered":"2024-03-18T16:22:05.822682+0000","last_clean":"2024-03-18T16:22:05.822682+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:22:05.822682+0000","last_undegraded":"2024-03-18T16:22:05.822682+0000","last_fullsized":"2024-03-18T16:22:05.822682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148262+0000","last_change":"2024-03-18T16:21:29.064624+0000","last_active":"2024-03-18T16:21:55.148262+0000","last_peered":"2024-03-18T16:21:55.148262+0000","last_clean":"2024-03-18T16:21:55.148262+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:21:55.148262+0000","last_undegraded":"2024-03-18T16:21:55.148262+0000","last_fullsized":"2024-03-18T16:21:55.148262+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278343+0000","last_change":"2024-03-18T16:21:23.055982+0000","last_active":"2024-03-18T16:21:55.278343+0000","last_peered":"2024-03-18T16:21:55.278343+0000","last_clean":"2024-03-18T16:21:55.278343+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:21:55.278343+0000","last_undegraded":"2024-03-18T16:21:55.278343+0000","last_fullsized":"2024-03-18T16:21:55.278343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.781767+0000","last_change":"2024-03-18T16:21:21.400843+0000","last_active":"2024-03-18T16:22:18.781767+0000","last_peered":"2024-03-18T16:22:18.781767+0000","last_clean":"2024-03-18T16:22:18.781767+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:22:18.781767+0000","last_undegraded":"2024-03-18T16:22:18.781767+0000","last_fullsized":"2024-03-18T16:22:18.781767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147884+0000","last_change":"2024-03-18T16:21:29.140888+0000","last_active":"2024-03-18T16:21:55.147884+0000","last_peered":"2024-03-18T16:21:55.147884+0000","last_clean":"2024-03-18T16:21:55.147884+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:21:55.147884+0000","last_undegraded":"2024-03-18T16:21:55.147884+0000","last_fullsized":"2024-03-18T16:21:55.147884+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842117+0000","last_change":"2024-03-18T16:21:22.994455+0000","last_active":"2024-03-18T16:21:54.842117+0000","last_peered":"2024-03-18T16:21:54.842117+0000","last_clean":"2024-03-18T16:21:54.842117+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:21:54.842117+0000","last_undegraded":"2024-03-18T16:21:54.842117+0000","last_fullsized":"2024-03-18T16:21:54.842117+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.421651+0000","last_change":"2024-03-18T16:21:21.402920+0000","last_active":"2024-03-18T16:22:15.421651+0000","last_peered":"2024-03-18T16:22:15.421651+0000","last_clean":"2024-03-18T16:22:15.421651+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:22:15.421651+0000","last_undegraded":"2024-03-18T16:22:15.421651+0000","last_fullsized":"2024-03-18T16:22:15.421651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.838803+0000","last_change":"2024-03-18T16:21:29.145235+0000","last_active":"2024-03-18T16:21:54.838803+0000","last_peered":"2024-03-18T16:21:54.838803+0000","last_clean":"2024-03-18T16:21:54.838803+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:21:54.838803+0000","last_undegraded":"2024-03-18T16:21:54.838803+0000","last_fullsized":"2024-03-18T16:21:54.838803+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:42.379951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.833875+0000","last_change":"2024-03-18T16:21:22.975172+0000","last_active":"2024-03-18T16:21:54.833875+0000","last_peered":"2024-03-18T16:21:54.833875+0000","last_clean":"2024-03-18T16:21:54.833875+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:21:54.833875+0000","last_undegraded":"2024-03-18T16:21:54.833875+0000","last_fullsized":"2024-03-18T16:21:54.833875+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.422730+0000","last_change":"2024-03-18T16:21:21.399688+0000","last_active":"2024-03-18T16:22:15.422730+0000","last_peered":"2024-03-18T16:22:15.422730+0000","last_clean":"2024-03-18T16:22:15.422730+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:22:15.422730+0000","last_undegraded":"2024-03-18T16:22:15.422730+0000","last_fullsized":"2024-03-18T16:22:15.422730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:11.933810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277363+0000","last_change":"2024-03-18T16:21:29.141555+0000","last_active":"2024-03-18T16:21:55.277363+0000","last_peered":"2024-03-18T16:21:55.277363+0000","last_clean":"2024-03-18T16:21:55.277363+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:21:55.277363+0000","last_undegraded":"2024-03-18T16:21:55.277363+0000","last_fullsized":"2024-03-18T16:21:55.277363+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065093+0000","last_change":"2024-03-18T16:21:22.947980+0000","last_active":"2024-03-18T16:21:55.065093+0000","last_peered":"2024-03-18T16:21:55.065093+0000","last_clean":"2024-03-18T16:21:55.065093+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:21:55.065093+0000","last_undegraded":"2024-03-18T16:21:55.065093+0000","last_fullsized":"2024-03-18T16:21:55.065093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:31.739693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"29'1","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834250+0000","last_change":"2024-03-18T16:21:21.400087+0000","last_active":"2024-03-18T16:21:54.834250+0000","last_peered":"2024-03-18T16:21:54.834250+0000","last_clean":"2024-03-18T16:21:54.834250+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:21:54.834250+0000","last_undegraded":"2024-03-18T16:21:54.834250+0000","last_fullsized":"2024-03-18T16:21:54.834250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.830782+0000","last_change":"2024-03-18T16:21:29.076629+0000","last_active":"2024-03-18T16:21:54.830782+0000","last_peered":"2024-03-18T16:21:54.830782+0000","last_clean":"2024-03-18T16:21:54.830782+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:21:54.830782+0000","last_undegraded":"2024-03-18T16:21:54.830782+0000","last_fullsized":"2024-03-18T16:21:54.830782+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148132+0000","last_change":"2024-03-18T16:21:23.055810+0000","last_active":"2024-03-18T16:21:55.148132+0000","last_peered":"2024-03-18T16:21:55.148132+0000","last_clean":"2024-03-18T16:21:55.148132+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:21:55.148132+0000","last_undegraded":"2024-03-18T16:21:55.148132+0000","last_fullsized":"2024-03-18T16:21:55.148132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:40.451255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.866023+0000","last_change":"2024-03-18T16:21:21.399134+0000","last_active":"2024-03-18T16:22:18.866023+0000","last_peered":"2024-03-18T16:22:18.866023+0000","last_clean":"2024-03-18T16:22:18.866023+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:22:18.866023+0000","last_undegraded":"2024-03-18T16:22:18.866023+0000","last_fullsized":"2024-03-18T16:22:18.866023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834980+0000","last_change":"2024-03-18T16:21:29.067567+0000","last_active":"2024-03-18T16:21:54.834980+0000","last_peered":"2024-03-18T16:21:54.834980+0000","last_clean":"2024-03-18T16:21:54.834980+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:21:54.834980+0000","last_undegraded":"2024-03-18T16:21:54.834980+0000","last_fullsized":"2024-03-18T16:21:54.834980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.832134+0000","last_change":"2024-03-18T16:21:22.977089+0000","last_active":"2024-03-18T16:21:54.832134+0000","last_peered":"2024-03-18T16:21:54.832134+0000","last_clean":"2024-03-18T16:21:54.832134+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:21:54.832134+0000","last_undegraded":"2024-03-18T16:21:54.832134+0000","last_fullsized":"2024-03-18T16:21:54.832134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:12.723995+0000","last_change":"2024-03-18T16:21:21.408450+0000","last_active":"2024-03-18T16:22:12.723995+0000","last_peered":"2024-03-18T16:22:12.723995+0000","last_clean":"2024-03-18T16:22:12.723995+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:22:12.723995+0000","last_undegraded":"2024-03-18T16:22:12.723995+0000","last_fullsized":"2024-03-18T16:22:12.723995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.835511+0000","last_change":"2024-03-18T16:21:29.076875+0000","last_active":"2024-03-18T16:21:54.835511+0000","last_peered":"2024-03-18T16:21:54.835511+0000","last_clean":"2024-03-18T16:21:54.835511+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:21:54.835511+0000","last_undegraded":"2024-03-18T16:21:54.835511+0000","last_fullsized":"2024-03-18T16:21:54.835511+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277013+0000","last_change":"2024-03-18T16:21:23.054497+0000","last_active":"2024-03-18T16:21:55.277013+0000","last_peered":"2024-03-18T16:21:55.277013+0000","last_clean":"2024-03-18T16:21:55.277013+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:21:55.277013+0000","last_undegraded":"2024-03-18T16:21:55.277013+0000","last_fullsized":"2024-03-18T16:21:55.277013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:37.455180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:12.603105+0000","last_change":"2024-03-18T16:21:21.404754+0000","last_active":"2024-03-18T16:22:12.603105+0000","last_peered":"2024-03-18T16:22:12.603105+0000","last_clean":"2024-03-18T16:22:12.603105+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:22:12.603105+0000","last_undegraded":"2024-03-18T16:22:12.603105+0000","last_fullsized":"2024-03-18T16:22:12.603105+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278071+0000","last_change":"2024-03-18T16:21:29.147316+0000","last_active":"2024-03-18T16:21:55.278071+0000","last_peered":"2024-03-18T16:21:55.278071+0000","last_clean":"2024-03-18T16:21:55.278071+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:21:55.278071+0000","last_undegraded":"2024-03-18T16:21:55.278071+0000","last_fullsized":"2024-03-18T16:21:55.278071+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.842010+0000","last_change":"2024-03-18T16:21:22.947635+0000","last_active":"2024-03-18T16:21:54.842010+0000","last_peered":"2024-03-18T16:21:54.842010+0000","last_clean":"2024-03-18T16:21:54.842010+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:21:54.842010+0000","last_undegraded":"2024-03-18T16:21:54.842010+0000","last_fullsized":"2024-03-18T16:21:54.842010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.422726+0000","last_change":"2024-03-18T16:21:21.381306+0000","last_active":"2024-03-18T16:22:15.422726+0000","last_peered":"2024-03-18T16:22:15.422726+0000","last_clean":"2024-03-18T16:22:15.422726+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:22:15.422726+0000","last_undegraded":"2024-03-18T16:22:15.422726+0000","last_fullsized":"2024-03-18T16:22:15.422726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":186,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265740+0000","last_change":"2024-03-18T16:21:08.662767+0000","last_active":"2024-03-18T16:21:55.265740+0000","last_peered":"2024-03-18T16:21:55.265740+0000","last_clean":"2024-03-18T16:21:55.265740+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:21:55.265740+0000","last_undegraded":"2024-03-18T16:21:55.265740+0000","last_fullsized":"2024-03-18T16:21:55.265740+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_clean_scrub_stamp":"2024-03-18T16:21:05.847994+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:12:10.494707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273700+0000","last_change":"2024-03-18T16:21:29.145697+0000","last_active":"2024-03-18T16:21:55.273700+0000","last_peered":"2024-03-18T16:21:55.273700+0000","last_clean":"2024-03-18T16:21:55.273700+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:21:55.273700+0000","last_undegraded":"2024-03-18T16:21:55.273700+0000","last_fullsized":"2024-03-18T16:21:55.273700+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:46.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266203+0000","last_change":"2024-03-18T16:21:22.995291+0000","last_active":"2024-03-18T16:21:55.266203+0000","last_peered":"2024-03-18T16:21:55.266203+0000","last_clean":"2024-03-18T16:21:55.266203+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:21:55.266203+0000","last_undegraded":"2024-03-18T16:21:55.266203+0000","last_fullsized":"2024-03-18T16:21:55.266203+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:32.918246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.446285+0000","last_change":"2024-03-18T16:21:21.382137+0000","last_active":"2024-03-18T16:22:15.446285+0000","last_peered":"2024-03-18T16:22:15.446285+0000","last_clean":"2024-03-18T16:22:15.446285+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:22:15.446285+0000","last_undegraded":"2024-03-18T16:22:15.446285+0000","last_fullsized":"2024-03-18T16:22:15.446285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273492+0000","last_change":"2024-03-18T16:21:29.139432+0000","last_active":"2024-03-18T16:21:55.273492+0000","last_peered":"2024-03-18T16:21:55.273492+0000","last_clean":"2024-03-18T16:21:55.273492+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:21:55.273492+0000","last_undegraded":"2024-03-18T16:21:55.273492+0000","last_fullsized":"2024-03-18T16:21:55.273492+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:14.828125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064603+0000","last_change":"2024-03-18T16:21:22.949578+0000","last_active":"2024-03-18T16:21:55.064603+0000","last_peered":"2024-03-18T16:21:55.064603+0000","last_clean":"2024-03-18T16:21:55.064603+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:21:55.064603+0000","last_undegraded":"2024-03-18T16:21:55.064603+0000","last_fullsized":"2024-03-18T16:21:55.064603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:36:20.997720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"29'2","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.835288+0000","last_change":"2024-03-18T16:21:21.400859+0000","last_active":"2024-03-18T16:21:54.835288+0000","last_peered":"2024-03-18T16:21:54.835288+0000","last_clean":"2024-03-18T16:21:54.835288+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:21:54.835288+0000","last_undegraded":"2024-03-18T16:21:54.835288+0000","last_fullsized":"2024-03-18T16:21:54.835288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277608+0000","last_change":"2024-03-18T16:21:29.076813+0000","last_active":"2024-03-18T16:21:55.277608+0000","last_peered":"2024-03-18T16:21:55.277608+0000","last_clean":"2024-03-18T16:21:55.277608+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:21:55.277608+0000","last_undegraded":"2024-03-18T16:21:55.277608+0000","last_fullsized":"2024-03-18T16:21:55.277608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266390+0000","last_change":"2024-03-18T16:21:23.000009+0000","last_active":"2024-03-18T16:21:55.266390+0000","last_peered":"2024-03-18T16:21:55.266390+0000","last_clean":"2024-03-18T16:21:55.266390+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:21:55.266390+0000","last_undegraded":"2024-03-18T16:21:55.266390+0000","last_fullsized":"2024-03-18T16:21:55.266390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:39.383342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.888727+0000","last_change":"2024-03-18T16:21:21.409271+0000","last_active":"2024-03-18T16:22:18.888727+0000","last_peered":"2024-03-18T16:22:18.888727+0000","last_clean":"2024-03-18T16:22:18.888727+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:22:18.888727+0000","last_undegraded":"2024-03-18T16:22:18.888727+0000","last_fullsized":"2024-03-18T16:22:18.888727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064256+0000","last_change":"2024-03-18T16:21:29.139595+0000","last_active":"2024-03-18T16:21:55.064256+0000","last_peered":"2024-03-18T16:21:55.064256+0000","last_clean":"2024-03-18T16:21:55.064256+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:21:55.064256+0000","last_undegraded":"2024-03-18T16:21:55.064256+0000","last_fullsized":"2024-03-18T16:21:55.064256+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:49.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266190+0000","last_change":"2024-03-18T16:21:23.000054+0000","last_active":"2024-03-18T16:21:55.266190+0000","last_peered":"2024-03-18T16:21:55.266190+0000","last_clean":"2024-03-18T16:21:55.266190+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:21:55.266190+0000","last_undegraded":"2024-03-18T16:21:55.266190+0000","last_fullsized":"2024-03-18T16:21:55.266190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:34:41.535042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"29'2","reported_seq":32,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148537+0000","last_change":"2024-03-18T16:21:21.399315+0000","last_active":"2024-03-18T16:21:55.148537+0000","last_peered":"2024-03-18T16:21:55.148537+0000","last_clean":"2024-03-18T16:21:55.148537+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:21:55.148537+0000","last_undegraded":"2024-03-18T16:21:55.148537+0000","last_fullsized":"2024-03-18T16:21:55.148537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:29.543106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147629+0000","last_change":"2024-03-18T16:21:29.065243+0000","last_active":"2024-03-18T16:21:55.147629+0000","last_peered":"2024-03-18T16:21:55.147629+0000","last_clean":"2024-03-18T16:21:55.147629+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:21:55.147629+0000","last_undegraded":"2024-03-18T16:21:55.147629+0000","last_fullsized":"2024-03-18T16:21:55.147629+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265799+0000","last_change":"2024-03-18T16:21:22.996080+0000","last_active":"2024-03-18T16:21:55.265799+0000","last_peered":"2024-03-18T16:21:55.265799+0000","last_clean":"2024-03-18T16:21:55.265799+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:21:55.265799+0000","last_undegraded":"2024-03-18T16:21:55.265799+0000","last_fullsized":"2024-03-18T16:21:55.265799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"29'17","reported_seq":47,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.463973+0000","last_change":"2024-03-18T16:21:21.404930+0000","last_active":"2024-03-18T16:22:15.463973+0000","last_peered":"2024-03-18T16:22:15.463973+0000","last_clean":"2024-03-18T16:22:15.463973+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:22:15.463973+0000","last_undegraded":"2024-03-18T16:22:15.463973+0000","last_fullsized":"2024-03-18T16:22:15.463973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:02.571905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278055+0000","last_change":"2024-03-18T16:21:29.079031+0000","last_active":"2024-03-18T16:21:55.278055+0000","last_peered":"2024-03-18T16:21:55.278055+0000","last_clean":"2024-03-18T16:21:55.278055+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:21:55.278055+0000","last_undegraded":"2024-03-18T16:21:55.278055+0000","last_fullsized":"2024-03-18T16:21:55.278055+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273519+0000","last_change":"2024-03-18T16:21:23.056832+0000","last_active":"2024-03-18T16:21:55.273519+0000","last_peered":"2024-03-18T16:21:55.273519+0000","last_clean":"2024-03-18T16:21:55.273519+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:21:55.273519+0000","last_undegraded":"2024-03-18T16:21:55.273519+0000","last_fullsized":"2024-03-18T16:21:55.273519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:56.124747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.396156+0000","last_change":"2024-03-18T16:21:21.404392+0000","last_active":"2024-03-18T16:22:15.396156+0000","last_peered":"2024-03-18T16:22:15.396156+0000","last_clean":"2024-03-18T16:22:15.396156+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:22:15.396156+0000","last_undegraded":"2024-03-18T16:22:15.396156+0000","last_fullsized":"2024-03-18T16:22:15.396156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841565+0000","last_change":"2024-03-18T16:21:29.139956+0000","last_active":"2024-03-18T16:21:54.841565+0000","last_peered":"2024-03-18T16:21:54.841565+0000","last_clean":"2024-03-18T16:21:54.841565+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:21:54.841565+0000","last_undegraded":"2024-03-18T16:21:54.841565+0000","last_fullsized":"2024-03-18T16:21:54.841565+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:52.356673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265953+0000","last_change":"2024-03-18T16:21:22.995275+0000","last_active":"2024-03-18T16:21:55.265953+0000","last_peered":"2024-03-18T16:21:55.265953+0000","last_clean":"2024-03-18T16:21:55.265953+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:21:55.265953+0000","last_undegraded":"2024-03-18T16:21:55.265953+0000","last_fullsized":"2024-03-18T16:21:55.265953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:48.024985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.429963+0000","last_change":"2024-03-18T16:21:21.412044+0000","last_active":"2024-03-18T16:22:15.429963+0000","last_peered":"2024-03-18T16:22:15.429963+0000","last_clean":"2024-03-18T16:22:15.429963+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:22:15.429963+0000","last_undegraded":"2024-03-18T16:22:15.429963+0000","last_fullsized":"2024-03-18T16:22:15.429963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.270056+0000","last_change":"2024-03-18T16:21:55.270056+0000","last_active":"2024-03-18T16:21:55.270056+0000","last_peered":"2024-03-18T16:21:55.270056+0000","last_clean":"2024-03-18T16:21:55.270056+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:21:55.270056+0000","last_undegraded":"2024-03-18T16:21:55.270056+0000","last_fullsized":"2024-03-18T16:21:55.270056+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:50.466826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277605+0000","last_change":"2024-03-18T16:21:23.056116+0000","last_active":"2024-03-18T16:21:55.277605+0000","last_peered":"2024-03-18T16:21:55.277605+0000","last_clean":"2024-03-18T16:21:55.277605+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:21:55.277605+0000","last_undegraded":"2024-03-18T16:21:55.277605+0000","last_fullsized":"2024-03-18T16:21:55.277605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:18.455315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.784136+0000","last_change":"2024-03-18T16:21:21.400378+0000","last_active":"2024-03-18T16:22:18.784136+0000","last_peered":"2024-03-18T16:22:18.784136+0000","last_clean":"2024-03-18T16:22:18.784136+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:22:18.784136+0000","last_undegraded":"2024-03-18T16:22:18.784136+0000","last_fullsized":"2024-03-18T16:22:18.784136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064943+0000","last_change":"2024-03-18T16:21:29.141554+0000","last_active":"2024-03-18T16:21:55.064943+0000","last_peered":"2024-03-18T16:21:55.064943+0000","last_clean":"2024-03-18T16:21:55.064943+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:21:55.064943+0000","last_undegraded":"2024-03-18T16:21:55.064943+0000","last_fullsized":"2024-03-18T16:21:55.064943+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:25.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266115+0000","last_change":"2024-03-18T16:21:22.996694+0000","last_active":"2024-03-18T16:21:55.266115+0000","last_peered":"2024-03-18T16:21:55.266115+0000","last_clean":"2024-03-18T16:21:55.266115+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:21:55.266115+0000","last_undegraded":"2024-03-18T16:21:55.266115+0000","last_fullsized":"2024-03-18T16:21:55.266115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:03.704090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":44,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.784931+0000","last_change":"2024-03-18T16:21:21.412149+0000","last_active":"2024-03-18T16:22:18.784931+0000","last_peered":"2024-03-18T16:22:18.784931+0000","last_clean":"2024-03-18T16:22:18.784931+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:22:18.784931+0000","last_undegraded":"2024-03-18T16:22:18.784931+0000","last_fullsized":"2024-03-18T16:22:18.784931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831282+0000","last_change":"2024-03-18T16:21:29.077350+0000","last_active":"2024-03-18T16:21:54.831282+0000","last_peered":"2024-03-18T16:21:54.831282+0000","last_clean":"2024-03-18T16:21:54.831282+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:21:54.831282+0000","last_undegraded":"2024-03-18T16:21:54.831282+0000","last_fullsized":"2024-03-18T16:21:54.831282+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148199+0000","last_change":"2024-03-18T16:21:22.992928+0000","last_active":"2024-03-18T16:21:55.148199+0000","last_peered":"2024-03-18T16:21:55.148199+0000","last_clean":"2024-03-18T16:21:55.148199+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:21:55.148199+0000","last_undegraded":"2024-03-18T16:21:55.148199+0000","last_fullsized":"2024-03-18T16:21:55.148199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:50.978267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:05.826412+0000","last_change":"2024-03-18T16:21:21.403749+0000","last_active":"2024-03-18T16:22:05.826412+0000","last_peered":"2024-03-18T16:22:05.826412+0000","last_clean":"2024-03-18T16:22:05.826412+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:22:05.826412+0000","last_undegraded":"2024-03-18T16:22:05.826412+0000","last_fullsized":"2024-03-18T16:22:05.826412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:29:30.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.147816+0000","last_change":"2024-03-18T16:21:29.065378+0000","last_active":"2024-03-18T16:21:55.147816+0000","last_peered":"2024-03-18T16:21:55.147816+0000","last_clean":"2024-03-18T16:21:55.147816+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:21:55.147816+0000","last_undegraded":"2024-03-18T16:21:55.147816+0000","last_fullsized":"2024-03-18T16:21:55.147816+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:22:12.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278432+0000","last_change":"2024-03-18T16:21:23.054168+0000","last_active":"2024-03-18T16:21:55.278432+0000","last_peered":"2024-03-18T16:21:55.278432+0000","last_clean":"2024-03-18T16:21:55.278432+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:21:55.278432+0000","last_undegraded":"2024-03-18T16:21:55.278432+0000","last_fullsized":"2024-03-18T16:21:55.278432+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:18.861925+0000","last_change":"2024-03-18T16:21:21.398307+0000","last_active":"2024-03-18T16:22:18.861925+0000","last_peered":"2024-03-18T16:22:18.861925+0000","last_clean":"2024-03-18T16:22:18.861925+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:22:18.861925+0000","last_undegraded":"2024-03-18T16:22:18.861925+0000","last_fullsized":"2024-03-18T16:22:18.861925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840228+0000","last_change":"2024-03-18T16:21:29.065855+0000","last_active":"2024-03-18T16:21:54.840228+0000","last_peered":"2024-03-18T16:21:54.840228+0000","last_clean":"2024-03-18T16:21:54.840228+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:21:54.840228+0000","last_undegraded":"2024-03-18T16:21:54.840228+0000","last_fullsized":"2024-03-18T16:21:54.840228+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:43.207302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064726+0000","last_change":"2024-03-18T16:21:22.993606+0000","last_active":"2024-03-18T16:21:55.064726+0000","last_peered":"2024-03-18T16:21:55.064726+0000","last_clean":"2024-03-18T16:21:55.064726+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:21:55.064726+0000","last_undegraded":"2024-03-18T16:21:55.064726+0000","last_fullsized":"2024-03-18T16:21:55.064726+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:38:23.121609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.041784+0000","last_change":"2024-03-18T16:21:21.399588+0000","last_active":"2024-03-18T16:22:06.041784+0000","last_peered":"2024-03-18T16:22:06.041784+0000","last_clean":"2024-03-18T16:22:06.041784+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:22:06.041784+0000","last_undegraded":"2024-03-18T16:22:06.041784+0000","last_fullsized":"2024-03-18T16:22:06.041784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.843888+0000","last_change":"2024-03-18T16:21:29.144832+0000","last_active":"2024-03-18T16:21:54.843888+0000","last_peered":"2024-03-18T16:21:54.843888+0000","last_clean":"2024-03-18T16:21:54.843888+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:21:54.843888+0000","last_undegraded":"2024-03-18T16:21:54.843888+0000","last_fullsized":"2024-03-18T16:21:54.843888+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:10.103662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277724+0000","last_change":"2024-03-18T16:21:29.148215+0000","last_active":"2024-03-18T16:21:55.277724+0000","last_peered":"2024-03-18T16:21:55.277724+0000","last_clean":"2024-03-18T16:21:55.277724+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:21:55.277724+0000","last_undegraded":"2024-03-18T16:21:55.277724+0000","last_fullsized":"2024-03-18T16:21:55.277724+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.829181+0000","last_change":"2024-03-18T16:21:22.980300+0000","last_active":"2024-03-18T16:21:54.829181+0000","last_peered":"2024-03-18T16:21:54.829181+0000","last_clean":"2024-03-18T16:21:54.829181+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:21:54.829181+0000","last_undegraded":"2024-03-18T16:21:54.829181+0000","last_fullsized":"2024-03-18T16:21:54.829181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.423345+0000","last_change":"2024-03-18T16:21:21.329294+0000","last_active":"2024-03-18T16:22:15.423345+0000","last_peered":"2024-03-18T16:22:15.423345+0000","last_clean":"2024-03-18T16:22:15.423345+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:22:15.423345+0000","last_undegraded":"2024-03-18T16:22:15.423345+0000","last_fullsized":"2024-03-18T16:22:15.423345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273909+0000","last_change":"2024-03-18T16:21:29.076706+0000","last_active":"2024-03-18T16:21:55.273909+0000","last_peered":"2024-03-18T16:21:55.273909+0000","last_clean":"2024-03-18T16:21:55.273909+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:21:55.273909+0000","last_undegraded":"2024-03-18T16:21:55.273909+0000","last_fullsized":"2024-03-18T16:21:55.273909+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:57.880669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840470+0000","last_change":"2024-03-18T16:21:23.052989+0000","last_active":"2024-03-18T16:21:54.840470+0000","last_peered":"2024-03-18T16:21:54.840470+0000","last_clean":"2024-03-18T16:21:54.840470+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:21:54.840470+0000","last_undegraded":"2024-03-18T16:21:54.840470+0000","last_fullsized":"2024-03-18T16:21:54.840470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:12.607471+0000","last_change":"2024-03-18T16:21:21.413020+0000","last_active":"2024-03-18T16:22:12.607471+0000","last_peered":"2024-03-18T16:22:12.607471+0000","last_clean":"2024-03-18T16:22:12.607471+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:22:12.607471+0000","last_undegraded":"2024-03-18T16:22:12.607471+0000","last_fullsized":"2024-03-18T16:22:12.607471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:49.237529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":37,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:15.400410+0000","last_change":"2024-03-18T16:21:21.397354+0000","last_active":"2024-03-18T16:22:15.400410+0000","last_peered":"2024-03-18T16:22:15.400410+0000","last_clean":"2024-03-18T16:22:15.400410+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:22:15.400410+0000","last_undegraded":"2024-03-18T16:22:15.400410+0000","last_fullsized":"2024-03-18T16:22:15.400410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.278026+0000","last_change":"2024-03-18T16:21:23.059441+0000","last_active":"2024-03-18T16:21:55.278026+0000","last_peered":"2024-03-18T16:21:55.278026+0000","last_clean":"2024-03-18T16:21:55.278026+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:21:55.278026+0000","last_undegraded":"2024-03-18T16:21:55.278026+0000","last_fullsized":"2024-03-18T16:21:55.278026+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827518+0000","last_change":"2024-03-18T16:21:29.058800+0000","last_active":"2024-03-18T16:21:54.827518+0000","last_peered":"2024-03-18T16:21:54.827518+0000","last_clean":"2024-03-18T16:21:54.827518+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:21:54.827518+0000","last_undegraded":"2024-03-18T16:21:54.827518+0000","last_fullsized":"2024-03-18T16:21:54.827518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:20.279579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:22:06.074918+0000","last_change":"2024-03-18T16:21:21.412536+0000","last_active":"2024-03-18T16:22:06.074918+0000","last_peered":"2024-03-18T16:22:06.074918+0000","last_clean":"2024-03-18T16:22:06.074918+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:22:06.074918+0000","last_undegraded":"2024-03-18T16:22:06.074918+0000","last_fullsized":"2024-03-18T16:22:06.074918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148039+0000","last_change":"2024-03-18T16:21:22.974963+0000","last_active":"2024-03-18T16:21:55.148039+0000","last_peered":"2024-03-18T16:21:55.148039+0000","last_clean":"2024-03-18T16:21:55.148039+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:21:55.148039+0000","last_undegraded":"2024-03-18T16:21:55.148039+0000","last_fullsized":"2024-03-18T16:21:55.148039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:51.820583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.151933+0000","last_change":"2024-03-18T16:21:55.151933+0000","last_active":"2024-03-18T16:21:55.151933+0000","last_peered":"2024-03-18T16:21:55.151933+0000","last_clean":"2024-03-18T16:21:55.151933+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:21:55.151933+0000","last_undegraded":"2024-03-18T16:21:55.151933+0000","last_fullsized":"2024-03-18T16:21:55.151933+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:38.438575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273462+0000","last_change":"2024-03-18T16:21:23.056857+0000","last_active":"2024-03-18T16:21:55.273462+0000","last_peered":"2024-03-18T16:21:55.273462+0000","last_clean":"2024-03-18T16:21:55.273462+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:21:55.273462+0000","last_undegraded":"2024-03-18T16:21:55.273462+0000","last_fullsized":"2024-03-18T16:21:55.273462+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:13.728335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.827238+0000","last_change":"2024-03-18T16:21:22.949289+0000","last_active":"2024-03-18T16:21:54.827238+0000","last_peered":"2024-03-18T16:21:54.827238+0000","last_clean":"2024-03-18T16:21:54.827238+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:21:54.827238+0000","last_undegraded":"2024-03-18T16:21:54.827238+0000","last_fullsized":"2024-03-18T16:21:54.827238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:51:51.357680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.848377+0000","last_change":"2024-03-18T16:21:54.845858+0000","last_active":"2024-03-18T16:21:54.848377+0000","last_peered":"2024-03-18T16:21:54.848377+0000","last_clean":"2024-03-18T16:21:54.848377+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:21:54.848377+0000","last_undegraded":"2024-03-18T16:21:54.848377+0000","last_fullsized":"2024-03-18T16:21:54.848377+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277673+0000","last_change":"2024-03-18T16:21:23.059438+0000","last_active":"2024-03-18T16:21:55.277673+0000","last_peered":"2024-03-18T16:21:55.277673+0000","last_clean":"2024-03-18T16:21:55.277673+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:21:55.277673+0000","last_undegraded":"2024-03-18T16:21:55.277673+0000","last_fullsized":"2024-03-18T16:21:55.277673+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841882+0000","last_change":"2024-03-18T16:21:22.849115+0000","last_active":"2024-03-18T16:21:54.841882+0000","last_peered":"2024-03-18T16:21:54.841882+0000","last_clean":"2024-03-18T16:21:54.841882+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:21:54.841882+0000","last_undegraded":"2024-03-18T16:21:54.841882+0000","last_fullsized":"2024-03-18T16:21:54.841882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:57.469261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834154+0000","last_change":"2024-03-18T16:21:22.980253+0000","last_active":"2024-03-18T16:21:54.834154+0000","last_peered":"2024-03-18T16:21:54.834154+0000","last_clean":"2024-03-18T16:21:54.834154+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:21:54.834154+0000","last_undegraded":"2024-03-18T16:21:54.834154+0000","last_fullsized":"2024-03-18T16:21:54.834154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273836+0000","last_change":"2024-03-18T16:21:23.042188+0000","last_active":"2024-03-18T16:21:55.273836+0000","last_peered":"2024-03-18T16:21:55.273836+0000","last_clean":"2024-03-18T16:21:55.273836+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:21:55.273836+0000","last_undegraded":"2024-03-18T16:21:55.273836+0000","last_fullsized":"2024-03-18T16:21:55.273836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841847+0000","last_change":"2024-03-18T16:21:22.851677+0000","last_active":"2024-03-18T16:21:54.841847+0000","last_peered":"2024-03-18T16:21:54.841847+0000","last_clean":"2024-03-18T16:21:54.841847+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:21:54.841847+0000","last_undegraded":"2024-03-18T16:21:54.841847+0000","last_fullsized":"2024-03-18T16:21:54.841847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064494+0000","last_change":"2024-03-18T16:21:22.947869+0000","last_active":"2024-03-18T16:21:55.064494+0000","last_peered":"2024-03-18T16:21:55.064494+0000","last_clean":"2024-03-18T16:21:55.064494+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:21:55.064494+0000","last_undegraded":"2024-03-18T16:21:55.064494+0000","last_fullsized":"2024-03-18T16:21:55.064494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.840151+0000","last_change":"2024-03-18T16:21:22.851663+0000","last_active":"2024-03-18T16:21:54.840151+0000","last_peered":"2024-03-18T16:21:54.840151+0000","last_clean":"2024-03-18T16:21:54.840151+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:21:54.840151+0000","last_undegraded":"2024-03-18T16:21:54.840151+0000","last_fullsized":"2024-03-18T16:21:54.840151+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.271992+0000","last_change":"2024-03-18T16:21:55.268883+0000","last_active":"2024-03-18T16:21:55.271992+0000","last_peered":"2024-03-18T16:21:55.271992+0000","last_clean":"2024-03-18T16:21:55.271992+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:21:55.271992+0000","last_undegraded":"2024-03-18T16:21:55.271992+0000","last_fullsized":"2024-03-18T16:21:55.271992+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277397+0000","last_change":"2024-03-18T16:21:23.059733+0000","last_active":"2024-03-18T16:21:55.277397+0000","last_peered":"2024-03-18T16:21:55.277397+0000","last_clean":"2024-03-18T16:21:55.277397+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:21:55.277397+0000","last_undegraded":"2024-03-18T16:21:55.277397+0000","last_fullsized":"2024-03-18T16:21:55.277397+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273660+0000","last_change":"2024-03-18T16:21:23.042457+0000","last_active":"2024-03-18T16:21:55.273660+0000","last_peered":"2024-03-18T16:21:55.273660+0000","last_clean":"2024-03-18T16:21:55.273660+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:21:55.273660+0000","last_undegraded":"2024-03-18T16:21:55.273660+0000","last_fullsized":"2024-03-18T16:21:55.273660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:09.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.845542+0000","last_change":"2024-03-18T16:21:54.842423+0000","last_active":"2024-03-18T16:21:54.845542+0000","last_peered":"2024-03-18T16:21:54.845542+0000","last_clean":"2024-03-18T16:21:54.845542+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:21:54.845542+0000","last_undegraded":"2024-03-18T16:21:54.845542+0000","last_fullsized":"2024-03-18T16:21:54.845542+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:36.656058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.828079+0000","last_change":"2024-03-18T16:21:23.045061+0000","last_active":"2024-03-18T16:21:54.828079+0000","last_peered":"2024-03-18T16:21:54.828079+0000","last_clean":"2024-03-18T16:21:54.828079+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:21:54.828079+0000","last_undegraded":"2024-03-18T16:21:54.828079+0000","last_fullsized":"2024-03-18T16:21:54.828079+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:54.119675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.065127+0000","last_change":"2024-03-18T16:21:22.993632+0000","last_active":"2024-03-18T16:21:55.065127+0000","last_peered":"2024-03-18T16:21:55.065127+0000","last_clean":"2024-03-18T16:21:55.065127+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:21:55.065127+0000","last_undegraded":"2024-03-18T16:21:55.065127+0000","last_fullsized":"2024-03-18T16:21:55.065127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:35:42.249037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064410+0000","last_change":"2024-03-18T16:21:23.056575+0000","last_active":"2024-03-18T16:21:55.064410+0000","last_peered":"2024-03-18T16:21:55.064410+0000","last_clean":"2024-03-18T16:21:55.064410+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:21:55.064410+0000","last_undegraded":"2024-03-18T16:21:55.064410+0000","last_fullsized":"2024-03-18T16:21:55.064410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.831640+0000","last_change":"2024-03-18T16:21:22.973283+0000","last_active":"2024-03-18T16:21:54.831640+0000","last_peered":"2024-03-18T16:21:54.831640+0000","last_clean":"2024-03-18T16:21:54.831640+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:21:54.831640+0000","last_undegraded":"2024-03-18T16:21:54.831640+0000","last_fullsized":"2024-03-18T16:21:54.831640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277550+0000","last_change":"2024-03-18T16:21:23.057150+0000","last_active":"2024-03-18T16:21:55.277550+0000","last_peered":"2024-03-18T16:21:55.277550+0000","last_clean":"2024-03-18T16:21:55.277550+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:21:55.277550+0000","last_undegraded":"2024-03-18T16:21:55.277550+0000","last_fullsized":"2024-03-18T16:21:55.277550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.841077+0000","last_change":"2024-03-18T16:21:23.047968+0000","last_active":"2024-03-18T16:21:54.841077+0000","last_peered":"2024-03-18T16:21:54.841077+0000","last_clean":"2024-03-18T16:21:54.841077+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:21:54.841077+0000","last_undegraded":"2024-03-18T16:21:54.841077+0000","last_fullsized":"2024-03-18T16:21:54.841077+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.148114+0000","last_change":"2024-03-18T16:21:23.041794+0000","last_active":"2024-03-18T16:21:55.148114+0000","last_peered":"2024-03-18T16:21:55.148114+0000","last_clean":"2024-03-18T16:21:55.148114+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:21:55.148114+0000","last_undegraded":"2024-03-18T16:21:55.148114+0000","last_fullsized":"2024-03-18T16:21:55.148114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:13.065494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273737+0000","last_change":"2024-03-18T16:21:23.036940+0000","last_active":"2024-03-18T16:21:55.273737+0000","last_peered":"2024-03-18T16:21:55.273737+0000","last_clean":"2024-03-18T16:21:55.273737+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:21:55.273737+0000","last_undegraded":"2024-03-18T16:21:55.273737+0000","last_fullsized":"2024-03-18T16:21:55.273737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:45:38.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.273810+0000","last_change":"2024-03-18T16:21:23.042140+0000","last_active":"2024-03-18T16:21:55.273810+0000","last_peered":"2024-03-18T16:21:55.273810+0000","last_clean":"2024-03-18T16:21:55.273810+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:21:55.273810+0000","last_undegraded":"2024-03-18T16:21:55.273810+0000","last_fullsized":"2024-03-18T16:21:55.273810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266289+0000","last_change":"2024-03-18T16:21:22.995117+0000","last_active":"2024-03-18T16:21:55.266289+0000","last_peered":"2024-03-18T16:21:55.266289+0000","last_clean":"2024-03-18T16:21:55.266289+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:21:55.266289+0000","last_undegraded":"2024-03-18T16:21:55.266289+0000","last_fullsized":"2024-03-18T16:21:55.266289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:57.544738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.839997+0000","last_change":"2024-03-18T16:21:22.948123+0000","last_active":"2024-03-18T16:21:54.839997+0000","last_peered":"2024-03-18T16:21:54.839997+0000","last_clean":"2024-03-18T16:21:54.839997+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:21:54.839997+0000","last_undegraded":"2024-03-18T16:21:54.839997+0000","last_fullsized":"2024-03-18T16:21:54.839997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265696+0000","last_change":"2024-03-18T16:21:23.049877+0000","last_active":"2024-03-18T16:21:55.265696+0000","last_peered":"2024-03-18T16:21:55.265696+0000","last_clean":"2024-03-18T16:21:55.265696+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:21:55.265696+0000","last_undegraded":"2024-03-18T16:21:55.265696+0000","last_fullsized":"2024-03-18T16:21:55.265696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064503+0000","last_change":"2024-03-18T16:21:23.049483+0000","last_active":"2024-03-18T16:21:55.064503+0000","last_peered":"2024-03-18T16:21:55.064503+0000","last_clean":"2024-03-18T16:21:55.064503+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:21:55.064503+0000","last_undegraded":"2024-03-18T16:21:55.064503+0000","last_fullsized":"2024-03-18T16:21:55.064503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.265858+0000","last_change":"2024-03-18T16:21:22.995726+0000","last_active":"2024-03-18T16:21:55.265858+0000","last_peered":"2024-03-18T16:21:55.265858+0000","last_clean":"2024-03-18T16:21:55.265858+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:21:55.265858+0000","last_undegraded":"2024-03-18T16:21:55.265858+0000","last_fullsized":"2024-03-18T16:21:55.265858+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.064410+0000","last_change":"2024-03-18T16:21:22.949640+0000","last_active":"2024-03-18T16:21:55.064410+0000","last_peered":"2024-03-18T16:21:55.064410+0000","last_clean":"2024-03-18T16:21:55.064410+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:21:55.064410+0000","last_undegraded":"2024-03-18T16:21:55.064410+0000","last_fullsized":"2024-03-18T16:21:55.064410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:21.734293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.277697+0000","last_change":"2024-03-18T16:21:23.050203+0000","last_active":"2024-03-18T16:21:55.277697+0000","last_peered":"2024-03-18T16:21:55.277697+0000","last_clean":"2024-03-18T16:21:55.277697+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:21:55.277697+0000","last_undegraded":"2024-03-18T16:21:55.277697+0000","last_fullsized":"2024-03-18T16:21:55.277697+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:21.982408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:54.834586+0000","last_change":"2024-03-18T16:21:23.053003+0000","last_active":"2024-03-18T16:21:54.834586+0000","last_peered":"2024-03-18T16:21:54.834586+0000","last_clean":"2024-03-18T16:21:54.834586+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:21:54.834586+0000","last_undegraded":"2024-03-18T16:21:54.834586+0000","last_fullsized":"2024-03-18T16:21:54.834586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-03-18T16:21:55.266295+0000","last_change":"2024-03-18T16:21:23.041863+0000","last_active":"2024-03-18T16:21:55.266295+0000","last_peered":"2024-03-18T16:21:55.266295+0000","last_clean":"2024-03-18T16:21:55.266295+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:21:55.266295+0000","last_undegraded":"2024-03-18T16:21:55.266295+0000","last_fullsized":"2024-03-18T16:21:55.266295+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:26.190759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":94,"num_object_clones":0,"num_object_copies":188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":203,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":19940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509460,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":409600,"data_stored":721970,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509460,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":130832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509459,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":134168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509460,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":331776,"data_stored":130446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509459,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":401408,"data_stored":720744,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542163,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":339968,"data_stored":130964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":335872,"data_stored":134134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":344064,"data_stored":132230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:22:23.231 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-18T16:22:23.232 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T16:22:23.232 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-18T16:22:23.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-18T16:22:23.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.436+0000 7f4935a00640 1 Processor -- start 2024-03-18T16:22:23.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.436+0000 7f4935a00640 1 -- start start 2024-03-18T16:22:23.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.438+0000 7f4935a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f49300cd320 0x7f49300ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:23.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.438+0000 7f4935a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49300ca9b0 con 0x7f493015aa20 2024-03-18T16:22:23.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.438+0000 7f4935a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49300cc390 con 0x7f49300cde90 2024-03-18T16:22:23.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.438+0000 7f4935a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4930154ae0 con 0x7f49300cd320 2024-03-18T16:22:23.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.438+0000 7f492f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f493015aa20 0x7f49300ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36150/0 (socket says 172.21.15.102:36150) 2024-03-18T16:22:23.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.438+0000 7f492f400640 1 -- 172.21.15.102:0/3906234817 learned_addr learned my addr 172.21.15.102:0/3906234817 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:23.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.439+0000 7f4935000640 1 --2- 172.21.15.102:0/3906234817 >> v2:172.21.15.154:3300/0 conn(0x7f49300cd320 0x7f49300ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:23.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.442+0000 7f492ea00640 1 -- 172.21.15.102:0/3906234817 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127815514 0 0) 0x7f49300ca9b0 con 0x7f493015aa20 2024-03-18T16:22:23.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.442+0000 7f492ea00640 1 -- 172.21.15.102:0/3906234817 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4914003520 con 0x7f493015aa20 2024-03-18T16:22:23.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.444+0000 7f4935000640 1 -- 172.21.15.102:0/3906234817 >> v1:172.21.15.154:6789/0 conn(0x7f49300cde90 legacy=0x7f493015a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:23.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.444+0000 7f4935000640 1 -- 172.21.15.102:0/3906234817 >> v1:172.21.15.102:6789/0 conn(0x7f493015aa20 legacy=0x7f49300ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:23.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.444+0000 7f4935000640 1 -- 172.21.15.102:0/3906234817 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49300cc670 con 0x7f49300cd320 2024-03-18T16:22:23.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.448+0000 7f4935000640 1 --2- 172.21.15.102:0/3906234817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 0x7f49300ce980 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4928009d90 tx=0x7f492802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6290f1cb29a27307 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:23.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.453+0000 7f492ea00640 1 -- 172.21.15.102:0/3906234817 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4928039070 con 0x7f49300cd320 2024-03-18T16:22:23.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.453+0000 7f492ea00640 1 -- 172.21.15.102:0/3906234817 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f492802cd00 con 0x7f49300cd320 2024-03-18T16:22:23.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.453+0000 7f492ea00640 1 -- 172.21.15.102:0/3906234817 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4928034660 con 0x7f49300cd320 2024-03-18T16:22:23.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.454+0000 7f4935a00640 1 -- 172.21.15.102:0/3906234817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 msgr2=0x7f49300ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:23.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.454+0000 7f4935a00640 1 --2- 172.21.15.102:0/3906234817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 0x7f49300ce980 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4928009d90 tx=0x7f492802bf50 comp rx=0 tx=0).stop 2024-03-18T16:22:23.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.454+0000 7f4935a00640 1 -- 172.21.15.102:0/3906234817 shutdown_connections 2024-03-18T16:22:23.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.454+0000 7f4935a00640 1 --2- 172.21.15.102:0/3906234817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 0x7f49300ce980 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:23.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.454+0000 7f4935a00640 1 -- 172.21.15.102:0/3906234817 >> 172.21.15.102:0/3906234817 conn(0x7f49300cfdd0 msgr2=0x7f49300d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:23.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.454+0000 7f4935a00640 1 -- 172.21.15.102:0/3906234817 shutdown_connections 2024-03-18T16:22:23.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.454+0000 7f4935a00640 1 -- 172.21.15.102:0/3906234817 wait complete. 2024-03-18T16:22:23.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.456+0000 7f4935a00640 1 Processor -- start 2024-03-18T16:22:23.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.456+0000 7f4935a00640 1 -- start start 2024-03-18T16:22:23.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.457+0000 7f4935a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 0x7f49300c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:23.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.457+0000 7f4935a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49300cde90 0x7f493014dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:23.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.457+0000 7f4935a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f493015aa20 0x7f493014e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:23.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.457+0000 7f4935a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f49300cac90 con 0x7f49300cde90 2024-03-18T16:22:23.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.457+0000 7f4935a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f49300ceec0 con 0x7f49300cd320 2024-03-18T16:22:23.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.457+0000 7f4935a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f49300ca570 con 0x7f493015aa20 2024-03-18T16:22:23.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.458+0000 7f492f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49300cde90 0x7f493014dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:23.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.458+0000 7f492fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 0x7f49300c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:23.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.458+0000 7f492fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 0x7f49300c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54736/0 (socket says 172.21.15.102:54736) 2024-03-18T16:22:23.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.458+0000 7f492f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49300cde90 0x7f493014dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55582/0 (socket says 172.21.15.102:55582) 2024-03-18T16:22:23.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.458+0000 7f492fe00640 1 -- 172.21.15.102:0/2581115774 learned_addr learned my addr 172.21.15.102:0/2581115774 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:23.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.459+0000 7f4935000640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f493015aa20 0x7f493014e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:23.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.462+0000 7f492f400640 1 -- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f493015aa20 msgr2=0x7f493014e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:23.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.462+0000 7f492f400640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f493015aa20 0x7f493014e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:23.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.462+0000 7f492f400640 1 -- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 msgr2=0x7f49300c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:23.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.462+0000 7f492f400640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 0x7f49300c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:23.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.462+0000 7f492f400640 1 -- 172.21.15.102:0/2581115774 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49300c6ad0 con 0x7f49300cde90 2024-03-18T16:22:23.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.465+0000 7f492f400640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49300cde90 0x7f493014dee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f491c00c3c0 tx=0x7f491c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:23.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.471+0000 7f4927e00640 1 -- 172.21.15.102:0/2581115774 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c017070 con 0x7f49300cde90 2024-03-18T16:22:23.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.471+0000 7f4927e00640 1 -- 172.21.15.102:0/2581115774 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f491c005ce0 con 0x7f49300cde90 2024-03-18T16:22:23.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.471+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4930151850 con 0x7f49300cde90 2024-03-18T16:22:23.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.471+0000 7f4927e00640 1 -- 172.21.15.102:0/2581115774 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c0072d0 con 0x7f49300cde90 2024-03-18T16:22:23.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.471+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49300c6510 con 0x7f49300cde90 2024-03-18T16:22:23.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.473+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4930153c30 con 0x7f49300cde90 2024-03-18T16:22:23.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.510+0000 7f4927e00640 1 -- 172.21.15.102:0/2581115774 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f491c01a070 con 0x7f49300cde90 2024-03-18T16:22:23.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.512+0000 7f4927e00640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f491006bc20 0x7f491006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:23.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.512+0000 7f492fe00640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f491006bc20 0x7f491006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:23.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.513+0000 7f492fe00640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f491006bc20 0x7f491006e0e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4920001840 tx=0x7f4920009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:23.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.514+0000 7f4927e00640 1 -- 172.21.15.102:0/2581115774 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f491c0566e0 con 0x7f49300cde90 2024-03-18T16:22:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:23.821+0000 7f4927e00640 1 -- 172.21.15.102:0/2581115774 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4930153c30 con 0x7f49300cde90 2024-03-18T16:22:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.065+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f493004f5b0 con 0x7f49300cde90 2024-03-18T16:22:24.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.077+0000 7f4927e00640 1 -- 172.21.15.102:0/2581115774 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f491c058640 con 0x7f49300cde90 2024-03-18T16:22:24.080 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:24.080 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T16:22:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f491006bc20 msgr2=0x7f491006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f491006bc20 0x7f491006e0e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4920001840 tx=0x7f4920009040 comp rx=0 tx=0).stop 2024-03-18T16:22:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49300cde90 msgr2=0x7f493014dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49300cde90 0x7f493014dee0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f491c00c3c0 tx=0x7f491c001e70 comp rx=0 tx=0).stop 2024-03-18T16:22:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 shutdown_connections 2024-03-18T16:22:24.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f491006bc20 0x7f491006e0e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f493015aa20 0x7f493014e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49300cde90 0x7f493014dee0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 --2- 172.21.15.102:0/2581115774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49300cd320 0x7f49300c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.081+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 >> 172.21.15.102:0/2581115774 conn(0x7f49300cfdd0 msgr2=0x7f4930159b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.082+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 shutdown_connections 2024-03-18T16:22:24.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.082+0000 7f4935a00640 1 -- 172.21.15.102:0/2581115774 wait complete. 2024-03-18T16:22:24.107 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-18T16:22:24.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:22:24.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.279+0000 7f7fbfe00640 1 Processor -- start 2024-03-18T16:22:24.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.279+0000 7f7fbfe00640 1 -- start start 2024-03-18T16:22:24.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.280+0000 7f7fbfe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7fc015ae70 0x7f7fc00ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:24.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.280+0000 7f7fbfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7fc00c9be0 0x7f7fc00d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:24.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.280+0000 7f7fbfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00d05a0 0x7f7fc00cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:24.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.280+0000 7f7fbfe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7fc0154ab0 con 0x7f7fc00c9be0 2024-03-18T16:22:24.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.280+0000 7f7fbfe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7fc015b250 con 0x7f7fc015ae70 2024-03-18T16:22:24.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.280+0000 7f7fbfe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7fc006d2f0 con 0x7f7fc00d05a0 2024-03-18T16:22:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.281+0000 7f7fbea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7fc015ae70 0x7f7fc00ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.281+0000 7f7fbe000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7fc00c9be0 0x7f7fc00d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.281+0000 7f7fbf400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00d05a0 0x7f7fc00cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:24.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.281+0000 7f7fbea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7fc015ae70 0x7f7fc00ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54740/0 (socket says 172.21.15.102:54740) 2024-03-18T16:22:24.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.281+0000 7f7fbea00640 1 -- 172.21.15.102:0/3289940754 learned_addr learned my addr 172.21.15.102:0/3289940754 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.284+0000 7f7fbf400640 1 -- 172.21.15.102:0/3289940754 >> v2:172.21.15.154:3300/0 conn(0x7f7fc015ae70 msgr2=0x7f7fc00ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.284+0000 7f7fbf400640 1 --2- 172.21.15.102:0/3289940754 >> v2:172.21.15.154:3300/0 conn(0x7f7fc015ae70 0x7f7fc00ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.284+0000 7f7fbf400640 1 -- 172.21.15.102:0/3289940754 >> v2:172.21.15.102:3300/0 conn(0x7f7fc00c9be0 msgr2=0x7f7fc00d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.284+0000 7f7fbf400640 1 --2- 172.21.15.102:0/3289940754 >> v2:172.21.15.102:3300/0 conn(0x7f7fc00c9be0 0x7f7fc00d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.284+0000 7f7fbf400640 1 -- 172.21.15.102:0/3289940754 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc006d5d0 con 0x7f7fc00d05a0 2024-03-18T16:22:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.286+0000 7f7fbf400640 1 --2- 172.21.15.102:0/3289940754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00d05a0 0x7f7fc00cc4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00cd5c0 tx=0x7f7fb802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df00fa6df674b39 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:24.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.289+0000 7f7fbd600640 1 -- 172.21.15.102:0/3289940754 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8039070 con 0x7f7fc00d05a0 2024-03-18T16:22:24.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.290+0000 7f7fbd600640 1 -- 172.21.15.102:0/3289940754 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fb802cbd0 con 0x7f7fc00d05a0 2024-03-18T16:22:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbd600640 1 -- 172.21.15.102:0/3289940754 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb8034440 con 0x7f7fc00d05a0 2024-03-18T16:22:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbfe00640 1 -- 172.21.15.102:0/3289940754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00d05a0 msgr2=0x7f7fc00cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/3289940754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00d05a0 0x7f7fc00cc4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00cd5c0 tx=0x7f7fb802be20 comp rx=0 tx=0).stop 2024-03-18T16:22:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbfe00640 1 -- 172.21.15.102:0/3289940754 shutdown_connections 2024-03-18T16:22:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/3289940754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00d05a0 0x7f7fc00cc4f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/3289940754 >> v2:172.21.15.102:3300/0 conn(0x7f7fc00c9be0 0x7f7fc00d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/3289940754 >> v2:172.21.15.154:3300/0 conn(0x7f7fc015ae70 0x7f7fc00ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbfe00640 1 -- 172.21.15.102:0/3289940754 >> 172.21.15.102:0/3289940754 conn(0x7f7fc00673a0 msgr2=0x7f7fc00677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:24.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbfe00640 1 -- 172.21.15.102:0/3289940754 shutdown_connections 2024-03-18T16:22:24.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.291+0000 7f7fbfe00640 1 -- 172.21.15.102:0/3289940754 wait complete. 2024-03-18T16:22:24.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.293+0000 7f7fbfe00640 1 Processor -- start 2024-03-18T16:22:24.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.293+0000 7f7fbfe00640 1 -- start start 2024-03-18T16:22:24.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.294+0000 7f7fbfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00c9be0 0x7f7fc0150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:24.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.294+0000 7f7fbfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fc00d05a0 0x7f7fc00b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:24.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.294+0000 7f7fbfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc015ae70 0x7f7fc00b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:24.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.294+0000 7f7fbfe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc006de70 con 0x7f7fc00d05a0 2024-03-18T16:22:24.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.294+0000 7f7fbfe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc006db90 con 0x7f7fc015ae70 2024-03-18T16:22:24.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.294+0000 7f7fbfe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7fc006e150 con 0x7f7fc00c9be0 2024-03-18T16:22:24.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.295+0000 7f7fbe000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fc00d05a0 0x7f7fc00b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:24.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.295+0000 7f7fbf400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc015ae70 0x7f7fc00b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:24.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.295+0000 7f7fbea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00c9be0 0x7f7fc0150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:24.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.295+0000 7f7fbe000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fc00d05a0 0x7f7fc00b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55600/0 (socket says 172.21.15.102:55600) 2024-03-18T16:22:24.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.295+0000 7f7fbe000640 1 -- 172.21.15.102:0/872563977 learned_addr learned my addr 172.21.15.102:0/872563977 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:24.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.299+0000 7f7fbea00640 1 -- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc015ae70 msgr2=0x7f7fc00b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:24.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.299+0000 7f7fbea00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc015ae70 0x7f7fc00b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.299+0000 7f7fbea00640 1 -- 172.21.15.102:0/872563977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fc00d05a0 msgr2=0x7f7fc00b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:24.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.299+0000 7f7fbea00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fc00d05a0 0x7f7fc00b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.299+0000 7f7fbea00640 1 -- 172.21.15.102:0/872563977 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc01efa50 con 0x7f7fc00c9be0 2024-03-18T16:22:24.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.301+0000 7f7fbea00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00c9be0 0x7f7fc0150ea0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000e850 tx=0x7f7fb000ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:24.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.305+0000 7f7fb6a00640 1 -- 172.21.15.102:0/872563977 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0017070 con 0x7f7fc00c9be0 2024-03-18T16:22:24.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.305+0000 7f7fb6a00640 1 -- 172.21.15.102:0/872563977 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fb000c940 con 0x7f7fc00c9be0 2024-03-18T16:22:24.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.305+0000 7f7fb6a00640 1 -- 172.21.15.102:0/872563977 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb000ccf0 con 0x7f7fc00c9be0 2024-03-18T16:22:24.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.305+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc01efcf0 con 0x7f7fc00c9be0 2024-03-18T16:22:24.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.306+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc01f02b0 con 0x7f7fc00c9be0 2024-03-18T16:22:24.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.307+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc00c8510 con 0x7f7fc00c9be0 2024-03-18T16:22:24.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.326+0000 7f7fb6a00640 1 -- 172.21.15.102:0/872563977 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7fb001a070 con 0x7f7fc00c9be0 2024-03-18T16:22:24.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.327+0000 7f7fb6a00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fa006bc20 0x7f7fa006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:24.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.327+0000 7f7fbe000640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fa006bc20 0x7f7fa006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:24.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.328+0000 7f7fbe000640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fa006bc20 0x7f7fa006e0e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00b0030 tx=0x7f7fac009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:24.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.329+0000 7f7fb6a00640 1 -- 172.21.15.102:0/872563977 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f7fb0057630 con 0x7f7fc00c9be0 2024-03-18T16:22:24.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.578+0000 7f7fb6a00640 1 -- 172.21.15.102:0/872563977 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7fc00c8510 con 0x7f7fc00c9be0 2024-03-18T16:22:24.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.817+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7fc0074fd0 con 0x7f7fc00c9be0 2024-03-18T16:22:24.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.859+0000 7f7fb6a00640 1 -- 172.21.15.102:0/872563977 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18907 (secure 0 0 0) 0x7f7fb0059590 con 0x7f7fc00c9be0 2024-03-18T16:22:24.861 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:24.861 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":29,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:54.749120+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fa006bc20 msgr2=0x7f7fa006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fa006bc20 0x7f7fa006e0e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00b0030 tx=0x7f7fac009190 comp rx=0 tx=0).stop 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00c9be0 msgr2=0x7f7fc0150ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00c9be0 0x7f7fc0150ea0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7fb000e850 tx=0x7f7fb000ed10 comp rx=0 tx=0).stop 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 shutdown_connections 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fa006bc20 0x7f7fa006e0e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7fc015ae70 0x7f7fc00b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fc00d05a0 0x7f7fc00b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 --2- 172.21.15.102:0/872563977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7fc00c9be0 0x7f7fc0150ea0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 >> 172.21.15.102:0/872563977 conn(0x7f7fc00673a0 msgr2=0x7f7fc00c7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 shutdown_connections 2024-03-18T16:22:24.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:24.862+0000 7f7fbfe00640 1 -- 172.21.15.102:0/872563977 wait complete. 2024-03-18T16:22:24.884 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:21:05.204661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:21:19.524753+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:21:21.031843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:21:26.818690+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:22:24.884 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:22:25.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.064+0000 7f3e65200640 1 Processor -- start 2024-03-18T16:22:25.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.064+0000 7f3e65200640 1 -- start start 2024-03-18T16:22:25.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.065+0000 7f3e65200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3e60154620 0x7f3e600ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.065+0000 7f3e65200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e600d0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.065+0000 7f3e65200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e6015a4a0 con 0x7f3e60154a00 2024-03-18T16:22:25.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.065+0000 7f3e65200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3e600ced50 con 0x7f3e60154620 2024-03-18T16:22:25.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.065+0000 7f3e65200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3e6015b370 con 0x7f3e600cff10 2024-03-18T16:22:25.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.066+0000 7f3e5fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e600d0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:25.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.066+0000 7f3e5ea00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3e60154a00 0x7f3e600ca520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:49860/0 (socket says 172.21.15.102:49860) 2024-03-18T16:22:25.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.066+0000 7f3e5ea00640 1 -- 172.21.15.102:0/3719811948 learned_addr learned my addr 172.21.15.102:0/3719811948 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:25.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.069+0000 7f3e5fe00640 1 -- 172.21.15.102:0/3719811948 >> v2:172.21.15.102:3300/0 conn(0x7f3e60154620 msgr2=0x7f3e600ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:25.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.069+0000 7f3e5fe00640 1 --2- 172.21.15.102:0/3719811948 >> v2:172.21.15.102:3300/0 conn(0x7f3e60154620 0x7f3e600ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.069+0000 7f3e5fe00640 1 -- 172.21.15.102:0/3719811948 >> v1:172.21.15.154:6789/0 conn(0x7f3e60154a00 legacy=0x7f3e600ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.069+0000 7f3e5fe00640 1 -- 172.21.15.102:0/3719811948 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e6015a760 con 0x7f3e600cff10 2024-03-18T16:22:25.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.071+0000 7f3e5fe00640 1 --2- 172.21.15.102:0/3719811948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e600d0360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009d90 tx=0x7f3e5802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad60886c83df664 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:25.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.074+0000 7f3e5e000640 1 -- 172.21.15.102:0/3719811948 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58039070 con 0x7f3e600cff10 2024-03-18T16:22:25.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.076+0000 7f3e5e000640 1 -- 172.21.15.102:0/3719811948 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e5802cce0 con 0x7f3e600cff10 2024-03-18T16:22:25.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.076+0000 7f3e5e000640 1 -- 172.21.15.102:0/3719811948 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58034660 con 0x7f3e600cff10 2024-03-18T16:22:25.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.076+0000 7f3e65200640 1 -- 172.21.15.102:0/3719811948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 msgr2=0x7f3e600d0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.076+0000 7f3e65200640 1 --2- 172.21.15.102:0/3719811948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e600d0360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009d90 tx=0x7f3e5802bf30 comp rx=0 tx=0).stop 2024-03-18T16:22:25.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.077+0000 7f3e65200640 1 -- 172.21.15.102:0/3719811948 shutdown_connections 2024-03-18T16:22:25.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.077+0000 7f3e65200640 1 --2- 172.21.15.102:0/3719811948 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e600d0360 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.077+0000 7f3e65200640 1 --2- 172.21.15.102:0/3719811948 >> v2:172.21.15.102:3300/0 conn(0x7f3e60154620 0x7f3e600ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.077+0000 7f3e65200640 1 -- 172.21.15.102:0/3719811948 >> 172.21.15.102:0/3719811948 conn(0x7f3e60067160 msgr2=0x7f3e60067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:25.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.077+0000 7f3e65200640 1 -- 172.21.15.102:0/3719811948 shutdown_connections 2024-03-18T16:22:25.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.077+0000 7f3e65200640 1 -- 172.21.15.102:0/3719811948 wait complete. 2024-03-18T16:22:25.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.079+0000 7f3e65200640 1 Processor -- start 2024-03-18T16:22:25.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.079+0000 7f3e65200640 1 -- start start 2024-03-18T16:22:25.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.081+0000 7f3e65200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e60152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.081+0000 7f3e65200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e60154620 0x7f3e601f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.081+0000 7f3e65200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e60154a00 0x7f3e6015b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.081+0000 7f3e65200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6015b000 con 0x7f3e60154a00 2024-03-18T16:22:25.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.081+0000 7f3e65200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6015ad20 con 0x7f3e60154620 2024-03-18T16:22:25.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.081+0000 7f3e65200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3e6006d4b0 con 0x7f3e600cff10 2024-03-18T16:22:25.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.082+0000 7f3e5f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e60152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:25.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.082+0000 7f3e5f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e60152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48080/0 (socket says 172.21.15.102:48080) 2024-03-18T16:22:25.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.082+0000 7f3e5f400640 1 -- 172.21.15.102:0/968628731 learned_addr learned my addr 172.21.15.102:0/968628731 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:25.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.082+0000 7f3e5ea00640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e60154620 0x7f3e601f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:25.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.085+0000 7f3e5f400640 1 -- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e60154620 msgr2=0x7f3e601f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.085+0000 7f3e5f400640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e60154620 0x7f3e601f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.085+0000 7f3e5f400640 1 -- 172.21.15.102:0/968628731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e60154a00 msgr2=0x7f3e6015b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.085+0000 7f3e5f400640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e60154a00 0x7f3e6015b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.085+0000 7f3e5f400640 1 -- 172.21.15.102:0/968628731 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e6014d0b0 con 0x7f3e600cff10 2024-03-18T16:22:25.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.088+0000 7f3e5f400640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e60152680 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3e5000b4d0 tx=0x7f3e5000b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:25.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.093+0000 7f3e57400640 1 -- 172.21.15.102:0/968628731 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e50014070 con 0x7f3e600cff10 2024-03-18T16:22:25.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.093+0000 7f3e57400640 1 -- 172.21.15.102:0/968628731 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e50010870 con 0x7f3e600cff10 2024-03-18T16:22:25.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.093+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e6015bcd0 con 0x7f3e600cff10 2024-03-18T16:22:25.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.093+0000 7f3e57400640 1 -- 172.21.15.102:0/968628731 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e50010c60 con 0x7f3e600cff10 2024-03-18T16:22:25.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.093+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e601537c0 con 0x7f3e600cff10 2024-03-18T16:22:25.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.094+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e60074fd0 con 0x7f3e600cff10 2024-03-18T16:22:25.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.404+0000 7f3e57400640 1 -- 172.21.15.102:0/968628731 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3e50017070 con 0x7f3e600cff10 2024-03-18T16:22:25.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.406+0000 7f3e57400640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e4006bbd0 0x7f3e4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.407+0000 7f3e5ea00640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e4006bbd0 0x7f3e4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:25.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.407+0000 7f3e57400640 1 -- 172.21.15.102:0/968628731 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f3e5008af40 con 0x7f3e600cff10 2024-03-18T16:22:25.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.408+0000 7f3e5ea00640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e4006bbd0 0x7f3e4006e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3e600c6a00 tx=0x7f3e4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:25.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.428+0000 7f3e57400640 1 -- 172.21.15.102:0/968628731 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3e500590c0 con 0x7f3e600cff10 2024-03-18T16:22:25.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.624+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e600b5b20 con 0x7f3e600cff10 2024-03-18T16:22:25.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.629+0000 7f3e57400640 1 -- 172.21.15.102:0/968628731 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f3e5005d160 con 0x7f3e600cff10 2024-03-18T16:22:25.631 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-18T16:22:25.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e4006bbd0 msgr2=0x7f3e4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e4006bbd0 0x7f3e4006e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3e600c6a00 tx=0x7f3e4c009040 comp rx=0 tx=0).stop 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 msgr2=0x7f3e60152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e60152680 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3e5000b4d0 tx=0x7f3e5000b990 comp rx=0 tx=0).stop 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 shutdown_connections 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e4006bbd0 0x7f3e4006e090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e60154a00 0x7f3e6015b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e60154620 0x7f3e601f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 --2- 172.21.15.102:0/968628731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e600cff10 0x7f3e60152680 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.631+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 >> 172.21.15.102:0/968628731 conn(0x7f3e60067160 msgr2=0x7f3e6014e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.632+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 shutdown_connections 2024-03-18T16:22:25.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.632+0000 7f3e65200640 1 -- 172.21.15.102:0/968628731 wait complete. 2024-03-18T16:22:25.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:22:25.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.817+0000 7ff26a000640 1 Processor -- start 2024-03-18T16:22:25.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.817+0000 7ff26a000640 1 -- start start 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.818+0000 7ff26a000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff264152410 0x7ff264158b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.819+0000 7ff26a000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2641527f0 con 0x7ff26406bdd0 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.819+0000 7ff26a000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff26415bc00 con 0x7ff26415b630 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.819+0000 7ff26a000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff264159080 con 0x7ff264152410 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.819+0000 7ff269600640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff26415b630 0x7ff2640c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:49872/0 (socket says 172.21.15.102:49872) 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.819+0000 7ff269600640 1 -- 172.21.15.102:0/1225370824 learned_addr learned my addr 172.21.15.102:0/1225370824 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.820+0000 7ff268c00640 1 --2- 172.21.15.102:0/1225370824 >> v2:172.21.15.102:3300/0 conn(0x7ff264152410 0x7ff264158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.823+0000 7ff263400640 1 -- 172.21.15.102:0/1225370824 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914596073 0 0) 0x7ff26415bc00 con 0x7ff26415b630 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.823+0000 7ff263400640 1 -- 172.21.15.102:0/1225370824 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff24c003520 con 0x7ff26415b630 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.827+0000 7ff268c00640 1 -- 172.21.15.102:0/1225370824 >> v1:172.21.15.154:6789/0 conn(0x7ff26415b630 legacy=0x7ff2640c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.827+0000 7ff268c00640 1 -- 172.21.15.102:0/1225370824 >> v1:172.21.15.102:6789/0 conn(0x7ff26406bdd0 legacy=0x7ff26406d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.827+0000 7ff268c00640 1 -- 172.21.15.102:0/1225370824 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26415bee0 con 0x7ff264152410 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.832+0000 7ff268c00640 1 --2- 172.21.15.102:0/1225370824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff264152410 0x7ff264158b40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff2580097a0 tx=0x7ff25802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b0647c60691549 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.838+0000 7ff263400640 1 -- 172.21.15.102:0/1225370824 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258034070 con 0x7ff264152410 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.838+0000 7ff263400640 1 -- 172.21.15.102:0/1225370824 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff258030610 con 0x7ff264152410 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.838+0000 7ff263400640 1 -- 172.21.15.102:0/1225370824 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2580309e0 con 0x7ff264152410 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.838+0000 7ff26a000640 1 -- 172.21.15.102:0/1225370824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff264152410 msgr2=0x7ff264158b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.838+0000 7ff26a000640 1 --2- 172.21.15.102:0/1225370824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff264152410 0x7ff264158b40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff2580097a0 tx=0x7ff25802b960 comp rx=0 tx=0).stop 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.839+0000 7ff26a000640 1 -- 172.21.15.102:0/1225370824 shutdown_connections 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.839+0000 7ff26a000640 1 --2- 172.21.15.102:0/1225370824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff264152410 0x7ff264158b40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.839+0000 7ff26a000640 1 -- 172.21.15.102:0/1225370824 >> 172.21.15.102:0/1225370824 conn(0x7ff26406b200 msgr2=0x7ff26406b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.839+0000 7ff26a000640 1 -- 172.21.15.102:0/1225370824 shutdown_connections 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.839+0000 7ff26a000640 1 -- 172.21.15.102:0/1225370824 wait complete. 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.841+0000 7ff26a000640 1 Processor -- start 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.841+0000 7ff26a000640 1 -- start start 2024-03-18T16:22:25.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.842+0000 7ff26a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26406bdd0 0x7ff2641efc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.842+0000 7ff26a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264152410 0x7ff26414d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.842+0000 7ff26a000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff26415b630 0x7ff2640c3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.842+0000 7ff26a000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff26415c4a0 con 0x7ff26406bdd0 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.842+0000 7ff26a000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff26406d250 con 0x7ff26415b630 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.842+0000 7ff26a000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff26406cee0 con 0x7ff264152410 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.842+0000 7ff269600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff26415b630 0x7ff2640c3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.842+0000 7ff268c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26406bdd0 0x7ff2641efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.843+0000 7ff263e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264152410 0x7ff26414d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.843+0000 7ff269600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff26415b630 0x7ff2640c3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54766/0 (socket says 172.21.15.102:54766) 2024-03-18T16:22:25.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.843+0000 7ff269600640 1 -- 172.21.15.102:0/3791817711 learned_addr learned my addr 172.21.15.102:0/3791817711 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:25.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.846+0000 7ff269600640 1 -- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264152410 msgr2=0x7ff26414d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.846+0000 7ff269600640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264152410 0x7ff26414d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.846+0000 7ff269600640 1 -- 172.21.15.102:0/3791817711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26406bdd0 msgr2=0x7ff2641efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:25.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.846+0000 7ff269600640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26406bdd0 0x7ff2641efc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:25.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.846+0000 7ff269600640 1 -- 172.21.15.102:0/3791817711 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff264159660 con 0x7ff26415b630 2024-03-18T16:22:25.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.848+0000 7ff269600640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff26415b630 0x7ff2640c3520 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00cb70 tx=0x7ff25c00a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.852+0000 7ff260c00640 1 -- 172.21.15.102:0/3791817711 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c017070 con 0x7ff26415b630 2024-03-18T16:22:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.852+0000 7ff260c00640 1 -- 172.21.15.102:0/3791817711 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff25c00ad40 con 0x7ff26415b630 2024-03-18T16:22:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.852+0000 7ff260c00640 1 -- 172.21.15.102:0/3791817711 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c012620 con 0x7ff26415b630 2024-03-18T16:22:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.852+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff264159940 con 0x7ff26415b630 2024-03-18T16:22:25.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.852+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff264159f00 con 0x7ff26415b630 2024-03-18T16:22:25.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.854+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff264074fd0 con 0x7ff26415b630 2024-03-18T16:22:25.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.873+0000 7ff260c00640 1 -- 172.21.15.102:0/3791817711 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff25c01a070 con 0x7ff26415b630 2024-03-18T16:22:25.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.874+0000 7ff260c00640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24406bc20 0x7ff24406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:25.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.875+0000 7ff268c00640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24406bc20 0x7ff24406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:25.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.875+0000 7ff260c00640 1 -- 172.21.15.102:0/3791817711 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7ff25c0571f0 con 0x7ff26415b630 2024-03-18T16:22:25.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:25.876+0000 7ff268c00640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24406bc20 0x7ff24406e0e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff258036000 tx=0x7ff25803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:26.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.133+0000 7ff260c00640 1 -- 172.21.15.102:0/3791817711 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff25c021020 con 0x7ff26415b630 2024-03-18T16:22:26.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.417+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff264155ad0 con 0x7ff26415b630 2024-03-18T16:22:26.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.425+0000 7ff260c00640 1 -- 172.21.15.102:0/3791817711 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) ==== 104+0+11 (secure 0 0 0) 0x7ff25c059150 con 0x7ff26415b630 2024-03-18T16:22:26.427 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-18T16:22:26.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24406bc20 msgr2=0x7ff24406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:26.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24406bc20 0x7ff24406e0e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff258036000 tx=0x7ff25803b040 comp rx=0 tx=0).stop 2024-03-18T16:22:26.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff26415b630 msgr2=0x7ff2640c3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff26415b630 0x7ff2640c3520 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff25c00cb70 tx=0x7ff25c00a430 comp rx=0 tx=0).stop 2024-03-18T16:22:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 shutdown_connections 2024-03-18T16:22:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24406bc20 0x7ff24406e0e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff26415b630 0x7ff2640c3520 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff264152410 0x7ff26414d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 --2- 172.21.15.102:0/3791817711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff26406bdd0 0x7ff2641efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 >> 172.21.15.102:0/3791817711 conn(0x7ff26406b200 msgr2=0x7ff2640c5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.427+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 shutdown_connections 2024-03-18T16:22:26.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.428+0000 7ff26a000640 1 -- 172.21.15.102:0/3791817711 wait complete. 2024-03-18T16:22:26.447 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:22:26.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.615+0000 7fb0d5400640 1 Processor -- start 2024-03-18T16:22:26.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.615+0000 7fb0d5400640 1 -- start start 2024-03-18T16:22:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.616+0000 7fb0d5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d0154520 0x7fb0d00c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.616+0000 7fb0d5400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0d00c5920 0x7fb0d00c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.616+0000 7fb0d5400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d00c62f0 con 0x7fb0d015ac20 2024-03-18T16:22:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.616+0000 7fb0d5400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d0154900 con 0x7fb0d00c5920 2024-03-18T16:22:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.616+0000 7fb0d5400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d00cb8b0 con 0x7fb0d0154520 2024-03-18T16:22:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.617+0000 7fb0cf400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb0d015ac20 0x7fb0d00cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:49884/0 (socket says 172.21.15.102:49884) 2024-03-18T16:22:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.617+0000 7fb0cf400640 1 -- 172.21.15.102:0/562994458 learned_addr learned my addr 172.21.15.102:0/562994458 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.617+0000 7fb0d4a00640 1 --2- 172.21.15.102:0/562994458 >> v2:172.21.15.102:3300/0 conn(0x7fb0d00c5920 0x7fb0d00c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:26.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.617+0000 7fb0cfe00640 1 --2- 172.21.15.102:0/562994458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d0154520 0x7fb0d00c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.620+0000 7fb0d4a00640 1 -- 172.21.15.102:0/562994458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d0154520 msgr2=0x7fb0d00c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.620+0000 7fb0d4a00640 1 --2- 172.21.15.102:0/562994458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d0154520 0x7fb0d00c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.620+0000 7fb0d4a00640 1 -- 172.21.15.102:0/562994458 >> v1:172.21.15.154:6789/0 conn(0x7fb0d015ac20 legacy=0x7fb0d00cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.620+0000 7fb0d4a00640 1 -- 172.21.15.102:0/562994458 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d00cbb90 con 0x7fb0d00c5920 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.623+0000 7fb0d4a00640 1 --2- 172.21.15.102:0/562994458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d00c5920 0x7fb0d00c5db0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009dc0 tx=0x7fb0c802bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952c361fbe7c69cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.627+0000 7fb0cea00640 1 -- 172.21.15.102:0/562994458 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8039070 con 0x7fb0d00c5920 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.627+0000 7fb0cea00640 1 -- 172.21.15.102:0/562994458 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0c802ccc0 con 0x7fb0d00c5920 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.627+0000 7fb0cea00640 1 -- 172.21.15.102:0/562994458 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8034660 con 0x7fb0d00c5920 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.628+0000 7fb0d5400640 1 -- 172.21.15.102:0/562994458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d00c5920 msgr2=0x7fb0d00c5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.628+0000 7fb0d5400640 1 --2- 172.21.15.102:0/562994458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d00c5920 0x7fb0d00c5db0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8009dc0 tx=0x7fb0c802bf10 comp rx=0 tx=0).stop 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.628+0000 7fb0d5400640 1 -- 172.21.15.102:0/562994458 shutdown_connections 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.628+0000 7fb0d5400640 1 --2- 172.21.15.102:0/562994458 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d00c5920 0x7fb0d00c5db0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.628+0000 7fb0d5400640 1 --2- 172.21.15.102:0/562994458 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d0154520 0x7fb0d00c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.628+0000 7fb0d5400640 1 -- 172.21.15.102:0/562994458 >> 172.21.15.102:0/562994458 conn(0x7fb0d006c9a0 msgr2=0x7fb0d006cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.628+0000 7fb0d5400640 1 -- 172.21.15.102:0/562994458 shutdown_connections 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.628+0000 7fb0d5400640 1 -- 172.21.15.102:0/562994458 wait complete. 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.631+0000 7fb0d5400640 1 Processor -- start 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.631+0000 7fb0d5400640 1 -- start start 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.633+0000 7fb0d5400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0d00c5920 0x7fb0d014da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.633+0000 7fb0d5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d0154520 0x7fb0d00c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.633+0000 7fb0d5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d015ac20 0x7fb0d00c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.633+0000 7fb0d5400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00d09a0 con 0x7fb0d0154520 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.633+0000 7fb0d5400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d00cae20 con 0x7fb0d00c5920 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.633+0000 7fb0d5400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb0d015b040 con 0x7fb0d015ac20 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.633+0000 7fb0cfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0d00c5920 0x7fb0d014da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.634+0000 7fb0d4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d015ac20 0x7fb0d00c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.634+0000 7fb0cf400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d0154520 0x7fb0d00c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.634+0000 7fb0cf400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d0154520 0x7fb0d00c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55652/0 (socket says 172.21.15.102:55652) 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.634+0000 7fb0cfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0d00c5920 0x7fb0d014da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54770/0 (socket says 172.21.15.102:54770) 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.634+0000 7fb0cf400640 1 -- 172.21.15.102:0/2690010155 learned_addr learned my addr 172.21.15.102:0/2690010155 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.637+0000 7fb0d4a00640 1 -- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0d00c5920 msgr2=0x7fb0d014da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.637+0000 7fb0d4a00640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0d00c5920 0x7fb0d014da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.638+0000 7fb0d4a00640 1 -- 172.21.15.102:0/2690010155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d0154520 msgr2=0x7fb0d00c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.638+0000 7fb0d4a00640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d0154520 0x7fb0d00c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.638+0000 7fb0d4a00640 1 -- 172.21.15.102:0/2690010155 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d01f4b40 con 0x7fb0d015ac20 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.638+0000 7fb0cf400640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d0154520 0x7fb0d00c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.640+0000 7fb0d4a00640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d015ac20 0x7fb0d00c9f50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00d0e80 tx=0x7fb0c8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.644+0000 7fb0c7e00640 1 -- 172.21.15.102:0/2690010155 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8039050 con 0x7fb0d015ac20 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.644+0000 7fb0c7e00640 1 -- 172.21.15.102:0/2690010155 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0c8045070 con 0x7fb0d015ac20 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.644+0000 7fb0c7e00640 1 -- 172.21.15.102:0/2690010155 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c8009450 con 0x7fb0d015ac20 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.644+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d01f3ce0 con 0x7fb0d015ac20 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.644+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d01f4580 con 0x7fb0d015ac20 2024-03-18T16:22:26.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.646+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0d0074fd0 con 0x7fb0d015ac20 2024-03-18T16:22:26.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.910+0000 7fb0c7e00640 1 -- 172.21.15.102:0/2690010155 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb0c803e850 con 0x7fb0d015ac20 2024-03-18T16:22:26.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.911+0000 7fb0c7e00640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0b006bc20 0x7fb0b006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:26.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.911+0000 7fb0cfe00640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0b006bc20 0x7fb0b006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:26.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.912+0000 7fb0c7e00640 1 -- 172.21.15.102:0/2690010155 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fb0c807a790 con 0x7fb0d015ac20 2024-03-18T16:22:26.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.912+0000 7fb0cfe00640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0b006bc20 0x7fb0b006e0e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009b40 tx=0x7fb0c0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:26.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:26.923+0000 7fb0c7e00640 1 -- 172.21.15.102:0/2690010155 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb0c807c6f0 con 0x7fb0d015ac20 2024-03-18T16:22:27.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.161+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb0d008ca00 con 0x7fb0d015ac20 2024-03-18T16:22:27.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.170+0000 7fb0c7e00640 1 -- 172.21.15.102:0/2690010155 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) ==== 100+0+11 (secure 0 0 0) 0x7fb0c8080790 con 0x7fb0d015ac20 2024-03-18T16:22:27.172 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-18T16:22:27.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0b006bc20 msgr2=0x7fb0b006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:27.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0b006bc20 0x7fb0b006e0e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009b40 tx=0x7fb0c0009190 comp rx=0 tx=0).stop 2024-03-18T16:22:27.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d015ac20 msgr2=0x7fb0d00c9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:27.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d015ac20 0x7fb0d00c9f50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb0d00d0e80 tx=0x7fb0c8004a40 comp rx=0 tx=0).stop 2024-03-18T16:22:27.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 shutdown_connections 2024-03-18T16:22:27.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0b006bc20 0x7fb0b006e0e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0d015ac20 0x7fb0d00c9f50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0d0154520 0x7fb0d00c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 --2- 172.21.15.102:0/2690010155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0d00c5920 0x7fb0d014da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 >> 172.21.15.102:0/2690010155 conn(0x7fb0d006c9a0 msgr2=0x7fb0d0153340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:27.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.172+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 shutdown_connections 2024-03-18T16:22:27.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.173+0000 7fb0d5400640 1 -- 172.21.15.102:0/2690010155 wait complete. 2024-03-18T16:22:27.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:22:27.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.367+0000 7f528b200640 1 Processor -- start 2024-03-18T16:22:27.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.367+0000 7f528b200640 1 -- start start 2024-03-18T16:22:27.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.368+0000 7f528b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c154a00 0x7f528c0ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:27.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.368+0000 7f528b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f528c0cff10 0x7f528c0d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:27.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.368+0000 7f528b200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f528c15a4a0 con 0x7f528c154620 2024-03-18T16:22:27.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.368+0000 7f528b200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f528c0ced50 con 0x7f528c0cff10 2024-03-18T16:22:27.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.368+0000 7f528b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f528c15b370 con 0x7f528c154a00 2024-03-18T16:22:27.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.369+0000 7f5289e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f528c154620 0x7f528c0ca520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:49888/0 (socket says 172.21.15.102:49888) 2024-03-18T16:22:27.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.369+0000 7f5289e00640 1 -- 172.21.15.102:0/1453353559 learned_addr learned my addr 172.21.15.102:0/1453353559 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:27.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.369+0000 7f5289400640 1 --2- 172.21.15.102:0/1453353559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c154a00 0x7f528c0ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:27.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.372+0000 7f5289400640 1 -- 172.21.15.102:0/1453353559 >> v2:172.21.15.102:3300/0 conn(0x7f528c0cff10 msgr2=0x7f528c0d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:27.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.372+0000 7f5289400640 1 --2- 172.21.15.102:0/1453353559 >> v2:172.21.15.102:3300/0 conn(0x7f528c0cff10 0x7f528c0d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.372+0000 7f5289400640 1 -- 172.21.15.102:0/1453353559 >> v1:172.21.15.154:6789/0 conn(0x7f528c154620 legacy=0x7f528c0ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:27.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.372+0000 7f5289400640 1 -- 172.21.15.102:0/1453353559 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528c15a760 con 0x7f528c154a00 2024-03-18T16:22:27.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.374+0000 7f5289400640 1 --2- 172.21.15.102:0/1453353559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c154a00 0x7f528c0ce3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f528c15b4f0 tx=0x7f527802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10399804a9709118 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:27.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.378+0000 7f5288a00640 1 -- 172.21.15.102:0/1453353559 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278035020 con 0x7f528c154a00 2024-03-18T16:22:27.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.378+0000 7f5288a00640 1 -- 172.21.15.102:0/1453353559 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5278031890 con 0x7f528c154a00 2024-03-18T16:22:27.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.378+0000 7f5288a00640 1 -- 172.21.15.102:0/1453353559 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5278031c60 con 0x7f528c154a00 2024-03-18T16:22:27.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.379+0000 7f528b200640 1 -- 172.21.15.102:0/1453353559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c154a00 msgr2=0x7f528c0ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:27.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.379+0000 7f528b200640 1 --2- 172.21.15.102:0/1453353559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c154a00 0x7f528c0ce3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f528c15b4f0 tx=0x7f527802b930 comp rx=0 tx=0).stop 2024-03-18T16:22:27.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.379+0000 7f528b200640 1 -- 172.21.15.102:0/1453353559 shutdown_connections 2024-03-18T16:22:27.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.379+0000 7f528b200640 1 --2- 172.21.15.102:0/1453353559 >> v2:172.21.15.102:3300/0 conn(0x7f528c0cff10 0x7f528c0d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.379+0000 7f528b200640 1 --2- 172.21.15.102:0/1453353559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c154a00 0x7f528c0ce3c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.379+0000 7f528b200640 1 -- 172.21.15.102:0/1453353559 >> 172.21.15.102:0/1453353559 conn(0x7f528c067160 msgr2=0x7f528c067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:27.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.379+0000 7f528b200640 1 -- 172.21.15.102:0/1453353559 shutdown_connections 2024-03-18T16:22:27.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.380+0000 7f528b200640 1 -- 172.21.15.102:0/1453353559 wait complete. 2024-03-18T16:22:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.382+0000 7f528b200640 1 Processor -- start 2024-03-18T16:22:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.382+0000 7f528b200640 1 -- start start 2024-03-18T16:22:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.384+0000 7f528b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c0cff10 0x7f528c157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.384+0000 7f528b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f528c154620 0x7f528c155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.384+0000 7f528b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f528c154a00 0x7f528c1eff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.384+0000 7f528b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f528c15b000 con 0x7f528c154a00 2024-03-18T16:22:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.384+0000 7f528b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f528c15ad20 con 0x7f528c154620 2024-03-18T16:22:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.384+0000 7f528b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f528c06d4b0 con 0x7f528c0cff10 2024-03-18T16:22:27.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.387+0000 7f528a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f528c154a00 0x7f528c1eff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:27.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.387+0000 7f528a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f528c154a00 0x7f528c1eff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55684/0 (socket says 172.21.15.102:55684) 2024-03-18T16:22:27.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.387+0000 7f528a800640 1 -- 172.21.15.102:0/1932380293 learned_addr learned my addr 172.21.15.102:0/1932380293 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:27.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.390+0000 7f528a800640 1 -- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c0cff10 msgr2=0x7f528c157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:27.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.390+0000 7f528a800640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c0cff10 0x7f528c157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.390+0000 7f528a800640 1 -- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f528c154620 msgr2=0x7f528c155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:27.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.390+0000 7f528a800640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f528c154620 0x7f528c155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.390+0000 7f528a800640 1 -- 172.21.15.102:0/1932380293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528c1565c0 con 0x7f528c154a00 2024-03-18T16:22:27.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.393+0000 7f528a800640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f528c154a00 0x7f528c1eff80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f528400b9a0 tx=0x7f528400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.395+0000 7f5282000640 1 -- 172.21.15.102:0/1932380293 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284019070 con 0x7f528c154a00 2024-03-18T16:22:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.395+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f528c1f08a0 con 0x7f528c154a00 2024-03-18T16:22:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.395+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f528c156860 con 0x7f528c154a00 2024-03-18T16:22:27.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.397+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f528c0ceee0 con 0x7f528c154a00 2024-03-18T16:22:27.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.398+0000 7f5282000640 1 -- 172.21.15.102:0/1932380293 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f528400cba0 con 0x7f528c154a00 2024-03-18T16:22:27.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.398+0000 7f5282000640 1 -- 172.21.15.102:0/1932380293 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5284023060 con 0x7f528c154a00 2024-03-18T16:22:27.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.657+0000 7f5282000640 1 -- 172.21.15.102:0/1932380293 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f528401c070 con 0x7f528c154a00 2024-03-18T16:22:27.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.658+0000 7f5282000640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f526c06bbd0 0x7f526c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:27.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.659+0000 7f5289e00640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f526c06bbd0 0x7f526c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:27.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.660+0000 7f5289e00640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f526c06bbd0 0x7f526c06e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f527c000f60 tx=0x7f527c007a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:27.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.691+0000 7f5282000640 1 -- 172.21.15.102:0/1932380293 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f5284057140 con 0x7f528c154a00 2024-03-18T16:22:27.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.692+0000 7f5282000640 1 -- 172.21.15.102:0/1932380293 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f528c0ceee0 con 0x7f528c154a00 2024-03-18T16:22:27.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.885+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f528c0cc9d0 con 0x7f528c154a00 2024-03-18T16:22:27.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.894+0000 7f5282000640 1 -- 172.21.15.102:0/1932380293 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) ==== 103+0+11 (secure 0 0 0) 0x7f52840590a0 con 0x7f528c154a00 2024-03-18T16:22:27.896 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f526c06bbd0 msgr2=0x7f526c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f526c06bbd0 0x7f526c06e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f527c000f60 tx=0x7f527c007a80 comp rx=0 tx=0).stop 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f528c154a00 msgr2=0x7f528c1eff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f528c154a00 0x7f528c1eff80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f528400b9a0 tx=0x7f528400be60 comp rx=0 tx=0).stop 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 shutdown_connections 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f526c06bbd0 0x7f526c06e090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f528c154a00 0x7f528c1eff80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f528c154620 0x7f528c155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 --2- 172.21.15.102:0/1932380293 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f528c0cff10 0x7f528c157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 >> 172.21.15.102:0/1932380293 conn(0x7f528c067160 msgr2=0x7f528c0ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 shutdown_connections 2024-03-18T16:22:27.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:27.897+0000 7f528b200640 1 -- 172.21.15.102:0/1932380293 wait complete. 2024-03-18T16:22:27.916 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:22:28.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.083+0000 7fa465200640 1 Processor -- start 2024-03-18T16:22:28.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.083+0000 7fa465200640 1 -- start start 2024-03-18T16:22:28.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.084+0000 7fa465200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa4600c9be0 0x7fa4600d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.084+0000 7fa465200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 0x7fa4600cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.084+0000 7fa465200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa460154ab0 con 0x7fa46015ae70 2024-03-18T16:22:28.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.084+0000 7fa465200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa4600caba0 con 0x7fa4600c9be0 2024-03-18T16:22:28.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.084+0000 7fa465200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa46006d2f0 con 0x7fa4600d05a0 2024-03-18T16:22:28.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.085+0000 7fa45fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 0x7fa4600cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.085+0000 7fa45f400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa46015ae70 0x7fa4600ca490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:49892/0 (socket says 172.21.15.102:49892) 2024-03-18T16:22:28.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.085+0000 7fa45f400640 1 -- 172.21.15.102:0/1185679732 learned_addr learned my addr 172.21.15.102:0/1185679732 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:28.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.085+0000 7fa45ea00640 1 --2- 172.21.15.102:0/1185679732 >> v2:172.21.15.154:3300/0 conn(0x7fa4600c9be0 0x7fa4600d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.088+0000 7fa45fe00640 1 -- 172.21.15.102:0/1185679732 >> v2:172.21.15.154:3300/0 conn(0x7fa4600c9be0 msgr2=0x7fa4600d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:28.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.088+0000 7fa45fe00640 1 --2- 172.21.15.102:0/1185679732 >> v2:172.21.15.154:3300/0 conn(0x7fa4600c9be0 0x7fa4600d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.088+0000 7fa45fe00640 1 -- 172.21.15.102:0/1185679732 >> v1:172.21.15.154:6789/0 conn(0x7fa46015ae70 legacy=0x7fa4600ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.088+0000 7fa45fe00640 1 -- 172.21.15.102:0/1185679732 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46015b250 con 0x7fa4600d05a0 2024-03-18T16:22:28.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.091+0000 7fa45fe00640 1 --2- 172.21.15.102:0/1185679732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 0x7fa4600cc4f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa458009d90 tx=0x7fa45802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f346e02097313692 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:28.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.094+0000 7fa45e000640 1 -- 172.21.15.102:0/1185679732 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458039070 con 0x7fa4600d05a0 2024-03-18T16:22:28.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa45e000640 1 -- 172.21.15.102:0/1185679732 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa45802cc50 con 0x7fa4600d05a0 2024-03-18T16:22:28.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa45e000640 1 -- 172.21.15.102:0/1185679732 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458034440 con 0x7fa4600d05a0 2024-03-18T16:22:28.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa465200640 1 -- 172.21.15.102:0/1185679732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 msgr2=0x7fa4600cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa465200640 1 --2- 172.21.15.102:0/1185679732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 0x7fa4600cc4f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa458009d90 tx=0x7fa45802bea0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa465200640 1 -- 172.21.15.102:0/1185679732 shutdown_connections 2024-03-18T16:22:28.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa465200640 1 --2- 172.21.15.102:0/1185679732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 0x7fa4600cc4f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa465200640 1 --2- 172.21.15.102:0/1185679732 >> v2:172.21.15.154:3300/0 conn(0x7fa4600c9be0 0x7fa4600d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa465200640 1 -- 172.21.15.102:0/1185679732 >> 172.21.15.102:0/1185679732 conn(0x7fa4600673a0 msgr2=0x7fa4600677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:28.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa465200640 1 -- 172.21.15.102:0/1185679732 shutdown_connections 2024-03-18T16:22:28.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.095+0000 7fa465200640 1 -- 172.21.15.102:0/1185679732 wait complete. 2024-03-18T16:22:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.098+0000 7fa465200640 1 Processor -- start 2024-03-18T16:22:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.098+0000 7fa465200640 1 -- start start 2024-03-18T16:22:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.099+0000 7fa465200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4600c9be0 0x7fa4600b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.100+0000 7fa465200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 0x7fa4601efc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.100+0000 7fa465200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa46015ae70 0x7fa4600b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.100+0000 7fa465200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa46006db90 con 0x7fa4600c9be0 2024-03-18T16:22:28.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.100+0000 7fa465200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa46006d8b0 con 0x7fa46015ae70 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.100+0000 7fa465200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa46006de70 con 0x7fa4600d05a0 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.100+0000 7fa45fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa46015ae70 0x7fa4600b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.100+0000 7fa45ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 0x7fa4601efc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.100+0000 7fa45f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4600c9be0 0x7fa4600b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.100+0000 7fa45fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa46015ae70 0x7fa4600b3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54808/0 (socket says 172.21.15.102:54808) 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.101+0000 7fa45fe00640 1 -- 172.21.15.102:0/825382586 learned_addr learned my addr 172.21.15.102:0/825382586 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.104+0000 7fa45fe00640 1 -- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 msgr2=0x7fa4601efc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.104+0000 7fa45fe00640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 0x7fa4601efc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.104+0000 7fa45fe00640 1 -- 172.21.15.102:0/825382586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4600c9be0 msgr2=0x7fa4600b5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.104+0000 7fa45fe00640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4600c9be0 0x7fa4600b5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.104+0000 7fa45fe00640 1 -- 172.21.15.102:0/825382586 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4600b5560 con 0x7fa46015ae70 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.107+0000 7fa45fe00640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa46015ae70 0x7fa4600b3c70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa458009eb0 tx=0x7fa458004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.110+0000 7fa457400640 1 -- 172.21.15.102:0/825382586 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458044020 con 0x7fa46015ae70 2024-03-18T16:22:28.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.110+0000 7fa457400640 1 -- 172.21.15.102:0/825382586 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa45802ca50 con 0x7fa46015ae70 2024-03-18T16:22:28.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.110+0000 7fa457400640 1 -- 172.21.15.102:0/825382586 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458039040 con 0x7fa46015ae70 2024-03-18T16:22:28.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.110+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4600b45b0 con 0x7fa46015ae70 2024-03-18T16:22:28.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.111+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4600b4b30 con 0x7fa46015ae70 2024-03-18T16:22:28.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.112+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa460074fd0 con 0x7fa46015ae70 2024-03-18T16:22:28.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.133+0000 7fa457400640 1 -- 172.21.15.102:0/825382586 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa458047020 con 0x7fa46015ae70 2024-03-18T16:22:28.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.134+0000 7fa457400640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa44006bc20 0x7fa44006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.134+0000 7fa45f400640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa44006bc20 0x7fa44006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.135+0000 7fa45f400640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa44006bc20 0x7fa44006e0e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa450000f60 tx=0x7fa450007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:28.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.375+0000 7fa457400640 1 -- 172.21.15.102:0/825382586 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fa45802cc00 con 0x7fa46015ae70 2024-03-18T16:22:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.386+0000 7fa457400640 1 -- 172.21.15.102:0/825382586 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa45804e030 con 0x7fa46015ae70 2024-03-18T16:22:28.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.704+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4600b5a90 con 0x7fa46015ae70 2024-03-18T16:22:28.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.720+0000 7fa457400640 1 -- 172.21.15.102:0/825382586 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6924 (secure 0 0 0) 0x7fa45807ba60 con 0x7fa46015ae70 2024-03-18T16:22:28.722 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:28.722 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:22:20.541204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4250,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4250":{"gid":4250,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":17,"addr":"172.21.15.102:6834/3423054813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3423054813},{"type":"v1","addr":"172.21.15.102:6834","nonce":3423054813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4247,4250,4253,4256,4270]},"id":1}]} 2024-03-18T16:22:28.722 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 19 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.722+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa44006bc20 msgr2=0x7fa44006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.722+0000 7fa465200640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa44006bc20 0x7fa44006e0e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa450000f60 tx=0x7fa450007e80 comp rx=0 tx=0).stop 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.722+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa46015ae70 msgr2=0x7fa4600b3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.722+0000 7fa465200640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa46015ae70 0x7fa4600b3c70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa458009eb0 tx=0x7fa458004a40 comp rx=0 tx=0).stop 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.723+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 shutdown_connections 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.723+0000 7fa465200640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa44006bc20 0x7fa44006e0e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.723+0000 7fa465200640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa46015ae70 0x7fa4600b3c70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.723+0000 7fa465200640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4600d05a0 0x7fa4601efc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.723+0000 7fa465200640 1 --2- 172.21.15.102:0/825382586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4600c9be0 0x7fa4600b5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.723+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 >> 172.21.15.102:0/825382586 conn(0x7fa4600673a0 msgr2=0x7fa460158720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.723+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 shutdown_connections 2024-03-18T16:22:28.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.723+0000 7fa465200640 1 -- 172.21.15.102:0/825382586 wait complete. 2024-03-18T16:22:28.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:22:28.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.920+0000 7f64aa400640 1 Processor -- start 2024-03-18T16:22:28.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.920+0000 7f64aa400640 1 -- start start 2024-03-18T16:22:28.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.921+0000 7f64aa400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a40ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.921+0000 7f64aa400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a4154ab0 con 0x7f64a40cc4f0 2024-03-18T16:22:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.921+0000 7f64aa400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64a415b250 con 0x7f64a40c9be0 2024-03-18T16:22:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.921+0000 7f64aa400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f64a406d2f0 con 0x7f64a415ae70 2024-03-18T16:22:28.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.922+0000 7f64a3e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f64a40c9be0 0x7f64a40d0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:49906/0 (socket says 172.21.15.102:49906) 2024-03-18T16:22:28.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.922+0000 7f64a3e00640 1 -- 172.21.15.102:0/2620850021 learned_addr learned my addr 172.21.15.102:0/2620850021 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:28.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.922+0000 7f64a9000640 1 --2- 172.21.15.102:0/2620850021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a40ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.925+0000 7f64a3400640 1 -- 172.21.15.102:0/2620850021 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086188254 0 0) 0x7f64a415b250 con 0x7f64a40c9be0 2024-03-18T16:22:28.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.925+0000 7f64a3400640 1 -- 172.21.15.102:0/2620850021 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f648c003520 con 0x7f64a40c9be0 2024-03-18T16:22:28.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.925+0000 7f64a9000640 1 -- 172.21.15.102:0/2620850021 >> v1:172.21.15.154:6789/0 conn(0x7f64a40c9be0 legacy=0x7f64a40d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.925+0000 7f64a9000640 1 -- 172.21.15.102:0/2620850021 >> v1:172.21.15.102:6789/0 conn(0x7f64a40cc4f0 legacy=0x7f64a40cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.925+0000 7f64a9000640 1 -- 172.21.15.102:0/2620850021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a406d5d0 con 0x7f64a415ae70 2024-03-18T16:22:28.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.928+0000 7f64a9000640 1 --2- 172.21.15.102:0/2620850021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a40ca490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f64980097d0 tx=0x7f649802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aad54a3986a59d8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:28.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.930+0000 7f64a3400640 1 -- 172.21.15.102:0/2620850021 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498035020 con 0x7f64a415ae70 2024-03-18T16:22:28.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.932+0000 7f64a3400640 1 -- 172.21.15.102:0/2620850021 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6498031890 con 0x7f64a415ae70 2024-03-18T16:22:28.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.932+0000 7f64a3400640 1 -- 172.21.15.102:0/2620850021 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6498031c40 con 0x7f64a415ae70 2024-03-18T16:22:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.933+0000 7f64aa400640 1 -- 172.21.15.102:0/2620850021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 msgr2=0x7f64a40ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.933+0000 7f64aa400640 1 --2- 172.21.15.102:0/2620850021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a40ca490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f64980097d0 tx=0x7f649802b930 comp rx=0 tx=0).stop 2024-03-18T16:22:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.933+0000 7f64aa400640 1 -- 172.21.15.102:0/2620850021 shutdown_connections 2024-03-18T16:22:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.933+0000 7f64aa400640 1 --2- 172.21.15.102:0/2620850021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a40ca490 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.933+0000 7f64aa400640 1 -- 172.21.15.102:0/2620850021 >> 172.21.15.102:0/2620850021 conn(0x7f64a40673a0 msgr2=0x7f64a40677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.933+0000 7f64aa400640 1 -- 172.21.15.102:0/2620850021 shutdown_connections 2024-03-18T16:22:28.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.933+0000 7f64aa400640 1 -- 172.21.15.102:0/2620850021 wait complete. 2024-03-18T16:22:28.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.935+0000 7f64aa400640 1 Processor -- start 2024-03-18T16:22:28.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.935+0000 7f64aa400640 1 -- start start 2024-03-18T16:22:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.936+0000 7f64aa400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64a40c9be0 0x7f64a40c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.936+0000 7f64aa400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a40cc4f0 0x7f64a41efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.936+0000 7f64aa400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a4153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.936+0000 7f64aa400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f64a406de70 con 0x7f64a40c9be0 2024-03-18T16:22:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.936+0000 7f64aa400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f64a406db90 con 0x7f64a40cc4f0 2024-03-18T16:22:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.936+0000 7f64aa400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f64a406e150 con 0x7f64a415ae70 2024-03-18T16:22:28.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.937+0000 7f64a9a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a4153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.937+0000 7f64a3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a40cc4f0 0x7f64a41efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.937+0000 7f64a9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64a40c9be0 0x7f64a40c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.937+0000 7f64a9a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a4153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48164/0 (socket says 172.21.15.102:48164) 2024-03-18T16:22:28.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.937+0000 7f64a9a00640 1 -- 172.21.15.102:0/2595201288 learned_addr learned my addr 172.21.15.102:0/2595201288 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.940+0000 7f64a9a00640 1 -- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a40cc4f0 msgr2=0x7f64a41efe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.940+0000 7f64a9a00640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a40cc4f0 0x7f64a41efe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.940+0000 7f64a9a00640 1 -- 172.21.15.102:0/2595201288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64a40c9be0 msgr2=0x7f64a40c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.940+0000 7f64a9a00640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64a40c9be0 0x7f64a40c80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:28.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.940+0000 7f64a9a00640 1 -- 172.21.15.102:0/2595201288 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64a41f3410 con 0x7f64a415ae70 2024-03-18T16:22:28.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.942+0000 7f64a9a00640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a4153910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c7a0 tx=0x7f649c00cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.946+0000 7f64a0c00640 1 -- 172.21.15.102:0/2595201288 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c003b70 con 0x7f64a415ae70 2024-03-18T16:22:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.946+0000 7f64a0c00640 1 -- 172.21.15.102:0/2595201288 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f649c003d30 con 0x7f64a415ae70 2024-03-18T16:22:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.946+0000 7f64a0c00640 1 -- 172.21.15.102:0/2595201288 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c00aa90 con 0x7f64a415ae70 2024-03-18T16:22:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.946+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64a40b5090 con 0x7f64a415ae70 2024-03-18T16:22:28.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.946+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64a41f3680 con 0x7f64a415ae70 2024-03-18T16:22:28.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.947+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64a4074fd0 con 0x7f64a415ae70 2024-03-18T16:22:28.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.967+0000 7f64a0c00640 1 -- 172.21.15.102:0/2595201288 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f649c00ac30 con 0x7f64a415ae70 2024-03-18T16:22:28.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.968+0000 7f64a0c00640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f648406bc20 0x7f648406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:28.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.969+0000 7f64a9000640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f648406bc20 0x7f648406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:28.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.969+0000 7f64a0c00640 1 -- 172.21.15.102:0/2595201288 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f649c0578d0 con 0x7f64a415ae70 2024-03-18T16:22:28.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:28.969+0000 7f64a9000640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f648406bc20 0x7f648406e0e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6498036000 tx=0x7f6498008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.260+0000 7f64a0c00640 1 -- 172.21.15.102:0/2595201288 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f649c059830 con 0x7f64a415ae70 2024-03-18T16:22:29.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.465+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64a404f5d0 con 0x7f64a415ae70 2024-03-18T16:22:29.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.560+0000 7f64a0c00640 1 -- 172.21.15.102:0/2595201288 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18907 (secure 0 0 0) 0x7f649c05d8d0 con 0x7f64a415ae70 2024-03-18T16:22:29.562 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:29.563 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":29,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:54.749120+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.563+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f648406bc20 msgr2=0x7f648406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.563+0000 7f64aa400640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f648406bc20 0x7f648406e0e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6498036000 tx=0x7f6498008030 comp rx=0 tx=0).stop 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 msgr2=0x7f64a4153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a4153910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f649c00c7a0 tx=0x7f649c00cc60 comp rx=0 tx=0).stop 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 shutdown_connections 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f648406bc20 0x7f648406e0e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f64a415ae70 0x7f64a4153910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64a40cc4f0 0x7f64a41efe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 --2- 172.21.15.102:0/2595201288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64a40c9be0 0x7f64a40c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 >> 172.21.15.102:0/2595201288 conn(0x7f64a40673a0 msgr2=0x7f64a40cdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 shutdown_connections 2024-03-18T16:22:29.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.564+0000 7f64aa400640 1 -- 172.21.15.102:0/2595201288 wait complete. 2024-03-18T16:22:29.588 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:21:05.204661+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:21:19.524753+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:21:21.031843+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:21:26.818690+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:22:29.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:22:29.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.771+0000 7f6c46800640 1 Processor -- start 2024-03-18T16:22:29.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.771+0000 7f6c46800640 1 -- start start 2024-03-18T16:22:29.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.772+0000 7f6c46800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6c40154620 0x7f6c400ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.772+0000 7f6c46800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6c40154a00 0x7f6c400ca520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.772+0000 7f6c46800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c4015a4a0 con 0x7f6c400cff10 2024-03-18T16:22:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.772+0000 7f6c46800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6c400ced50 con 0x7f6c40154a00 2024-03-18T16:22:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.772+0000 7f6c46800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6c4015b370 con 0x7f6c40154620 2024-03-18T16:22:29.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.773+0000 7f6c45400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6c40154620 0x7f6c400ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:29.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.773+0000 7f6c44a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6c40154a00 0x7f6c400ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:29.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.773+0000 7f6c45e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f6c400cff10 0x7f6c4006d4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:49908/0 (socket says 172.21.15.102:49908) 2024-03-18T16:22:29.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.773+0000 7f6c45e00640 1 -- 172.21.15.102:0/813935581 learned_addr learned my addr 172.21.15.102:0/813935581 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.776+0000 7f6c45400640 1 -- 172.21.15.102:0/813935581 >> v2:172.21.15.102:3300/0 conn(0x7f6c40154a00 msgr2=0x7f6c400ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.776+0000 7f6c45400640 1 --2- 172.21.15.102:0/813935581 >> v2:172.21.15.102:3300/0 conn(0x7f6c40154a00 0x7f6c400ca520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.776+0000 7f6c45400640 1 -- 172.21.15.102:0/813935581 >> v1:172.21.15.154:6789/0 conn(0x7f6c400cff10 legacy=0x7f6c4006d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.776+0000 7f6c45400640 1 -- 172.21.15.102:0/813935581 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c4015a760 con 0x7f6c40154620 2024-03-18T16:22:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.780+0000 7f6c45400640 1 --2- 172.21.15.102:0/813935581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154620 0x7f6c400ce3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6c40068f40 tx=0x7f6c3402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0ad53daa405beaa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.784+0000 7f6c3fe00640 1 -- 172.21.15.102:0/813935581 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c34034070 con 0x7f6c40154620 2024-03-18T16:22:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.784+0000 7f6c3fe00640 1 -- 172.21.15.102:0/813935581 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c34030610 con 0x7f6c40154620 2024-03-18T16:22:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.784+0000 7f6c3fe00640 1 -- 172.21.15.102:0/813935581 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c340309e0 con 0x7f6c40154620 2024-03-18T16:22:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.785+0000 7f6c46800640 1 -- 172.21.15.102:0/813935581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154620 msgr2=0x7f6c400ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.785+0000 7f6c46800640 1 --2- 172.21.15.102:0/813935581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154620 0x7f6c400ce3c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6c40068f40 tx=0x7f6c3402b960 comp rx=0 tx=0).stop 2024-03-18T16:22:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.785+0000 7f6c46800640 1 -- 172.21.15.102:0/813935581 shutdown_connections 2024-03-18T16:22:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.785+0000 7f6c46800640 1 --2- 172.21.15.102:0/813935581 >> v2:172.21.15.102:3300/0 conn(0x7f6c40154a00 0x7f6c400ca520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.785+0000 7f6c46800640 1 --2- 172.21.15.102:0/813935581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154620 0x7f6c400ce3c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.785+0000 7f6c46800640 1 -- 172.21.15.102:0/813935581 >> 172.21.15.102:0/813935581 conn(0x7f6c40067160 msgr2=0x7f6c40067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.785+0000 7f6c46800640 1 -- 172.21.15.102:0/813935581 shutdown_connections 2024-03-18T16:22:29.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.785+0000 7f6c46800640 1 -- 172.21.15.102:0/813935581 wait complete. 2024-03-18T16:22:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.787+0000 7f6c46800640 1 Processor -- start 2024-03-18T16:22:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.787+0000 7f6c46800640 1 -- start start 2024-03-18T16:22:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.788+0000 7f6c46800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c400cff10 0x7f6c4014d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.788+0000 7f6c46800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c40154620 0x7f6c400b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.788+0000 7f6c46800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154a00 0x7f6c400b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.788+0000 7f6c46800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4015b000 con 0x7f6c400cff10 2024-03-18T16:22:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.788+0000 7f6c46800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4015ad20 con 0x7f6c40154a00 2024-03-18T16:22:29.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.788+0000 7f6c46800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6c400caa60 con 0x7f6c40154620 2024-03-18T16:22:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.789+0000 7f6c45e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154a00 0x7f6c400b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.789+0000 7f6c45e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154a00 0x7f6c400b1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54830/0 (socket says 172.21.15.102:54830) 2024-03-18T16:22:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.789+0000 7f6c45e00640 1 -- 172.21.15.102:0/3421857862 learned_addr learned my addr 172.21.15.102:0/3421857862 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.789+0000 7f6c44a00640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c40154620 0x7f6c400b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:29.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.789+0000 7f6c45400640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c400cff10 0x7f6c4014d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:29.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.792+0000 7f6c45e00640 1 -- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c40154620 msgr2=0x7f6c400b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.792+0000 7f6c45e00640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c40154620 0x7f6c400b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.792+0000 7f6c45e00640 1 -- 172.21.15.102:0/3421857862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c400cff10 msgr2=0x7f6c4014d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.792+0000 7f6c45e00640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c400cff10 0x7f6c4014d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:29.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.792+0000 7f6c45e00640 1 -- 172.21.15.102:0/3421857862 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c400c8dc0 con 0x7f6c40154a00 2024-03-18T16:22:29.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.795+0000 7f6c45e00640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154a00 0x7f6c400b1810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800cad0 tx=0x7f6c3800cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.798+0000 7f6c3d600640 1 -- 172.21.15.102:0/3421857862 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38017070 con 0x7f6c40154a00 2024-03-18T16:22:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.798+0000 7f6c3d600640 1 -- 172.21.15.102:0/3421857862 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3800ac30 con 0x7f6c40154a00 2024-03-18T16:22:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.798+0000 7f6c3d600640 1 -- 172.21.15.102:0/3421857862 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38012620 con 0x7f6c40154a00 2024-03-18T16:22:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.798+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c400c90a0 con 0x7f6c40154a00 2024-03-18T16:22:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.799+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c400b0af0 con 0x7f6c40154a00 2024-03-18T16:22:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.800+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c40074fd0 con 0x7f6c40154a00 2024-03-18T16:22:29.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.819+0000 7f6c3d600640 1 -- 172.21.15.102:0/3421857862 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6c3801a070 con 0x7f6c40154a00 2024-03-18T16:22:29.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.821+0000 7f6c3d600640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6c2006bc20 0x7f6c2006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:29.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.821+0000 7f6c3d600640 1 -- 172.21.15.102:0/3421857862 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f6c380570c0 con 0x7f6c40154a00 2024-03-18T16:22:29.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.821+0000 7f6c45400640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6c2006bc20 0x7f6c2006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:29.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:29.822+0000 7f6c45400640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6c2006bc20 0x7f6c2006e0e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c34036000 tx=0x7f6c3403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:30.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.075+0000 7f6c3d600640 1 -- 172.21.15.102:0/3421857862 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6c38021020 con 0x7f6c40154a00 2024-03-18T16:22:30.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.278+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6c400b5b20 con 0x7f6c40154a00 2024-03-18T16:22:30.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.284+0000 7f6c3d600640 1 -- 172.21.15.102:0/3421857862 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f6c3800ae20 con 0x7f6c40154a00 2024-03-18T16:22:30.285 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-18T16:22:30.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6c2006bc20 msgr2=0x7f6c2006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:30.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6c2006bc20 0x7f6c2006e0e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6c34036000 tx=0x7f6c3403b040 comp rx=0 tx=0).stop 2024-03-18T16:22:30.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154a00 msgr2=0x7f6c400b1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:30.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154a00 0x7f6c400b1810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800cad0 tx=0x7f6c3800cf90 comp rx=0 tx=0).stop 2024-03-18T16:22:30.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 shutdown_connections 2024-03-18T16:22:30.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6c2006bc20 0x7f6c2006e0e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c40154a00 0x7f6c400b1810 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6c40154620 0x7f6c400b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 --2- 172.21.15.102:0/3421857862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c400cff10 0x7f6c4014d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 >> 172.21.15.102:0/3421857862 conn(0x7f6c40067160 msgr2=0x7f6c400cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:30.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 shutdown_connections 2024-03-18T16:22:30.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.286+0000 7f6c46800640 1 -- 172.21.15.102:0/3421857862 wait complete. 2024-03-18T16:22:30.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:22:30.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.477+0000 7fd3e9c00640 1 Processor -- start 2024-03-18T16:22:30.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.477+0000 7fd3e9c00640 1 -- start start 2024-03-18T16:22:30.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.479+0000 7fd3e9c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd3e4158970 0x7fd3e40717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:30.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.479+0000 7fd3e9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 0x7fd3e406d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:30.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.479+0000 7fd3e9c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3e40c6a20 con 0x7fd3e4161c90 2024-03-18T16:22:30.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.479+0000 7fd3e9c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd3e40c5de0 con 0x7fd3e4158970 2024-03-18T16:22:30.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.479+0000 7fd3e9c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e40c7910 con 0x7fd3e40c7da0 2024-03-18T16:22:30.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.526+0000 7fd3e9200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd3e4161c90 0x7fd3e4162120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36176/0 (socket says 172.21.15.102:36176) 2024-03-18T16:22:30.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.526+0000 7fd3e9200640 1 -- 172.21.15.102:0/1122750849 learned_addr learned my addr 172.21.15.102:0/1122750849 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:30.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.526+0000 7fd3e3e00640 1 --2- 172.21.15.102:0/1122750849 >> v2:172.21.15.102:3300/0 conn(0x7fd3e4158970 0x7fd3e40717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:30.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.528+0000 7fd3e3400640 1 --2- 172.21.15.102:0/1122750849 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 0x7fd3e406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:30.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.531+0000 7fd3e2a00640 1 -- 172.21.15.102:0/1122750849 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3701081150 0 0) 0x7fd3e40c6a20 con 0x7fd3e4161c90 2024-03-18T16:22:30.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.531+0000 7fd3e2a00640 1 -- 172.21.15.102:0/1122750849 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3c8003520 con 0x7fd3e4161c90 2024-03-18T16:22:30.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.533+0000 7fd3e3e00640 1 -- 172.21.15.102:0/1122750849 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 msgr2=0x7fd3e406d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:30.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.533+0000 7fd3e3e00640 1 --2- 172.21.15.102:0/1122750849 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 0x7fd3e406d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.533+0000 7fd3e3e00640 1 -- 172.21.15.102:0/1122750849 >> v1:172.21.15.102:6789/0 conn(0x7fd3e4161c90 legacy=0x7fd3e4162120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:30.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.533+0000 7fd3e3e00640 1 -- 172.21.15.102:0/1122750849 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e40c6d00 con 0x7fd3e4158970 2024-03-18T16:22:30.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.538+0000 7fd3e3e00640 1 --2- 172.21.15.102:0/1122750849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e4158970 0x7fd3e40717d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40097a0 tx=0x7fd3d402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a6d98f2d93e62 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:30.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.547+0000 7fd3e2a00640 1 -- 172.21.15.102:0/1122750849 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4035020 con 0x7fd3e4158970 2024-03-18T16:22:30.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.547+0000 7fd3e2a00640 1 -- 172.21.15.102:0/1122750849 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d4031810 con 0x7fd3e4158970 2024-03-18T16:22:30.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.547+0000 7fd3e2a00640 1 -- 172.21.15.102:0/1122750849 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d4031be0 con 0x7fd3e4158970 2024-03-18T16:22:30.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.548+0000 7fd3e9c00640 1 -- 172.21.15.102:0/1122750849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e4158970 msgr2=0x7fd3e40717d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:30.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.548+0000 7fd3e9c00640 1 --2- 172.21.15.102:0/1122750849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e4158970 0x7fd3e40717d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40097a0 tx=0x7fd3d402b8b0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.548+0000 7fd3e9c00640 1 -- 172.21.15.102:0/1122750849 shutdown_connections 2024-03-18T16:22:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.548+0000 7fd3e9c00640 1 --2- 172.21.15.102:0/1122750849 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 0x7fd3e406d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.548+0000 7fd3e9c00640 1 --2- 172.21.15.102:0/1122750849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e4158970 0x7fd3e40717d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.548+0000 7fd3e9c00640 1 -- 172.21.15.102:0/1122750849 >> 172.21.15.102:0/1122750849 conn(0x7fd3e406b200 msgr2=0x7fd3e406b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.549+0000 7fd3e9c00640 1 -- 172.21.15.102:0/1122750849 shutdown_connections 2024-03-18T16:22:30.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.549+0000 7fd3e9c00640 1 -- 172.21.15.102:0/1122750849 wait complete. 2024-03-18T16:22:30.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.551+0000 7fd3e9c00640 1 Processor -- start 2024-03-18T16:22:30.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.551+0000 7fd3e9c00640 1 -- start start 2024-03-18T16:22:30.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.552+0000 7fd3e9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 0x7fd3e40c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:30.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.552+0000 7fd3e9c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e4158970 0x7fd3e41f7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:30.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.552+0000 7fd3e9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e4161c90 0x7fd3e406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:30.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.552+0000 7fd3e9c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e40c75a0 con 0x7fd3e4161c90 2024-03-18T16:22:30.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.552+0000 7fd3e9c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e40c72c0 con 0x7fd3e4158970 2024-03-18T16:22:30.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.552+0000 7fd3e9c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd3e4158d50 con 0x7fd3e40c7da0 2024-03-18T16:22:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.552+0000 7fd3e3400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e4158970 0x7fd3e41f7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.552+0000 7fd3e9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e4161c90 0x7fd3e406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.552+0000 7fd3e3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 0x7fd3e40c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.553+0000 7fd3e3400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e4158970 0x7fd3e41f7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54844/0 (socket says 172.21.15.102:54844) 2024-03-18T16:22:30.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.553+0000 7fd3e3400640 1 -- 172.21.15.102:0/3108136578 learned_addr learned my addr 172.21.15.102:0/3108136578 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.556+0000 7fd3e3400640 1 -- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 msgr2=0x7fd3e40c21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.556+0000 7fd3e3400640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 0x7fd3e40c21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.556+0000 7fd3e3400640 1 -- 172.21.15.102:0/3108136578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e4161c90 msgr2=0x7fd3e406f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.556+0000 7fd3e3400640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e4161c90 0x7fd3e406f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.556+0000 7fd3e3400640 1 -- 172.21.15.102:0/3108136578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e41f41f0 con 0x7fd3e4158970 2024-03-18T16:22:30.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.559+0000 7fd3e3400640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e4158970 0x7fd3e41f7fd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000d800 tx=0x7fd3d000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:30.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.563+0000 7fd3dbe00640 1 -- 172.21.15.102:0/3108136578 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d000b6c0 con 0x7fd3e4158970 2024-03-18T16:22:30.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.563+0000 7fd3dbe00640 1 -- 172.21.15.102:0/3108136578 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3d0019070 con 0x7fd3e4158970 2024-03-18T16:22:30.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.563+0000 7fd3dbe00640 1 -- 172.21.15.102:0/3108136578 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d000ba80 con 0x7fd3e4158970 2024-03-18T16:22:30.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.563+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e41f44d0 con 0x7fd3e4158970 2024-03-18T16:22:30.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.563+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e41f4a90 con 0x7fd3e4158970 2024-03-18T16:22:30.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.565+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3e40cb010 con 0x7fd3e4158970 2024-03-18T16:22:30.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.823+0000 7fd3dbe00640 1 -- 172.21.15.102:0/3108136578 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd3d000bc20 con 0x7fd3e4158970 2024-03-18T16:22:30.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.825+0000 7fd3dbe00640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd3c406bc20 0x7fd3c406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:30.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.825+0000 7fd3e3e00640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd3c406bc20 0x7fd3c406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:30.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.825+0000 7fd3dbe00640 1 -- 172.21.15.102:0/3108136578 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fd3d0057e00 con 0x7fd3e4158970 2024-03-18T16:22:30.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.826+0000 7fd3e3e00640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd3c406bc20 0x7fd3c406e0e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4036000 tx=0x7fd3d4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:30.849+0000 7fd3dbe00640 1 -- 172.21.15.102:0/3108136578 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd3e40cb010 con 0x7fd3e4158970 2024-03-18T16:22:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.044+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd3e4074fd0 con 0x7fd3e4158970 2024-03-18T16:22:31.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.049+0000 7fd3dbe00640 1 -- 172.21.15.102:0/3108136578 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v29) ==== 104+0+11 (secure 0 0 0) 0x7fd3d0059d60 con 0x7fd3e4158970 2024-03-18T16:22:31.051 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-18T16:22:31.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.051+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd3c406bc20 msgr2=0x7fd3c406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.051+0000 7fd3e9c00640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd3c406bc20 0x7fd3c406e0e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4036000 tx=0x7fd3d4008030 comp rx=0 tx=0).stop 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.051+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e4158970 msgr2=0x7fd3e41f7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.051+0000 7fd3e9c00640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e4158970 0x7fd3e41f7fd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000d800 tx=0x7fd3d000dcc0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.052+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 shutdown_connections 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.052+0000 7fd3e9c00640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd3c406bc20 0x7fd3c406e0e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.052+0000 7fd3e9c00640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3e4161c90 0x7fd3e406f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.052+0000 7fd3e9c00640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3e4158970 0x7fd3e41f7fd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.052+0000 7fd3e9c00640 1 --2- 172.21.15.102:0/3108136578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3e40c7da0 0x7fd3e40c21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.052+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 >> 172.21.15.102:0/3108136578 conn(0x7fd3e406b200 msgr2=0x7fd3e4157d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.052+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 shutdown_connections 2024-03-18T16:22:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.052+0000 7fd3e9c00640 1 -- 172.21.15.102:0/3108136578 wait complete. 2024-03-18T16:22:31.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:22:31.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.281+0000 7efd64a00640 1 Processor -- start 2024-03-18T16:22:31.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.281+0000 7efd64a00640 1 -- start start 2024-03-18T16:22:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.282+0000 7efd64a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efd60154620 0x7efd600ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.282+0000 7efd64a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd6015a4a0 con 0x7efd60154a00 2024-03-18T16:22:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.282+0000 7efd64a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd6015a760 con 0x7efd600cff10 2024-03-18T16:22:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.282+0000 7efd64a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7efd600ced50 con 0x7efd60154620 2024-03-18T16:22:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.283+0000 7efd5fe00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7efd600cff10 0x7efd6006d4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:48414/0 (socket says 172.21.15.102:48414) 2024-03-18T16:22:31.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.283+0000 7efd5fe00640 1 -- 172.21.15.102:0/171949734 learned_addr learned my addr 172.21.15.102:0/171949734 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:31.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.283+0000 7efd5f400640 1 --2- 172.21.15.102:0/171949734 >> v2:172.21.15.102:3300/0 conn(0x7efd60154620 0x7efd600ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.285+0000 7efd5e000640 1 -- 172.21.15.102:0/171949734 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2016618115 0 0) 0x7efd6015a760 con 0x7efd600cff10 2024-03-18T16:22:31.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.285+0000 7efd5e000640 1 -- 172.21.15.102:0/171949734 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd44003520 con 0x7efd600cff10 2024-03-18T16:22:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.288+0000 7efd5f400640 1 -- 172.21.15.102:0/171949734 >> v1:172.21.15.154:6789/0 conn(0x7efd600cff10 legacy=0x7efd6006d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.288+0000 7efd5f400640 1 -- 172.21.15.102:0/171949734 >> v1:172.21.15.102:6789/0 conn(0x7efd60154a00 legacy=0x7efd600ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.288+0000 7efd5f400640 1 -- 172.21.15.102:0/171949734 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6015aa40 con 0x7efd60154620 2024-03-18T16:22:31.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.294+0000 7efd5f400640 1 --2- 172.21.15.102:0/171949734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd60154620 0x7efd600ce3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd500097a0 tx=0x7efd5002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f525a74921a21bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:31.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.301+0000 7efd5e000640 1 -- 172.21.15.102:0/171949734 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5002f810 con 0x7efd60154620 2024-03-18T16:22:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.301+0000 7efd5e000640 1 -- 172.21.15.102:0/171949734 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd5002f9b0 con 0x7efd60154620 2024-03-18T16:22:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.301+0000 7efd5e000640 1 -- 172.21.15.102:0/171949734 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5002fd80 con 0x7efd60154620 2024-03-18T16:22:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.301+0000 7efd64a00640 1 -- 172.21.15.102:0/171949734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd60154620 msgr2=0x7efd600ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.301+0000 7efd64a00640 1 --2- 172.21.15.102:0/171949734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd60154620 0x7efd600ce3c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efd500097a0 tx=0x7efd5002b9f0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.302+0000 7efd64a00640 1 -- 172.21.15.102:0/171949734 shutdown_connections 2024-03-18T16:22:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.302+0000 7efd64a00640 1 --2- 172.21.15.102:0/171949734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd60154620 0x7efd600ce3c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.302+0000 7efd64a00640 1 -- 172.21.15.102:0/171949734 >> 172.21.15.102:0/171949734 conn(0x7efd60067160 msgr2=0x7efd60067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.302+0000 7efd64a00640 1 -- 172.21.15.102:0/171949734 shutdown_connections 2024-03-18T16:22:31.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.302+0000 7efd64a00640 1 -- 172.21.15.102:0/171949734 wait complete. 2024-03-18T16:22:31.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.304+0000 7efd64a00640 1 Processor -- start 2024-03-18T16:22:31.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.305+0000 7efd64a00640 1 -- start start 2024-03-18T16:22:31.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.306+0000 7efd64a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd600cff10 0x7efd6014ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:31.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.306+0000 7efd64a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60154620 0x7efd6015bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:31.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.306+0000 7efd64a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd60154a00 0x7efd6015c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:31.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.306+0000 7efd64a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efd6015b000 con 0x7efd600cff10 2024-03-18T16:22:31.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.306+0000 7efd64a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd6015b370 con 0x7efd60154a00 2024-03-18T16:22:31.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.306+0000 7efd64a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd6006dbc0 con 0x7efd60154620 2024-03-18T16:22:31.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.307+0000 7efd5fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd60154a00 0x7efd6015c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:31.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.307+0000 7efd5ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60154620 0x7efd6015bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:31.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.307+0000 7efd5f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd600cff10 0x7efd6014ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:31.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.307+0000 7efd5fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd60154a00 0x7efd6015c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40038/0 (socket says 172.21.15.102:40038) 2024-03-18T16:22:31.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.307+0000 7efd5fe00640 1 -- 172.21.15.102:0/2872369059 learned_addr learned my addr 172.21.15.102:0/2872369059 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:31.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.310+0000 7efd5fe00640 1 -- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60154620 msgr2=0x7efd6015bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:31.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.310+0000 7efd5fe00640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60154620 0x7efd6015bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.310+0000 7efd5fe00640 1 -- 172.21.15.102:0/2872369059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd600cff10 msgr2=0x7efd6014ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:31.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.310+0000 7efd5fe00640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd600cff10 0x7efd6014ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.310+0000 7efd5fe00640 1 -- 172.21.15.102:0/2872369059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6015b600 con 0x7efd60154a00 2024-03-18T16:22:31.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.313+0000 7efd5fe00640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd60154a00 0x7efd6015c180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efd5800cb70 tx=0x7efd5800a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:31.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.316+0000 7efd57400640 1 -- 172.21.15.102:0/2872369059 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58017070 con 0x7efd60154a00 2024-03-18T16:22:31.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.316+0000 7efd57400640 1 -- 172.21.15.102:0/2872369059 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd5800ad40 con 0x7efd60154a00 2024-03-18T16:22:31.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.316+0000 7efd57400640 1 -- 172.21.15.102:0/2872369059 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58012620 con 0x7efd60154a00 2024-03-18T16:22:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.316+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd600b2990 con 0x7efd60154a00 2024-03-18T16:22:31.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.317+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd600b2f50 con 0x7efd60154a00 2024-03-18T16:22:31.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.318+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd60074fd0 con 0x7efd60154a00 2024-03-18T16:22:31.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.337+0000 7efd57400640 1 -- 172.21.15.102:0/2872369059 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efd5801a070 con 0x7efd60154a00 2024-03-18T16:22:31.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.338+0000 7efd57400640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd4006bc50 0x7efd4006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:31.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.339+0000 7efd5f400640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd4006bc50 0x7efd4006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:31.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.340+0000 7efd5f400640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd4006bc50 0x7efd4006e110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efd50036000 tx=0x7efd50008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:31.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.341+0000 7efd57400640 1 -- 172.21.15.102:0/2872369059 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7efd58057140 con 0x7efd60154a00 2024-03-18T16:22:31.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.593+0000 7efd57400640 1 -- 172.21.15.102:0/2872369059 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efd58021020 con 0x7efd60154a00 2024-03-18T16:22:31.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.791+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efd600b5b20 con 0x7efd60154a00 2024-03-18T16:22:31.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.850+0000 7efd57400640 1 -- 172.21.15.102:0/2872369059 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v29) ==== 100+0+11 (secure 0 0 0) 0x7efd580590a0 con 0x7efd60154a00 2024-03-18T16:22:31.852 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.853+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd4006bc50 msgr2=0x7efd4006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.853+0000 7efd64a00640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd4006bc50 0x7efd4006e110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efd50036000 tx=0x7efd50008030 comp rx=0 tx=0).stop 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.853+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd60154a00 msgr2=0x7efd6015c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.853+0000 7efd64a00640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd60154a00 0x7efd6015c180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efd5800cb70 tx=0x7efd5800a430 comp rx=0 tx=0).stop 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.853+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 shutdown_connections 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.853+0000 7efd64a00640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd4006bc50 0x7efd4006e110 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.853+0000 7efd64a00640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd60154a00 0x7efd6015c180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.853+0000 7efd64a00640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd60154620 0x7efd6015bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.853+0000 7efd64a00640 1 --2- 172.21.15.102:0/2872369059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd600cff10 0x7efd6014ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.854+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 >> 172.21.15.102:0/2872369059 conn(0x7efd60067160 msgr2=0x7efd6006d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.854+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 shutdown_connections 2024-03-18T16:22:31.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:31.854+0000 7efd64a00640 1 -- 172.21.15.102:0/2872369059 wait complete. 2024-03-18T16:22:31.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:22:32.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.050+0000 7ff9cbe00640 1 Processor -- start 2024-03-18T16:22:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.051+0000 7ff9cbe00640 1 -- start start 2024-03-18T16:22:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.052+0000 7ff9cbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff9cc154620 0x7ff9cc0ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.052+0000 7ff9cbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9cc0cff10 0x7ff9cc0d0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.052+0000 7ff9cbe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9cc15a4a0 con 0x7ff9cc154a00 2024-03-18T16:22:32.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.052+0000 7ff9cbe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff9cc0ced50 con 0x7ff9cc0cff10 2024-03-18T16:22:32.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.052+0000 7ff9cbe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff9cc15b370 con 0x7ff9cc154620 2024-03-18T16:22:32.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.052+0000 7ff9cb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9cc0cff10 0x7ff9cc0d0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.053+0000 7ff9caa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff9cc154620 0x7ff9cc0ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.053+0000 7ff9ca000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff9cc154a00 0x7ff9cc0ca520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:48428/0 (socket says 172.21.15.102:48428) 2024-03-18T16:22:32.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.053+0000 7ff9ca000640 1 -- 172.21.15.102:0/3114071269 learned_addr learned my addr 172.21.15.102:0/3114071269 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:32.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.055+0000 7ff9cb400640 1 -- 172.21.15.102:0/3114071269 >> v2:172.21.15.154:3300/0 conn(0x7ff9cc154620 msgr2=0x7ff9cc0ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:32.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.055+0000 7ff9cb400640 1 --2- 172.21.15.102:0/3114071269 >> v2:172.21.15.154:3300/0 conn(0x7ff9cc154620 0x7ff9cc0ce3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.055+0000 7ff9cb400640 1 -- 172.21.15.102:0/3114071269 >> v1:172.21.15.154:6789/0 conn(0x7ff9cc154a00 legacy=0x7ff9cc0ca520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:32.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.055+0000 7ff9cb400640 1 -- 172.21.15.102:0/3114071269 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9cc15a760 con 0x7ff9cc0cff10 2024-03-18T16:22:32.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.058+0000 7ff9cb400640 1 --2- 172.21.15.102:0/3114071269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9cc0cff10 0x7ff9cc0d0360 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc068f40 tx=0x7ff9c402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a9b1dd77121d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:32.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.063+0000 7ff9c9600640 1 -- 172.21.15.102:0/3114071269 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4039070 con 0x7ff9cc0cff10 2024-03-18T16:22:32.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.063+0000 7ff9c9600640 1 -- 172.21.15.102:0/3114071269 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c402cd00 con 0x7ff9cc0cff10 2024-03-18T16:22:32.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.063+0000 7ff9c9600640 1 -- 172.21.15.102:0/3114071269 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4034660 con 0x7ff9cc0cff10 2024-03-18T16:22:32.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.064+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3114071269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9cc0cff10 msgr2=0x7ff9cc0d0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:32.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.064+0000 7ff9cbe00640 1 --2- 172.21.15.102:0/3114071269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9cc0cff10 0x7ff9cc0d0360 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc068f40 tx=0x7ff9c402bf50 comp rx=0 tx=0).stop 2024-03-18T16:22:32.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.064+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3114071269 shutdown_connections 2024-03-18T16:22:32.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.064+0000 7ff9cbe00640 1 --2- 172.21.15.102:0/3114071269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9cc0cff10 0x7ff9cc0d0360 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.064+0000 7ff9cbe00640 1 --2- 172.21.15.102:0/3114071269 >> v2:172.21.15.154:3300/0 conn(0x7ff9cc154620 0x7ff9cc0ce3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.064+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3114071269 >> 172.21.15.102:0/3114071269 conn(0x7ff9cc067160 msgr2=0x7ff9cc067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:32.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.064+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3114071269 shutdown_connections 2024-03-18T16:22:32.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.064+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3114071269 wait complete. 2024-03-18T16:22:32.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.066+0000 7ff9cbe00640 1 Processor -- start 2024-03-18T16:22:32.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.066+0000 7ff9cbe00640 1 -- start start 2024-03-18T16:22:32.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.068+0000 7ff9cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9cc0cff10 0x7ff9cc0b3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.068+0000 7ff9cbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9cc154620 0x7ff9cc0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.068+0000 7ff9cbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9cc154a00 0x7ff9cc15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.068+0000 7ff9cbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc15b000 con 0x7ff9cc0cff10 2024-03-18T16:22:32.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.068+0000 7ff9cbe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc15ad20 con 0x7ff9cc154a00 2024-03-18T16:22:32.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.068+0000 7ff9cbe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff9cc06d4b0 con 0x7ff9cc154620 2024-03-18T16:22:32.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.068+0000 7ff9caa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9cc0cff10 0x7ff9cc0b3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.069+0000 7ff9ca000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9cc154620 0x7ff9cc0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.069+0000 7ff9cb400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9cc154a00 0x7ff9cc15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.069+0000 7ff9cb400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9cc154a00 0x7ff9cc15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40060/0 (socket says 172.21.15.102:40060) 2024-03-18T16:22:32.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.069+0000 7ff9cb400640 1 -- 172.21.15.102:0/3784007302 learned_addr learned my addr 172.21.15.102:0/3784007302 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:32.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.072+0000 7ff9cb400640 1 -- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9cc154620 msgr2=0x7ff9cc0afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:32.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.072+0000 7ff9cb400640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9cc154620 0x7ff9cc0afea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.072+0000 7ff9cb400640 1 -- 172.21.15.102:0/3784007302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9cc0cff10 msgr2=0x7ff9cc0b3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:32.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.072+0000 7ff9cb400640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9cc0cff10 0x7ff9cc0b3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.072+0000 7ff9cb400640 1 -- 172.21.15.102:0/3784007302 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9cc0b0810 con 0x7ff9cc154a00 2024-03-18T16:22:32.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.074+0000 7ff9cb400640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9cc154a00 0x7ff9cc15cc00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4009e80 tx=0x7ff9c4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.078+0000 7ff9c2a00640 1 -- 172.21.15.102:0/3784007302 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4039050 con 0x7ff9cc154a00 2024-03-18T16:22:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.078+0000 7ff9c2a00640 1 -- 172.21.15.102:0/3784007302 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c4045070 con 0x7ff9cc154a00 2024-03-18T16:22:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.078+0000 7ff9c2a00640 1 -- 172.21.15.102:0/3784007302 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4009450 con 0x7ff9cc154a00 2024-03-18T16:22:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.078+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9cc0b0af0 con 0x7ff9cc154a00 2024-03-18T16:22:32.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.079+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9cc0af6a0 con 0x7ff9cc154a00 2024-03-18T16:22:32.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.080+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9cc074fd0 con 0x7ff9cc154a00 2024-03-18T16:22:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.101+0000 7ff9c2a00640 1 -- 172.21.15.102:0/3784007302 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff9c403e850 con 0x7ff9cc154a00 2024-03-18T16:22:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.102+0000 7ff9c2a00640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff9ac06bc50 0x7ff9ac06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.102+0000 7ff9caa00640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff9ac06bc50 0x7ff9ac06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.103+0000 7ff9caa00640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff9ac06bc50 0x7ff9ac06e110 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009b40 tx=0x7ff9bc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:32.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.103+0000 7ff9c2a00640 1 -- 172.21.15.102:0/3784007302 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7ff9c407a680 con 0x7ff9cc154a00 2024-03-18T16:22:32.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.350+0000 7ff9c2a00640 1 -- 172.21.15.102:0/3784007302 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff9c407c5e0 con 0x7ff9cc154a00 2024-03-18T16:22:32.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.547+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff9cc0b5b20 con 0x7ff9cc154a00 2024-03-18T16:22:32.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.553+0000 7ff9c2a00640 1 -- 172.21.15.102:0/3784007302 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v29) ==== 103+0+11 (secure 0 0 0) 0x7ff9c4080680 con 0x7ff9cc154a00 2024-03-18T16:22:32.555 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff9ac06bc50 msgr2=0x7ff9ac06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff9ac06bc50 0x7ff9ac06e110 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009b40 tx=0x7ff9bc009190 comp rx=0 tx=0).stop 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9cc154a00 msgr2=0x7ff9cc15cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9cc154a00 0x7ff9cc15cc00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4009e80 tx=0x7ff9c4004a40 comp rx=0 tx=0).stop 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 shutdown_connections 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff9ac06bc50 0x7ff9ac06e110 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9cc154a00 0x7ff9cc15cc00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff9cc154620 0x7ff9cc0afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 --2- 172.21.15.102:0/3784007302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9cc0cff10 0x7ff9cc0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 >> 172.21.15.102:0/3784007302 conn(0x7ff9cc067160 msgr2=0x7ff9cc06d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:32.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 shutdown_connections 2024-03-18T16:22:32.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.556+0000 7ff9cbe00640 1 -- 172.21.15.102:0/3784007302 wait complete. 2024-03-18T16:22:32.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:22:32.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.748+0000 7fde8c800640 1 Processor -- start 2024-03-18T16:22:32.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.748+0000 7fde8c800640 1 -- start start 2024-03-18T16:22:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.749+0000 7fde8c800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde841524c0 0x7fde84072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.749+0000 7fde8c800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8415b8e0 0x7fde840c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.749+0000 7fde8c800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8415c200 con 0x7fde84158ee0 2024-03-18T16:22:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.749+0000 7fde8c800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fde84159300 con 0x7fde841524c0 2024-03-18T16:22:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.749+0000 7fde8c800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde8415c4e0 con 0x7fde8415b8e0 2024-03-18T16:22:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.790+0000 7fde89e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde841524c0 0x7fde84072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.790+0000 7fde89e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fde841524c0 0x7fde84072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40070/0 (socket says 172.21.15.102:40070) 2024-03-18T16:22:32.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.790+0000 7fde89e00640 1 -- 172.21.15.102:0/669832052 learned_addr learned my addr 172.21.15.102:0/669832052 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:32.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.799+0000 7fde8a800640 1 --2- 172.21.15.102:0/669832052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8415b8e0 0x7fde840c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.803+0000 7fde8a800640 1 -- 172.21.15.102:0/669832052 >> v2:172.21.15.154:3300/0 conn(0x7fde841524c0 msgr2=0x7fde84072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:32.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.803+0000 7fde8a800640 1 --2- 172.21.15.102:0/669832052 >> v2:172.21.15.154:3300/0 conn(0x7fde841524c0 0x7fde84072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.803+0000 7fde8a800640 1 -- 172.21.15.102:0/669832052 >> v1:172.21.15.102:6789/0 conn(0x7fde84158ee0 legacy=0x7fde8406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:32.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.803+0000 7fde8a800640 1 -- 172.21.15.102:0/669832052 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8415c7c0 con 0x7fde8415b8e0 2024-03-18T16:22:32.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.807+0000 7fde8a800640 1 --2- 172.21.15.102:0/669832052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8415b8e0 0x7fde840c51e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde80009d90 tx=0x7fde8002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a32333602b93532 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:32.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.811+0000 7fde88a00640 1 -- 172.21.15.102:0/669832052 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde80039070 con 0x7fde8415b8e0 2024-03-18T16:22:32.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.811+0000 7fde88a00640 1 -- 172.21.15.102:0/669832052 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde8002cbc0 con 0x7fde8415b8e0 2024-03-18T16:22:32.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.811+0000 7fde88a00640 1 -- 172.21.15.102:0/669832052 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde80034440 con 0x7fde8415b8e0 2024-03-18T16:22:32.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.812+0000 7fde8c800640 1 -- 172.21.15.102:0/669832052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8415b8e0 msgr2=0x7fde840c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:32.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.812+0000 7fde8c800640 1 --2- 172.21.15.102:0/669832052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8415b8e0 0x7fde840c51e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde80009d90 tx=0x7fde8002be10 comp rx=0 tx=0).stop 2024-03-18T16:22:32.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.812+0000 7fde8c800640 1 -- 172.21.15.102:0/669832052 shutdown_connections 2024-03-18T16:22:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.812+0000 7fde8c800640 1 --2- 172.21.15.102:0/669832052 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde8415b8e0 0x7fde840c51e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.812+0000 7fde8c800640 1 --2- 172.21.15.102:0/669832052 >> v2:172.21.15.154:3300/0 conn(0x7fde841524c0 0x7fde84072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.812+0000 7fde8c800640 1 -- 172.21.15.102:0/669832052 >> 172.21.15.102:0/669832052 conn(0x7fde840700d0 msgr2=0x7fde8406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.813+0000 7fde8c800640 1 -- 172.21.15.102:0/669832052 shutdown_connections 2024-03-18T16:22:32.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.813+0000 7fde8c800640 1 -- 172.21.15.102:0/669832052 wait complete. 2024-03-18T16:22:32.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.815+0000 7fde8c800640 1 Processor -- start 2024-03-18T16:22:32.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.815+0000 7fde8c800640 1 -- start start 2024-03-18T16:22:32.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.817+0000 7fde8c800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde841524c0 0x7fde841eda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.817+0000 7fde8c800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde84158ee0 0x7fde841f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.817+0000 7fde8c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde8415b8e0 0x7fde841f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.817+0000 7fde8c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde841589c0 con 0x7fde8415b8e0 2024-03-18T16:22:32.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.817+0000 7fde8c800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde841586e0 con 0x7fde841524c0 2024-03-18T16:22:32.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.817+0000 7fde8c800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde841528a0 con 0x7fde84158ee0 2024-03-18T16:22:32.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.818+0000 7fde89400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde84158ee0 0x7fde841f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.818+0000 7fde89400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde84158ee0 0x7fde841f05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52138/0 (socket says 172.21.15.102:52138) 2024-03-18T16:22:32.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.818+0000 7fde89400640 1 -- 172.21.15.102:0/3532935015 learned_addr learned my addr 172.21.15.102:0/3532935015 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:32.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.821+0000 7fde89400640 1 -- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde841524c0 msgr2=0x7fde841eda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:32.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.821+0000 7fde89400640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde841524c0 0x7fde841eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.821+0000 7fde89400640 1 -- 172.21.15.102:0/3532935015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde8415b8e0 msgr2=0x7fde841f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:32.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.821+0000 7fde89400640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde8415b8e0 0x7fde841f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:32.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.821+0000 7fde89400640 1 -- 172.21.15.102:0/3532935015 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde840b0000 con 0x7fde84158ee0 2024-03-18T16:22:32.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.827+0000 7fde89400640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde84158ee0 0x7fde841f05e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fde7400c410 tx=0x7fde7400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.831+0000 7fde7e000640 1 -- 172.21.15.102:0/3532935015 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde74017070 con 0x7fde84158ee0 2024-03-18T16:22:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.831+0000 7fde7e000640 1 -- 172.21.15.102:0/3532935015 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde740035f0 con 0x7fde84158ee0 2024-03-18T16:22:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.831+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde840b02e0 con 0x7fde84158ee0 2024-03-18T16:22:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.831+0000 7fde7e000640 1 -- 172.21.15.102:0/3532935015 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde740039e0 con 0x7fde84158ee0 2024-03-18T16:22:32.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.831+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde841efde0 con 0x7fde84158ee0 2024-03-18T16:22:32.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.833+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde84074fd0 con 0x7fde84158ee0 2024-03-18T16:22:32.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.864+0000 7fde7e000640 1 -- 172.21.15.102:0/3532935015 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fde7401a070 con 0x7fde84158ee0 2024-03-18T16:22:32.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.866+0000 7fde7e000640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:32.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.867+0000 7fde89e00640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:32.868+0000 7fde89e00640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde7800b850 tx=0x7fde78008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:33.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.109+0000 7fde7e000640 1 -- 172.21.15.102:0/3532935015 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fde74056710 con 0x7fde84158ee0 2024-03-18T16:22:33.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.129+0000 7fde7e000640 1 -- 172.21.15.102:0/3532935015 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fde74021020 con 0x7fde84158ee0 2024-03-18T16:22:33.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.449+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde84158020 con 0x7fde84158ee0 2024-03-18T16:22:33.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.465+0000 7fde7e000640 1 -- 172.21.15.102:0/3532935015 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6924 (secure 0 0 0) 0x7fde74058d70 con 0x7fde84158ee0 2024-03-18T16:22:33.467 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:33.468 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:22:20.541204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4250,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4250":{"gid":4250,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":17,"addr":"172.21.15.102:6834/3423054813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3423054813},{"type":"v1","addr":"172.21.15.102:6834","nonce":3423054813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4247,4250,4253,4256,4270]},"id":1}]} 2024-03-18T16:22:33.468 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 19 2024-03-18T16:22:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 msgr2=0x7fde6806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fde7800b850 tx=0x7fde78008040 comp rx=0 tx=0).stop 2024-03-18T16:22:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde84158ee0 msgr2=0x7fde841f05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde84158ee0 0x7fde841f05e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fde7400c410 tx=0x7fde7400c8d0 comp rx=0 tx=0).stop 2024-03-18T16:22:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 shutdown_connections 2024-03-18T16:22:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde6806bc20 0x7fde6806e0e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:33.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde8415b8e0 0x7fde841f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:33.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde84158ee0 0x7fde841f05e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:33.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 --2- 172.21.15.102:0/3532935015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde841524c0 0x7fde841eda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:33.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 >> 172.21.15.102:0/3532935015 conn(0x7fde840700d0 msgr2=0x7fde8406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:33.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 shutdown_connections 2024-03-18T16:22:33.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.469+0000 7fde8c800640 1 -- 172.21.15.102:0/3532935015 wait complete. 2024-03-18T16:22:33.495 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:22:33.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.668+0000 7fd077200640 1 Processor -- start 2024-03-18T16:22:33.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.669+0000 7fd077200640 1 -- start start 2024-03-18T16:22:33.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.670+0000 7fd077200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:33.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.670+0000 7fd077200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0780c72d0 con 0x7fd078141650 2024-03-18T16:22:33.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.670+0000 7fd077200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd078158840 con 0x7fd078158420 2024-03-18T16:22:33.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.670+0000 7fd077200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd078159370 con 0x7fd07815c3a0 2024-03-18T16:22:33.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.671+0000 7fd076800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:33.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.671+0000 7fd075400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd078158420 0x7fd0780c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:48432/0 (socket says 172.21.15.102:48432) 2024-03-18T16:22:33.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.671+0000 7fd075400640 1 -- 172.21.15.102:0/974166690 learned_addr learned my addr 172.21.15.102:0/974166690 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:22:33.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.674+0000 7fd076800640 1 -- 172.21.15.102:0/974166690 >> v1:172.21.15.154:6789/0 conn(0x7fd078158420 legacy=0x7fd0780c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:33.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.674+0000 7fd076800640 1 -- 172.21.15.102:0/974166690 >> v1:172.21.15.102:6789/0 conn(0x7fd078141650 legacy=0x7fd0780c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:33.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.674+0000 7fd076800640 1 -- 172.21.15.102:0/974166690 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd078158ad0 con 0x7fd07815c3a0 2024-03-18T16:22:33.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.676+0000 7fd076800640 1 --2- 172.21.15.102:0/974166690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078163cd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd070009dc0 tx=0x7fd07002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c20b3e27d829e21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:33.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.680+0000 7fd074a00640 1 -- 172.21.15.102:0/974166690 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070039070 con 0x7fd07815c3a0 2024-03-18T16:22:33.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.680+0000 7fd074a00640 1 -- 172.21.15.102:0/974166690 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd07002ccd0 con 0x7fd07815c3a0 2024-03-18T16:22:33.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.680+0000 7fd074a00640 1 -- 172.21.15.102:0/974166690 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070034660 con 0x7fd07815c3a0 2024-03-18T16:22:33.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.680+0000 7fd077200640 1 -- 172.21.15.102:0/974166690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 msgr2=0x7fd078163cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:33.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.680+0000 7fd077200640 1 --2- 172.21.15.102:0/974166690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078163cd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd070009dc0 tx=0x7fd07002bf20 comp rx=0 tx=0).stop 2024-03-18T16:22:33.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.680+0000 7fd077200640 1 -- 172.21.15.102:0/974166690 shutdown_connections 2024-03-18T16:22:33.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.680+0000 7fd077200640 1 --2- 172.21.15.102:0/974166690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078163cd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:33.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.680+0000 7fd077200640 1 -- 172.21.15.102:0/974166690 >> 172.21.15.102:0/974166690 conn(0x7fd0780cb890 msgr2=0x7fd0780cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:33.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.681+0000 7fd077200640 1 -- 172.21.15.102:0/974166690 shutdown_connections 2024-03-18T16:22:33.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.681+0000 7fd077200640 1 -- 172.21.15.102:0/974166690 wait complete. 2024-03-18T16:22:33.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.682+0000 7fd077200640 1 Processor -- start 2024-03-18T16:22:33.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.682+0000 7fd077200640 1 -- start start 2024-03-18T16:22:33.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.683+0000 7fd077200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd078141650 0x7fd0781f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:33.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.683+0000 7fd077200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd078158420 0x7fd078155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:33.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.683+0000 7fd077200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078157130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:33.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.683+0000 7fd077200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd0780c63c0 con 0x7fd078141650 2024-03-18T16:22:33.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.683+0000 7fd077200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd078159090 con 0x7fd078158420 2024-03-18T16:22:33.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.683+0000 7fd077200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd0780c66a0 con 0x7fd07815c3a0 2024-03-18T16:22:33.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.684+0000 7fd076800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078157130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:33.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.684+0000 7fd075400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd078158420 0x7fd078155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:33.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.684+0000 7fd075e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd078141650 0x7fd0781f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:33.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.684+0000 7fd076800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078157130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52154/0 (socket says 172.21.15.102:52154) 2024-03-18T16:22:33.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.684+0000 7fd076800640 1 -- 172.21.15.102:0/2438440029 learned_addr learned my addr 172.21.15.102:0/2438440029 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:33.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.688+0000 7fd076800640 1 -- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd078158420 msgr2=0x7fd078155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:33.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.688+0000 7fd076800640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd078158420 0x7fd078155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:33.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.688+0000 7fd076800640 1 -- 172.21.15.102:0/2438440029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd078141650 msgr2=0x7fd0781f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:33.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.688+0000 7fd076800640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd078141650 0x7fd0781f4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:33.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.688+0000 7fd076800640 1 -- 172.21.15.102:0/2438440029 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0781f4580 con 0x7fd07815c3a0 2024-03-18T16:22:33.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.690+0000 7fd076800640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078157130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd070037040 tx=0x7fd070004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:33.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.694+0000 7fd06e000640 1 -- 172.21.15.102:0/2438440029 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd070039050 con 0x7fd07815c3a0 2024-03-18T16:22:33.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.694+0000 7fd06e000640 1 -- 172.21.15.102:0/2438440029 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd070045070 con 0x7fd07815c3a0 2024-03-18T16:22:33.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.694+0000 7fd06e000640 1 -- 172.21.15.102:0/2438440029 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07003e850 con 0x7fd07815c3a0 2024-03-18T16:22:33.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.694+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07813f720 con 0x7fd07815c3a0 2024-03-18T16:22:33.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.694+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0781e1fe0 con 0x7fd07815c3a0 2024-03-18T16:22:33.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.695+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd078074fd0 con 0x7fd07815c3a0 2024-03-18T16:22:33.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.715+0000 7fd06e000640 1 -- 172.21.15.102:0/2438440029 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd07003e9f0 con 0x7fd07815c3a0 2024-03-18T16:22:33.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.716+0000 7fd06e000640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd05806bc50 0x7fd05806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:33.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.716+0000 7fd075e00640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd05806bc50 0x7fd05806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:33.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.717+0000 7fd075e00640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd05806bc50 0x7fd05806e110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd068001780 tx=0x7fd068009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:33.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.718+0000 7fd06e000640 1 -- 172.21.15.102:0/2438440029 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fd07007a740 con 0x7fd07815c3a0 2024-03-18T16:22:33.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:33.968+0000 7fd06e000640 1 -- 172.21.15.102:0/2438440029 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd07007c6a0 con 0x7fd07815c3a0 2024-03-18T16:22:34.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.179+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd078140e70 con 0x7fd07815c3a0 2024-03-18T16:22:34.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.269+0000 7fd06e000640 1 -- 172.21.15.102:0/2438440029 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+18907 (secure 0 0 0) 0x7fd070080740 con 0x7fd07815c3a0 2024-03-18T16:22:34.272 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:34.272 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":29,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:21:54.749120+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:22:34.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd05806bc50 msgr2=0x7fd05806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:34.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd05806bc50 0x7fd05806e110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd068001780 tx=0x7fd068009040 comp rx=0 tx=0).stop 2024-03-18T16:22:34.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 msgr2=0x7fd078157130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:34.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078157130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd070037040 tx=0x7fd070004a40 comp rx=0 tx=0).stop 2024-03-18T16:22:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 shutdown_connections 2024-03-18T16:22:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd05806bc50 0x7fd05806e110 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd07815c3a0 0x7fd078157130 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd078158420 0x7fd078155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 --2- 172.21.15.102:0/2438440029 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd078141650 0x7fd0781f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.273+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 >> 172.21.15.102:0/2438440029 conn(0x7fd0780cb890 msgr2=0x7fd07813e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.274+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 shutdown_connections 2024-03-18T16:22:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.274+0000 7fd077200640 1 -- 172.21.15.102:0/2438440029 wait complete. 2024-03-18T16:22:34.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:22:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.455+0000 7f026e400640 1 Processor -- start 2024-03-18T16:22:34.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.456+0000 7f026e400640 1 -- start start 2024-03-18T16:22:34.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.457+0000 7f026e400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f026815ae70 0x7f02680ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:34.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.457+0000 7f026e400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f02680c9be0 0x7f02680d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:34.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.457+0000 7f026e400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0268154ab0 con 0x7f02680d05a0 2024-03-18T16:22:34.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.457+0000 7f026e400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f026806d2f0 con 0x7f02680c9be0 2024-03-18T16:22:34.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.457+0000 7f026e400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f026806d5d0 con 0x7f026815ae70 2024-03-18T16:22:34.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.458+0000 7f026d000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f026815ae70 0x7f02680ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:34.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.458+0000 7f026d000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f026815ae70 0x7f02680ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40102/0 (socket says 172.21.15.102:40102) 2024-03-18T16:22:34.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.458+0000 7f026d000640 1 -- 172.21.15.102:0/631914770 learned_addr learned my addr 172.21.15.102:0/631914770 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:34.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.458+0000 7f0267e00640 1 --2- 172.21.15.102:0/631914770 >> v2:172.21.15.102:3300/0 conn(0x7f02680c9be0 0x7f02680d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:34.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.460+0000 7f026d000640 1 -- 172.21.15.102:0/631914770 >> v2:172.21.15.102:3300/0 conn(0x7f02680c9be0 msgr2=0x7f02680d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:34.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.460+0000 7f026d000640 1 --2- 172.21.15.102:0/631914770 >> v2:172.21.15.102:3300/0 conn(0x7f02680c9be0 0x7f02680d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.460+0000 7f026d000640 1 -- 172.21.15.102:0/631914770 >> v1:172.21.15.102:6789/0 conn(0x7f02680d05a0 legacy=0x7f02680cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:34.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.460+0000 7f026d000640 1 -- 172.21.15.102:0/631914770 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026815b250 con 0x7f026815ae70 2024-03-18T16:22:34.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.462+0000 7f026d000640 1 --2- 172.21.15.102:0/631914770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 0x7f02680ca490 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f02680cd5c0 tx=0x7f025c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3abb04a5ad710987 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:34.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.466+0000 7f0267400640 1 -- 172.21.15.102:0/631914770 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c034070 con 0x7f026815ae70 2024-03-18T16:22:34.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.466+0000 7f0267400640 1 -- 172.21.15.102:0/631914770 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f025c030860 con 0x7f026815ae70 2024-03-18T16:22:34.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.466+0000 7f0267400640 1 -- 172.21.15.102:0/631914770 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c030c30 con 0x7f026815ae70 2024-03-18T16:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.467+0000 7f026e400640 1 -- 172.21.15.102:0/631914770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 msgr2=0x7f02680ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.467+0000 7f026e400640 1 --2- 172.21.15.102:0/631914770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 0x7f02680ca490 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f02680cd5c0 tx=0x7f025c02b9a0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.467+0000 7f026e400640 1 -- 172.21.15.102:0/631914770 shutdown_connections 2024-03-18T16:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.467+0000 7f026e400640 1 --2- 172.21.15.102:0/631914770 >> v2:172.21.15.102:3300/0 conn(0x7f02680c9be0 0x7f02680d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.467+0000 7f026e400640 1 --2- 172.21.15.102:0/631914770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 0x7f02680ca490 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.467+0000 7f026e400640 1 -- 172.21.15.102:0/631914770 >> 172.21.15.102:0/631914770 conn(0x7f02680673a0 msgr2=0x7f02680677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.467+0000 7f026e400640 1 -- 172.21.15.102:0/631914770 shutdown_connections 2024-03-18T16:22:34.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.467+0000 7f026e400640 1 -- 172.21.15.102:0/631914770 wait complete. 2024-03-18T16:22:34.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.470+0000 7f026e400640 1 Processor -- start 2024-03-18T16:22:34.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.470+0000 7f026e400640 1 -- start start 2024-03-18T16:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.471+0000 7f026e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02680c9be0 0x7f02681f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.471+0000 7f026e400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02680d05a0 0x7f02681f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:34.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.471+0000 7f026e400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 0x7f02680c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.472+0000 7f026e400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f026806de70 con 0x7f02680c9be0 2024-03-18T16:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.472+0000 7f026e400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f026806db90 con 0x7f026815ae70 2024-03-18T16:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.472+0000 7f026e400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f026806e150 con 0x7f02680d05a0 2024-03-18T16:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.472+0000 7f026da00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 0x7f02680c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.472+0000 7f026d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02680c9be0 0x7f02681f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:34.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.472+0000 7f0267e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02680d05a0 0x7f02681f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:34.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.472+0000 7f026da00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 0x7f02680c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40104/0 (socket says 172.21.15.102:40104) 2024-03-18T16:22:34.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.472+0000 7f026da00640 1 -- 172.21.15.102:0/3404196271 learned_addr learned my addr 172.21.15.102:0/3404196271 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:34.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.476+0000 7f026da00640 1 -- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02680d05a0 msgr2=0x7f02681f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:34.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.476+0000 7f026da00640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02680d05a0 0x7f02681f0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.476+0000 7f026da00640 1 -- 172.21.15.102:0/3404196271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02680c9be0 msgr2=0x7f02681f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:34.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.476+0000 7f026da00640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02680c9be0 0x7f02681f0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:34.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.476+0000 7f026da00640 1 -- 172.21.15.102:0/3404196271 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02680b2df0 con 0x7f026815ae70 2024-03-18T16:22:34.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.479+0000 7f026da00640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 0x7f02680c8760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f026000c6b0 tx=0x7f026000cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:34.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.483+0000 7f0264c00640 1 -- 172.21.15.102:0/3404196271 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0260003b70 con 0x7f026815ae70 2024-03-18T16:22:34.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.483+0000 7f0264c00640 1 -- 172.21.15.102:0/3404196271 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0260003d30 con 0x7f026815ae70 2024-03-18T16:22:34.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.483+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02680d0260 con 0x7f026815ae70 2024-03-18T16:22:34.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.483+0000 7f0264c00640 1 -- 172.21.15.102:0/3404196271 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0260005310 con 0x7f026815ae70 2024-03-18T16:22:34.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.483+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02681f4090 con 0x7f026815ae70 2024-03-18T16:22:34.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.485+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0268074fd0 con 0x7f026815ae70 2024-03-18T16:22:34.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.504+0000 7f0264c00640 1 -- 172.21.15.102:0/3404196271 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f02600054b0 con 0x7f026815ae70 2024-03-18T16:22:34.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.505+0000 7f0264c00640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f024806bc50 0x7f024806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:34.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.505+0000 7f026d000640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f024806bc50 0x7f024806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:34.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.506+0000 7f026d000640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f024806bc50 0x7f024806e110 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f025c036000 tx=0x7f025c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:34.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.506+0000 7f0264c00640 1 -- 172.21.15.102:0/3404196271 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7f02600057a0 con 0x7f026815ae70 2024-03-18T16:22:34.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:34.760+0000 7f0264c00640 1 -- 172.21.15.102:0/3404196271 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f02680b2df0 con 0x7f026815ae70 2024-03-18T16:22:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.009+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02681efe70 con 0x7f026815ae70 2024-03-18T16:22:35.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.023+0000 7f0264c00640 1 -- 172.21.15.102:0/3404196271 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6924 (secure 0 0 0) 0x7f0260059670 con 0x7f026815ae70 2024-03-18T16:22:35.026 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:22:35.026 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:22:20.541204+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4250,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4250":{"gid":4250,"name":"c","rank":3,"incarnation":16,"state":"up:active","state_seq":17,"addr":"172.21.15.102:6834/3423054813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":3423054813},{"type":"v1","addr":"172.21.15.102:6834","nonce":3423054813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4247,4250,4253,4256,4270]},"id":1}]} 2024-03-18T16:22:35.026 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 19 2024-03-18T16:22:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f024806bc50 msgr2=0x7f024806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f024806bc50 0x7f024806e110 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f025c036000 tx=0x7f025c008030 comp rx=0 tx=0).stop 2024-03-18T16:22:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 msgr2=0x7f02680c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 0x7f02680c8760 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f026000c6b0 tx=0x7f026000cb70 comp rx=0 tx=0).stop 2024-03-18T16:22:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 shutdown_connections 2024-03-18T16:22:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f024806bc50 0x7f024806e110 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f026815ae70 0x7f02680c8760 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02680d05a0 0x7f02681f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 --2- 172.21.15.102:0/3404196271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02680c9be0 0x7f02681f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 >> 172.21.15.102:0/3404196271 conn(0x7f02680673a0 msgr2=0x7f02680cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:35.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 shutdown_connections 2024-03-18T16:22:35.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.026+0000 7f026e400640 1 -- 172.21.15.102:0/3404196271 wait complete. 2024-03-18T16:22:35.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:22:20.541204+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4250, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4250': {'gid': 4250, 'name': 'c', 'rank': 3, 'incarnation': 16, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.102:6834/3423054813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6832', 'nonce': 3423054813}, {'type': 'v1', 'addr': '172.21.15.102:6834', 'nonce': 3423054813}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4247, 4250, 4253, 4256, 4270]} 2024-03-18T16:22:35.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T16:22:35.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T16:22:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.829+0000 7fa0ebe00640 1 Processor -- start 2024-03-18T16:22:35.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.829+0000 7fa0ebe00640 1 -- start start 2024-03-18T16:22:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.830+0000 7fa0ebe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec06b470 0x7fa0ec158710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.830+0000 7fa0ebe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa0ec06b850 0x7fa0ec158c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.830+0000 7fa0ebe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0ec15baa0 con 0x7fa0ec15b5d0 2024-03-18T16:22:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.830+0000 7fa0ebe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa0ec15bd80 con 0x7fa0ec06b850 2024-03-18T16:22:35.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.830+0000 7fa0ebe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa0ec15c060 con 0x7fa0ec06b470 2024-03-18T16:22:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.831+0000 7fa0eaa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec06b470 0x7fa0ec158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.831+0000 7fa0eb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa0ec06b850 0x7fa0ec158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.832+0000 7fa0eaa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec06b470 0x7fa0ec158710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52176/0 (socket says 172.21.15.102:52176) 2024-03-18T16:22:35.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.832+0000 7fa0eaa00640 1 -- 172.21.15.102:0/3069635676 learned_addr learned my addr 172.21.15.102:0/3069635676 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:35.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.835+0000 7fa0eaa00640 1 -- 172.21.15.102:0/3069635676 >> v2:172.21.15.102:3300/0 conn(0x7fa0ec06b850 msgr2=0x7fa0ec158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.835+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/3069635676 >> v2:172.21.15.102:3300/0 conn(0x7fa0ec06b850 0x7fa0ec158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.835+0000 7fa0eaa00640 1 -- 172.21.15.102:0/3069635676 >> v1:172.21.15.102:6789/0 conn(0x7fa0ec15b5d0 legacy=0x7fa0ec151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:35.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.835+0000 7fa0eaa00640 1 -- 172.21.15.102:0/3069635676 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ec15c340 con 0x7fa0ec06b470 2024-03-18T16:22:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.838+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/3069635676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec06b470 0x7fa0ec158710 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0098a0 tx=0x7fa0dc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2827300728e82382 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:35.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.842+0000 7fa0e9600640 1 -- 172.21.15.102:0/3069635676 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc034070 con 0x7fa0ec06b470 2024-03-18T16:22:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.843+0000 7fa0e9600640 1 -- 172.21.15.102:0/3069635676 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0dc030840 con 0x7fa0ec06b470 2024-03-18T16:22:35.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.843+0000 7fa0e9600640 1 -- 172.21.15.102:0/3069635676 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc030c10 con 0x7fa0ec06b470 2024-03-18T16:22:35.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.844+0000 7fa0ebe00640 1 -- 172.21.15.102:0/3069635676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec06b470 msgr2=0x7fa0ec158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:35.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.844+0000 7fa0ebe00640 1 --2- 172.21.15.102:0/3069635676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec06b470 0x7fa0ec158710 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0098a0 tx=0x7fa0dc02b980 comp rx=0 tx=0).stop 2024-03-18T16:22:35.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.844+0000 7fa0ebe00640 1 -- 172.21.15.102:0/3069635676 shutdown_connections 2024-03-18T16:22:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.844+0000 7fa0ebe00640 1 --2- 172.21.15.102:0/3069635676 >> v2:172.21.15.102:3300/0 conn(0x7fa0ec06b850 0x7fa0ec158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.844+0000 7fa0ebe00640 1 --2- 172.21.15.102:0/3069635676 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec06b470 0x7fa0ec158710 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.844+0000 7fa0ebe00640 1 -- 172.21.15.102:0/3069635676 >> 172.21.15.102:0/3069635676 conn(0x7fa0ec0c8430 msgr2=0x7fa0ec0c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.844+0000 7fa0ebe00640 1 -- 172.21.15.102:0/3069635676 shutdown_connections 2024-03-18T16:22:35.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.844+0000 7fa0ebe00640 1 -- 172.21.15.102:0/3069635676 wait complete. 2024-03-18T16:22:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.846+0000 7fa0ebe00640 1 Processor -- start 2024-03-18T16:22:35.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.846+0000 7fa0ebe00640 1 -- start start 2024-03-18T16:22:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.847+0000 7fa0ebe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0ec06b470 0x7fa0ec1f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.847+0000 7fa0ebe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0ec06b850 0x7fa0ec0cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.847+0000 7fa0ebe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec15b5d0 0x7fa0ec1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.847+0000 7fa0ebe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ec0c5ac0 con 0x7fa0ec06b470 2024-03-18T16:22:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.847+0000 7fa0ebe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ec0c68c0 con 0x7fa0ec06b850 2024-03-18T16:22:35.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.847+0000 7fa0ebe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa0ec0c8100 con 0x7fa0ec15b5d0 2024-03-18T16:22:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.848+0000 7fa0eb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec15b5d0 0x7fa0ec1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.848+0000 7fa0eaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0ec06b470 0x7fa0ec1f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.848+0000 7fa0ea000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0ec06b850 0x7fa0ec0cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.848+0000 7fa0eb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec15b5d0 0x7fa0ec1563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52184/0 (socket says 172.21.15.102:52184) 2024-03-18T16:22:35.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.848+0000 7fa0eb400640 1 -- 172.21.15.102:0/3374018314 learned_addr learned my addr 172.21.15.102:0/3374018314 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.852+0000 7fa0ea000640 1 -- 172.21.15.102:0/3374018314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec15b5d0 msgr2=0x7fa0ec1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.852+0000 7fa0ea000640 1 --2- 172.21.15.102:0/3374018314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0ec15b5d0 0x7fa0ec1563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.852+0000 7fa0ea000640 1 -- 172.21.15.102:0/3374018314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0ec06b470 msgr2=0x7fa0ec1f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.852+0000 7fa0ea000640 1 --2- 172.21.15.102:0/3374018314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0ec06b470 0x7fa0ec1f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.852+0000 7fa0ea000640 1 -- 172.21.15.102:0/3374018314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ec072bc0 con 0x7fa0ec06b850 2024-03-18T16:22:35.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.854+0000 7fa0ea000640 1 --2- 172.21.15.102:0/3374018314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0ec06b850 0x7fa0ec0cb1d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec15b9d0 tx=0x7fa0d800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:35.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.858+0000 7fa0e2a00640 1 -- 172.21.15.102:0/3374018314 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8017070 con 0x7fa0ec06b850 2024-03-18T16:22:35.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.858+0000 7fa0e2a00640 1 -- 172.21.15.102:0/3374018314 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d80052f0 con 0x7fa0ec06b850 2024-03-18T16:22:35.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.858+0000 7fa0e2a00640 1 -- 172.21.15.102:0/3374018314 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8003850 con 0x7fa0ec06b850 2024-03-18T16:22:35.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.858+0000 7fa0ebe00640 1 -- 172.21.15.102:0/3374018314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0ec072e80 con 0x7fa0ec06b850 2024-03-18T16:22:35.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.858+0000 7fa0ebe00640 1 -- 172.21.15.102:0/3374018314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0ec0ca9d0 con 0x7fa0ec06b850 2024-03-18T16:22:35.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.862+0000 7fa0f4365740 1 Processor -- start 2024-03-18T16:22:35.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.862+0000 7fa0f4365740 1 -- start start 2024-03-18T16:22:35.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.863+0000 7fa0f4365740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36950 0x561034af66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.863+0000 7fa0f4365740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561034af53f0 0x561034b37310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.863+0000 7fa0f4365740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034b36d30 0x561034b35c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.863+0000 7fa0f4365740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561034a59c50 con 0x561034b36950 2024-03-18T16:22:35.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.863+0000 7fa0f4365740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561034a029d0 con 0x561034af53f0 2024-03-18T16:22:35.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.863+0000 7fa0f4365740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x561034a55cf0 con 0x561034b36d30 2024-03-18T16:22:35.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.864+0000 7fa0ea000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034b36d30 0x561034b35c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.864+0000 7fa0eb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36950 0x561034af66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.864+0000 7fa0ea000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034b36d30 0x561034b35c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52200/0 (socket says 172.21.15.102:52200) 2024-03-18T16:22:35.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.864+0000 7fa0ea000640 1 -- 172.21.15.102:0/1483752019 learned_addr learned my addr 172.21.15.102:0/1483752019 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:35.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.868+0000 7fa0eb400640 1 -- 172.21.15.102:0/1483752019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034b36d30 msgr2=0x561034b35c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:35.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.868+0000 7fa0eb400640 1 --2- 172.21.15.102:0/1483752019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034b36d30 0x561034b35c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.868+0000 7fa0eb400640 1 -- 172.21.15.102:0/1483752019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561034af53f0 msgr2=0x561034b37310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:35.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.868+0000 7fa0eb400640 1 --2- 172.21.15.102:0/1483752019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561034af53f0 0x561034b37310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.868+0000 7fa0eb400640 1 -- 172.21.15.102:0/1483752019 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561034a1cf10 con 0x561034b36950 2024-03-18T16:22:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.871+0000 7fa0eb400640 1 --2- 172.21.15.102:0/1483752019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36950 0x561034af66f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4011580 tx=0x7fa0e4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=776019adaeed38fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:35.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.875+0000 7fa0d7000640 1 -- 172.21.15.102:0/1483752019 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4015840 con 0x561034b36950 2024-03-18T16:22:35.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.876+0000 7fa0d7000640 1 -- 172.21.15.102:0/1483752019 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e40159e0 con 0x561034b36950 2024-03-18T16:22:35.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.876+0000 7fa0d7000640 1 -- 172.21.15.102:0/1483752019 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4015d90 con 0x561034b36950 2024-03-18T16:22:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.877+0000 7fa0f4365740 1 -- 172.21.15.102:0/1483752019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36950 msgr2=0x561034af66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:22:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.877+0000 7fa0f4365740 1 --2- 172.21.15.102:0/1483752019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36950 0x561034af66f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4011580 tx=0x7fa0e4009750 comp rx=0 tx=0).stop 2024-03-18T16:22:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.877+0000 7fa0f4365740 1 -- 172.21.15.102:0/1483752019 shutdown_connections 2024-03-18T16:22:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.877+0000 7fa0f4365740 1 --2- 172.21.15.102:0/1483752019 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034b36d30 0x561034b35c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.877+0000 7fa0f4365740 1 --2- 172.21.15.102:0/1483752019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561034af53f0 0x561034b37310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.877+0000 7fa0f4365740 1 --2- 172.21.15.102:0/1483752019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36950 0x561034af66f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.877+0000 7fa0f4365740 1 -- 172.21.15.102:0/1483752019 >> 172.21.15.102:0/1483752019 conn(0x561034af2560 msgr2=0x561034af5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:22:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.877+0000 7fa0f4365740 1 -- 172.21.15.102:0/1483752019 shutdown_connections 2024-03-18T16:22:35.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.877+0000 7fa0f4365740 1 -- 172.21.15.102:0/1483752019 wait complete. 2024-03-18T16:22:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.878+0000 7fa0f4365740 1 -- start start 2024-03-18T16:22:35.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.879+0000 7fa0f4365740 1 Processor -- start 2024-03-18T16:22:35.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.880+0000 7fa0e2a00640 1 -- 172.21.15.102:0/3374018314 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa0d801a070 con 0x7fa0ec06b850 2024-03-18T16:22:35.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.880+0000 7fa0f4365740 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561034b36950 0x561034b1ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.880+0000 7fa0f4365740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36d30 0x561034b46790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.880+0000 7fa0f4365740 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034af1a50 0x561034b46cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.880+0000 7fa0f4365740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x561034a15e20 con 0x561034b36d30 2024-03-18T16:22:35.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.880+0000 7fa0f4365740 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x561034b3fc80 con 0x561034b36950 2024-03-18T16:22:35.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.880+0000 7fa0f4365740 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x561034b48140 con 0x561034af1a50 2024-03-18T16:22:35.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.880+0000 7fa0f4365740 10 client.0 _ll_register_callbacks cb 0x561034a157f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:22:35.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.881+0000 7fa0e2a00640 1 --2- 172.21.15.102:0/3374018314 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0cc06bb90 0x7fa0cc06e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.881+0000 7fa0eb400640 1 --2- 172.21.15.102:0/3374018314 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0cc06bb90 0x7fa0cc06e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.881+0000 7fa0ea000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034af1a50 0x561034b46cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.882+0000 7fa0ea000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034af1a50 0x561034b46cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52206/0 (socket says 172.21.15.102:52206) 2024-03-18T16:22:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.882+0000 7fa0ea000640 1 -- 172.21.15.102:0/1081305754 learned_addr learned my addr 172.21.15.102:0/1081305754 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:22:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.882+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36d30 0x561034b46790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:22:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.882+0000 7fa0eb400640 1 --2- 172.21.15.102:0/3374018314 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0cc06bb90 0x7fa0cc06e050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009790 tx=0x7fa0e4016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:35.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.883+0000 7fa0e2a00640 1 -- 172.21.15.102:0/3374018314 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fa0d80885a0 con 0x7fa0ec06b850 2024-03-18T16:22:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.885+0000 7fa0ea000640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561034b36950 msgr2=0x561034b1ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.885+0000 7fa0ea000640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561034b36950 0x561034b1ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.885+0000 7fa0ea000640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36d30 msgr2=0x561034b46790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:22:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.885+0000 7fa0ea000640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561034b36d30 0x561034b46790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:22:35.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.885+0000 7fa0ea000640 1 -- 172.21.15.102:0/1081305754 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x56103476e2d0 con 0x561034af1a50 2024-03-18T16:22:35.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.888+0000 7fa0ea000640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x561034af1a50 0x561034b46cd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80569c0 tx=0x7fa0d8058550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a2c2b196aeca41 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:22:35.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.888+0000 7fa0cb400640 10 client.0 ms_handle_connect on v2:172.21.15.154:3301/0 2024-03-18T16:22:35.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.891+0000 7fa0cb400640 1 -- 172.21.15.102:0/1081305754 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d805c170 con 0x561034af1a50 2024-03-18T16:22:35.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.891+0000 7fa0cb400640 1 -- 172.21.15.102:0/1081305754 <== mon.2 v2:172.21.15.154:3301/0 2 ==== osd_map(29..29 src has 1..29) ==== 6735+0+0 (secure 0 0 0) 0x7fa0d80578e0 con 0x561034af1a50 2024-03-18T16:22:35.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.891+0000 7fa0f4365740 1 -- 172.21.15.102:0/1081305754 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0e40159e0 con 0x561034af1a50 2024-03-18T16:22:35.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.893+0000 7fa0cb400640 1 -- 172.21.15.102:0/1081305754 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa0d8088050 con 0x561034af1a50 2024-03-18T16:22:35.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.893+0000 7fa0f4365740 10 client.4637 fetch_fsmap learned FSMap version 19 2024-03-18T16:22:35.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.893+0000 7fa0f4365740 1 -- 172.21.15.102:0/1081305754 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5610348386b0 con 0x561034af1a50 2024-03-18T16:22:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.897+0000 7fa0cb400640 1 -- 172.21.15.102:0/1081305754 <== mon.2 v2:172.21.15.154:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e40159e0 con 0x561034af1a50 2024-03-18T16:22:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.897+0000 7fa0cb400640 1 -- 172.21.15.102:0/1081305754 <== mon.2 v2:172.21.15.154:3301/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7fa0d80541e0 con 0x561034af1a50 2024-03-18T16:22:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.897+0000 7fa0f4365740 10 client.4637 fetch_fsmap finished waiting for FSMap version 19 2024-03-18T16:22:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.897+0000 7fa0f4365740 10 client.4637 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T16:22:35.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.897+0000 7fa0f4365740 20 client.4637 populate_metadata read hostname 'smithi102' 2024-03-18T16:22:35.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.897+0000 7fa0f4365740 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 0x561034b558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:22:35.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.897+0000 7fa0f4365740 4 client.4637 mds_command: new command op to 4247 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:22:35.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:35.897+0000 7fa0f4365740 1 -- 172.21.15.102:0/1081305754 --> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561034ab72b0 con 0x561034b4a160 2024-03-18T16:22:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:45.903+0000 7fa0eaa00640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 msgr2=0x561034b558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.154:6836/120487002, fault. 2024-03-18T16:22:45.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:45.903+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 0x561034b558f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T16:22:50.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:22:50.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:22:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:56.113+0000 7fa0eaa00640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 msgr2=0x561034b558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.154:6836/120487002, fault. 2024-03-18T16:22:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:22:56.113+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 0x561034b558f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T16:23:06.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:06.525+0000 7fa0eaa00640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 msgr2=0x561034b558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.154:6836/120487002, fault. 2024-03-18T16:23:06.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:06.525+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 0x561034b558f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T16:23:17.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:17.336+0000 7fa0eaa00640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 msgr2=0x561034b558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.154:6836/120487002, fault. 2024-03-18T16:23:17.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:17.336+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 0x561034b558f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T16:23:20.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:23:20.837 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:23:28.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:28.948+0000 7fa0eaa00640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 msgr2=0x561034b558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.154:6836/120487002, fault. 2024-03-18T16:23:28.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:28.948+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 0x561034b558f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T16:23:42.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:42.160+0000 7fa0eaa00640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 msgr2=0x561034b558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.154:6836/120487002, fault. 2024-03-18T16:23:42.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:42.160+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 0x561034b558f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-18T16:23:50.934 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:23:50.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:23:58.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:58.573+0000 7fa0eaa00640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 msgr2=0x561034b558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.154:6836/120487002, fault. 2024-03-18T16:23:58.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:23:58.573+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 0x561034b558f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-18T16:24:21.046 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:24:21.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:24:21.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:21.395+0000 7fa0eaa00640 1 -- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 msgr2=0x561034b558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.154:6836/120487002, fault. 2024-03-18T16:24:21.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:21.395+0000 7fa0eaa00640 1 --2- 172.21.15.102:0/1081305754 >> [v2:172.21.15.154:6836/120487002,v1:172.21.15.154:6837/120487002] conn(0x561034b4a160 0x561034b558f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-18T16:24:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-18T16:24:36.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:36.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.923+0000 7f5394200640 1 Processor -- start 2024-03-18T16:24:36.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.924+0000 7f5394200640 1 -- start start 2024-03-18T16:24:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.925+0000 7f5394200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f538c158bb0 0x7f538c072c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.925+0000 7f5394200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f538c0c7830 0x7f538c071e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.925+0000 7f5394200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f538c15e950 con 0x7f538c0c7ca0 2024-03-18T16:24:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.925+0000 7f5394200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f538c15f540 con 0x7f538c158bb0 2024-03-18T16:24:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.925+0000 7f5394200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f538c0c8190 con 0x7f538c0c7830 2024-03-18T16:24:36.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.926+0000 7f5390e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f538c0c7830 0x7f538c071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:36.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.926+0000 7f5391800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f538c158bb0 0x7f538c072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:36.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.926+0000 7f5392200640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f538c0c7ca0 0x7f538c06d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:42094/0 (socket says 172.21.15.102:42094) 2024-03-18T16:24:36.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.926+0000 7f5392200640 1 -- 172.21.15.102:0/2336098785 learned_addr learned my addr 172.21.15.102:0/2336098785 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:24:36.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.929+0000 7f5390e00640 1 -- 172.21.15.102:0/2336098785 >> v2:172.21.15.102:3300/0 conn(0x7f538c158bb0 msgr2=0x7f538c072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:36.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.929+0000 7f5390e00640 1 --2- 172.21.15.102:0/2336098785 >> v2:172.21.15.102:3300/0 conn(0x7f538c158bb0 0x7f538c072c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:36.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.929+0000 7f5390e00640 1 -- 172.21.15.102:0/2336098785 >> v1:172.21.15.154:6789/0 conn(0x7f538c0c7ca0 legacy=0x7f538c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:36.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.929+0000 7f5390e00640 1 -- 172.21.15.102:0/2336098785 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538c15ec10 con 0x7f538c0c7830 2024-03-18T16:24:36.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.934+0000 7f5390e00640 1 --2- 172.21.15.102:0/2336098785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7830 0x7f538c071e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f538c06c170 tx=0x7f537802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e465bde8959a9b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.939+0000 7f538be00640 1 -- 172.21.15.102:0/2336098785 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378035020 con 0x7f538c0c7830 2024-03-18T16:24:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.940+0000 7f538be00640 1 -- 172.21.15.102:0/2336098785 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5378031830 con 0x7f538c0c7830 2024-03-18T16:24:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.940+0000 7f538be00640 1 -- 172.21.15.102:0/2336098785 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378031c00 con 0x7f538c0c7830 2024-03-18T16:24:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.941+0000 7f5394200640 1 -- 172.21.15.102:0/2336098785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7830 msgr2=0x7f538c071e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.941+0000 7f5394200640 1 --2- 172.21.15.102:0/2336098785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7830 0x7f538c071e20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f538c06c170 tx=0x7f537802b8d0 comp rx=0 tx=0).stop 2024-03-18T16:24:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.941+0000 7f5394200640 1 -- 172.21.15.102:0/2336098785 shutdown_connections 2024-03-18T16:24:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.941+0000 7f5394200640 1 --2- 172.21.15.102:0/2336098785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7830 0x7f538c071e20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.941+0000 7f5394200640 1 --2- 172.21.15.102:0/2336098785 >> v2:172.21.15.102:3300/0 conn(0x7f538c158bb0 0x7f538c072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.941+0000 7f5394200640 1 -- 172.21.15.102:0/2336098785 >> 172.21.15.102:0/2336098785 conn(0x7f538c0700d0 msgr2=0x7f538c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.941+0000 7f5394200640 1 -- 172.21.15.102:0/2336098785 shutdown_connections 2024-03-18T16:24:36.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.941+0000 7f5394200640 1 -- 172.21.15.102:0/2336098785 wait complete. 2024-03-18T16:24:36.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.943+0000 7f5394200640 1 Processor -- start 2024-03-18T16:24:36.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.944+0000 7f5394200640 1 -- start start 2024-03-18T16:24:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.945+0000 7f5394200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f538c0c7830 0x7f538c15b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.945+0000 7f5394200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7ca0 0x7f538c14ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.945+0000 7f5394200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538c158bb0 0x7f538c156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.945+0000 7f5394200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f538c0731c0 con 0x7f538c0c7830 2024-03-18T16:24:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.945+0000 7f5394200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f538c15f1d0 con 0x7f538c0c7ca0 2024-03-18T16:24:36.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.945+0000 7f5394200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f538c158f90 con 0x7f538c158bb0 2024-03-18T16:24:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.946+0000 7f5390e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7ca0 0x7f538c14ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.946+0000 7f5390e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7ca0 0x7f538c14ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49622/0 (socket says 172.21.15.102:49622) 2024-03-18T16:24:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.946+0000 7f5392200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538c158bb0 0x7f538c156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:36.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.946+0000 7f5390e00640 1 -- 172.21.15.102:0/2179409221 learned_addr learned my addr 172.21.15.102:0/2179409221 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:36.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.950+0000 7f5391800640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f538c0c7830 0x7f538c15b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.950+0000 7f5390e00640 1 -- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538c158bb0 msgr2=0x7f538c156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.950+0000 7f5390e00640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538c158bb0 0x7f538c156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.950+0000 7f5390e00640 1 -- 172.21.15.102:0/2179409221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f538c0c7830 msgr2=0x7f538c15b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.950+0000 7f5390e00640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f538c0c7830 0x7f538c15b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:36.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.950+0000 7f5390e00640 1 -- 172.21.15.102:0/2179409221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538c15fc80 con 0x7f538c0c7ca0 2024-03-18T16:24:36.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.953+0000 7f5390e00640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7ca0 0x7f538c14ed40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5378009890 tx=0x7f53780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:36.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.963+0000 7f5389600640 1 -- 172.21.15.102:0/2179409221 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378041070 con 0x7f538c0c7ca0 2024-03-18T16:24:36.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.964+0000 7f5389600640 1 -- 172.21.15.102:0/2179409221 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f537803b920 con 0x7f538c0c7ca0 2024-03-18T16:24:36.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.964+0000 7f5389600640 1 -- 172.21.15.102:0/2179409221 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378035040 con 0x7f538c0c7ca0 2024-03-18T16:24:36.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.964+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538c14f580 con 0x7f538c0c7ca0 2024-03-18T16:24:36.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.964+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538c1f7e70 con 0x7f538c0c7ca0 2024-03-18T16:24:36.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:36.965+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f538c0c61c0 con 0x7f538c0c7ca0 2024-03-18T16:24:37.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.241+0000 7f5389600640 1 -- 172.21.15.102:0/2179409221 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5378038020 con 0x7f538c0c7ca0 2024-03-18T16:24:37.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.242+0000 7f5389600640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f536c06bc50 0x7f536c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:37.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.243+0000 7f5391800640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f536c06bc50 0x7f536c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:37.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.244+0000 7f5391800640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f536c06bc50 0x7f536c06e110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5380009970 tx=0x7f5380008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:37.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.246+0000 7f5389600640 1 -- 172.21.15.102:0/2179409221 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5378078c20 con 0x7f538c0c7ca0 2024-03-18T16:24:37.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.258+0000 7f5389600640 1 -- 172.21.15.102:0/2179409221 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f538c0c61c0 con 0x7f538c0c7ca0 2024-03-18T16:24:37.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.547+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f538c074fd0 con 0x7f538c0c7ca0 2024-03-18T16:24:37.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.561+0000 7f5389600640 1 -- 172.21.15.102:0/2179409221 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f538c15fc80 con 0x7f538c0c7ca0 2024-03-18T16:24:37.564 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:37.564 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:37.564 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:37.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f536c06bc50 msgr2=0x7f536c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f536c06bc50 0x7f536c06e110 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5380009970 tx=0x7f5380008040 comp rx=0 tx=0).stop 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7ca0 msgr2=0x7f538c14ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7ca0 0x7f538c14ed40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5378009890 tx=0x7f53780046c0 comp rx=0 tx=0).stop 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 shutdown_connections 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f536c06bc50 0x7f536c06e110 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f538c158bb0 0x7f538c156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f538c0c7ca0 0x7f538c14ed40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 --2- 172.21.15.102:0/2179409221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f538c0c7830 0x7f538c15b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.564+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 >> 172.21.15.102:0/2179409221 conn(0x7f538c0700d0 msgr2=0x7f538c15d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.565+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 shutdown_connections 2024-03-18T16:24:37.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:37.565+0000 7f5394200640 1 -- 172.21.15.102:0/2179409221 wait complete. 2024-03-18T16:24:38.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:24:38.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:24:39.190 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:39.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.391+0000 7fc0b3200640 1 Processor -- start 2024-03-18T16:24:39.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.391+0000 7fc0b3200640 1 -- start start 2024-03-18T16:24:39.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.392+0000 7fc0b3200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc0b40c9be0 0x7fc0b40d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:39.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.392+0000 7fc0b3200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b40d05a0 0x7fc0b40cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:39.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.392+0000 7fc0b3200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0b4154ab0 con 0x7fc0b415ae70 2024-03-18T16:24:39.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.392+0000 7fc0b3200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc0b40caba0 con 0x7fc0b40c9be0 2024-03-18T16:24:39.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.392+0000 7fc0b3200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc0b406d2f0 con 0x7fc0b40d05a0 2024-03-18T16:24:39.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.418+0000 7fc0b1400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc0b40c9be0 0x7fc0b40d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:39.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.418+0000 7fc0b1e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc0b415ae70 0x7fc0b40ca490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:42104/0 (socket says 172.21.15.102:42104) 2024-03-18T16:24:39.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.418+0000 7fc0b1e00640 1 -- 172.21.15.102:0/3472939212 learned_addr learned my addr 172.21.15.102:0/3472939212 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:24:39.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.419+0000 7fc0b2800640 1 --2- 172.21.15.102:0/3472939212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b40d05a0 0x7fc0b40cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:39.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.422+0000 7fc0b1400640 1 -- 172.21.15.102:0/3472939212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b40d05a0 msgr2=0x7fc0b40cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:39.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.422+0000 7fc0b1400640 1 --2- 172.21.15.102:0/3472939212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b40d05a0 0x7fc0b40cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:39.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.422+0000 7fc0b1400640 1 -- 172.21.15.102:0/3472939212 >> v1:172.21.15.154:6789/0 conn(0x7fc0b415ae70 legacy=0x7fc0b40ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:39.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.422+0000 7fc0b1400640 1 -- 172.21.15.102:0/3472939212 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b415b250 con 0x7fc0b40c9be0 2024-03-18T16:24:39.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.425+0000 7fc0b1400640 1 --2- 172.21.15.102:0/3472939212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 0x7fc0b40d0060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00097a0 tx=0x7fc0a002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bd87a7fd3434737 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:39.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.429+0000 7fc0b0a00640 1 -- 172.21.15.102:0/3472939212 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0035020 con 0x7fc0b40c9be0 2024-03-18T16:24:39.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.429+0000 7fc0b0a00640 1 -- 172.21.15.102:0/3472939212 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0a0031810 con 0x7fc0b40c9be0 2024-03-18T16:24:39.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.429+0000 7fc0b0a00640 1 -- 172.21.15.102:0/3472939212 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0031be0 con 0x7fc0b40c9be0 2024-03-18T16:24:39.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.430+0000 7fc0b3200640 1 -- 172.21.15.102:0/3472939212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 msgr2=0x7fc0b40d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:39.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.430+0000 7fc0b3200640 1 --2- 172.21.15.102:0/3472939212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 0x7fc0b40d0060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00097a0 tx=0x7fc0a002b8b0 comp rx=0 tx=0).stop 2024-03-18T16:24:39.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.430+0000 7fc0b3200640 1 -- 172.21.15.102:0/3472939212 shutdown_connections 2024-03-18T16:24:39.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.430+0000 7fc0b3200640 1 --2- 172.21.15.102:0/3472939212 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b40d05a0 0x7fc0b40cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:39.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.430+0000 7fc0b3200640 1 --2- 172.21.15.102:0/3472939212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 0x7fc0b40d0060 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:39.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.430+0000 7fc0b3200640 1 -- 172.21.15.102:0/3472939212 >> 172.21.15.102:0/3472939212 conn(0x7fc0b40673a0 msgr2=0x7fc0b40677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:39.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.430+0000 7fc0b3200640 1 -- 172.21.15.102:0/3472939212 shutdown_connections 2024-03-18T16:24:39.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.431+0000 7fc0b3200640 1 -- 172.21.15.102:0/3472939212 wait complete. 2024-03-18T16:24:39.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.433+0000 7fc0b3200640 1 Processor -- start 2024-03-18T16:24:39.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.433+0000 7fc0b3200640 1 -- start start 2024-03-18T16:24:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.435+0000 7fc0b3200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 0x7fc0b4157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.435+0000 7fc0b3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0b40d05a0 0x7fc0b40c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.435+0000 7fc0b3200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b415ae70 0x7fc0b40b52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.435+0000 7fc0b3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b406db90 con 0x7fc0b40d05a0 2024-03-18T16:24:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.435+0000 7fc0b3200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0b406d8b0 con 0x7fc0b40c9be0 2024-03-18T16:24:39.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.435+0000 7fc0b3200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc0b406de70 con 0x7fc0b415ae70 2024-03-18T16:24:39.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.435+0000 7fc0b1e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 0x7fc0b4157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:39.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.436+0000 7fc0b1e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 0x7fc0b4157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49650/0 (socket says 172.21.15.102:49650) 2024-03-18T16:24:39.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.436+0000 7fc0b1e00640 1 -- 172.21.15.102:0/1435700832 learned_addr learned my addr 172.21.15.102:0/1435700832 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:39.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.436+0000 7fc0b2800640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b415ae70 0x7fc0b40b52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:39.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.440+0000 7fc0b1e00640 1 -- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b415ae70 msgr2=0x7fc0b40b52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:39.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.440+0000 7fc0b1e00640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b415ae70 0x7fc0b40b52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:39.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.440+0000 7fc0b1e00640 1 -- 172.21.15.102:0/1435700832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0b40d05a0 msgr2=0x7fc0b40c43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:39.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.440+0000 7fc0b1e00640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0b40d05a0 0x7fc0b40c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:39.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.440+0000 7fc0b1e00640 1 -- 172.21.15.102:0/1435700832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b4158260 con 0x7fc0b40c9be0 2024-03-18T16:24:39.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.444+0000 7fc0b1e00640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 0x7fc0b4157c40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400c400 tx=0x7fc0a400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:39.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.453+0000 7fc0aa000640 1 -- 172.21.15.102:0/1435700832 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4017070 con 0x7fc0b40c9be0 2024-03-18T16:24:39.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.453+0000 7fc0aa000640 1 -- 172.21.15.102:0/1435700832 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0a4005ce0 con 0x7fc0b40c9be0 2024-03-18T16:24:39.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.453+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0b41587e0 con 0x7fc0b40c9be0 2024-03-18T16:24:39.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.453+0000 7fc0aa000640 1 -- 172.21.15.102:0/1435700832 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4007310 con 0x7fc0b40c9be0 2024-03-18T16:24:39.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.453+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0b40b4810 con 0x7fc0b40c9be0 2024-03-18T16:24:39.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.454+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0b40733c0 con 0x7fc0b40c9be0 2024-03-18T16:24:39.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.485+0000 7fc0aa000640 1 -- 172.21.15.102:0/1435700832 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc0a401a070 con 0x7fc0b40c9be0 2024-03-18T16:24:39.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.487+0000 7fc0aa000640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09406bc20 0x7fc09406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:39.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.487+0000 7fc0b1400640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09406bc20 0x7fc09406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:39.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.488+0000 7fc0b1400640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09406bc20 0x7fc09406e0e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0036000 tx=0x7fc0a0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:39.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.490+0000 7fc0aa000640 1 -- 172.21.15.102:0/1435700832 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fc0a40567c0 con 0x7fc0b40c9be0 2024-03-18T16:24:39.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:39.796+0000 7fc0aa000640 1 -- 172.21.15.102:0/1435700832 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc0b40733c0 con 0x7fc0b40c9be0 2024-03-18T16:24:40.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.038+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0b40b5a90 con 0x7fc0b40c9be0 2024-03-18T16:24:40.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.052+0000 7fc0aa000640 1 -- 172.21.15.102:0/1435700832 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fc0a4021020 con 0x7fc0b40c9be0 2024-03-18T16:24:40.054 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:40.054 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:40.055 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09406bc20 msgr2=0x7fc09406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09406bc20 0x7fc09406e0e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0036000 tx=0x7fc0a0030000 comp rx=0 tx=0).stop 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 msgr2=0x7fc0b4157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 0x7fc0b4157c40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc0a400c400 tx=0x7fc0a400c8c0 comp rx=0 tx=0).stop 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 shutdown_connections 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc09406bc20 0x7fc09406e0e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0b415ae70 0x7fc0b40b52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0b40d05a0 0x7fc0b40c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 --2- 172.21.15.102:0/1435700832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b40c9be0 0x7fc0b4157c40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:40.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.055+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 >> 172.21.15.102:0/1435700832 conn(0x7fc0b40673a0 msgr2=0x7fc0b4154000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:40.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.056+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 shutdown_connections 2024-03-18T16:24:40.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:40.056+0000 7fc0b3200640 1 -- 172.21.15.102:0/1435700832 wait complete. 2024-03-18T16:24:40.678 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:24:40.678 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:24:41.679 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.856+0000 7f8579600640 1 Processor -- start 2024-03-18T16:24:41.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.856+0000 7f8579600640 1 -- start start 2024-03-18T16:24:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.857+0000 7f8579600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f857415ae70 0x7f85740ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.857+0000 7f8579600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85740c9be0 0x7f85740d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.857+0000 7f8579600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f85740d05a0 0x7f85740cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.857+0000 7f8579600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8574154ab0 con 0x7f85740d05a0 2024-03-18T16:24:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.857+0000 7f8579600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f857415b250 con 0x7f857415ae70 2024-03-18T16:24:41.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.857+0000 7f8579600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f857406d2f0 con 0x7f85740c9be0 2024-03-18T16:24:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.858+0000 7f8573e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f857415ae70 0x7f85740ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.858+0000 7f8578c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f85740d05a0 0x7f85740cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.858+0000 7f8573e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f857415ae70 0x7f85740ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47996/0 (socket says 172.21.15.102:47996) 2024-03-18T16:24:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.858+0000 7f8573e00640 1 -- 172.21.15.102:0/2968020367 learned_addr learned my addr 172.21.15.102:0/2968020367 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:41.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.858+0000 7f8573400640 1 --2- 172.21.15.102:0/2968020367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85740c9be0 0x7f85740d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:41.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.861+0000 7f8573e00640 1 -- 172.21.15.102:0/2968020367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85740c9be0 msgr2=0x7f85740d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:41.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.861+0000 7f8573e00640 1 --2- 172.21.15.102:0/2968020367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85740c9be0 0x7f85740d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.861+0000 7f8573e00640 1 -- 172.21.15.102:0/2968020367 >> v2:172.21.15.102:3300/0 conn(0x7f85740d05a0 msgr2=0x7f85740cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.861+0000 7f8573e00640 1 --2- 172.21.15.102:0/2968020367 >> v2:172.21.15.102:3300/0 conn(0x7f85740d05a0 0x7f85740cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.861+0000 7f8573e00640 1 -- 172.21.15.102:0/2968020367 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857406d5d0 con 0x7f857415ae70 2024-03-18T16:24:41.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.863+0000 7f8573e00640 1 --2- 172.21.15.102:0/2968020367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f857415ae70 0x7f85740ca490 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f85740cd5c0 tx=0x7f856402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=916cd8593c2eb716 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:41.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.868+0000 7f8572a00640 1 -- 172.21.15.102:0/2968020367 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564035020 con 0x7f857415ae70 2024-03-18T16:24:41.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.868+0000 7f8572a00640 1 -- 172.21.15.102:0/2968020367 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8564031870 con 0x7f857415ae70 2024-03-18T16:24:41.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.868+0000 7f8572a00640 1 -- 172.21.15.102:0/2968020367 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8564031c40 con 0x7f857415ae70 2024-03-18T16:24:41.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.869+0000 7f8579600640 1 -- 172.21.15.102:0/2968020367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f857415ae70 msgr2=0x7f85740ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:41.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.869+0000 7f8579600640 1 --2- 172.21.15.102:0/2968020367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f857415ae70 0x7f85740ca490 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f85740cd5c0 tx=0x7f856402b910 comp rx=0 tx=0).stop 2024-03-18T16:24:41.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.869+0000 7f8579600640 1 -- 172.21.15.102:0/2968020367 shutdown_connections 2024-03-18T16:24:41.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.869+0000 7f8579600640 1 --2- 172.21.15.102:0/2968020367 >> v2:172.21.15.102:3300/0 conn(0x7f85740d05a0 0x7f85740cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:41.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.869+0000 7f8579600640 1 --2- 172.21.15.102:0/2968020367 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85740c9be0 0x7f85740d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:41.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.869+0000 7f8579600640 1 --2- 172.21.15.102:0/2968020367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f857415ae70 0x7f85740ca490 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:41.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.869+0000 7f8579600640 1 -- 172.21.15.102:0/2968020367 >> 172.21.15.102:0/2968020367 conn(0x7f85740673a0 msgr2=0x7f85740677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:41.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.869+0000 7f8579600640 1 -- 172.21.15.102:0/2968020367 shutdown_connections 2024-03-18T16:24:41.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.869+0000 7f8579600640 1 -- 172.21.15.102:0/2968020367 wait complete. 2024-03-18T16:24:41.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.871+0000 7f8579600640 1 Processor -- start 2024-03-18T16:24:41.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.871+0000 7f8579600640 1 -- start start 2024-03-18T16:24:41.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.872+0000 7f8579600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85740c9be0 0x7f85740b5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:41.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.872+0000 7f8579600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85740d05a0 0x7f8574159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:41.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.872+0000 7f8579600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f857415ae70 0x7f857414b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:41.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.872+0000 7f8579600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f857406de70 con 0x7f85740c9be0 2024-03-18T16:24:41.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.872+0000 7f8579600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f857406db90 con 0x7f85740d05a0 2024-03-18T16:24:41.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.872+0000 7f8579600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f857406e150 con 0x7f857415ae70 2024-03-18T16:24:41.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.873+0000 7f8573400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85740d05a0 0x7f8574159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:41.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.873+0000 7f8573e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85740c9be0 0x7f85740b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:41.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.873+0000 7f8578c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f857415ae70 0x7f857414b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:41.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.873+0000 7f8573400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85740d05a0 0x7f8574159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48010/0 (socket says 172.21.15.102:48010) 2024-03-18T16:24:41.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.873+0000 7f8573400640 1 -- 172.21.15.102:0/2479965308 learned_addr learned my addr 172.21.15.102:0/2479965308 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:41.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.876+0000 7f8573400640 1 -- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f857415ae70 msgr2=0x7f857414b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:41.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.876+0000 7f8573400640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f857415ae70 0x7f857414b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:41.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.876+0000 7f8573400640 1 -- 172.21.15.102:0/2479965308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85740c9be0 msgr2=0x7f85740b5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:41.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.876+0000 7f8573400640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85740c9be0 0x7f85740b5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:41.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.876+0000 7f8573400640 1 -- 172.21.15.102:0/2479965308 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857406c9c0 con 0x7f85740d05a0 2024-03-18T16:24:41.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.879+0000 7f8573400640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85740d05a0 0x7f8574159ca0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f856000e870 tx=0x7f856000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:41.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.883+0000 7f856be00640 1 -- 172.21.15.102:0/2479965308 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8560017070 con 0x7f85740d05a0 2024-03-18T16:24:41.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.883+0000 7f856be00640 1 -- 172.21.15.102:0/2479965308 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f856000c940 con 0x7f85740d05a0 2024-03-18T16:24:41.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.883+0000 7f856be00640 1 -- 172.21.15.102:0/2479965308 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f856000cd30 con 0x7f85740d05a0 2024-03-18T16:24:41.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.883+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85740b5660 con 0x7f85740d05a0 2024-03-18T16:24:41.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.883+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f857414c8e0 con 0x7f85740d05a0 2024-03-18T16:24:41.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.884+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8574074fd0 con 0x7f85740d05a0 2024-03-18T16:24:41.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.903+0000 7f856be00640 1 -- 172.21.15.102:0/2479965308 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f856001a070 con 0x7f85740d05a0 2024-03-18T16:24:41.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.904+0000 7f856be00640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f855406bc20 0x7f855406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:41.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.905+0000 7f8573e00640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f855406bc20 0x7f855406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:41.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:41.906+0000 7f8573e00640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f855406bc20 0x7f855406e0e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8564036000 tx=0x7f856403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:42.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.144+0000 7f856be00640 1 -- 172.21.15.102:0/2479965308 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8560057550 con 0x7f85740d05a0 2024-03-18T16:24:42.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.155+0000 7f856be00640 1 -- 172.21.15.102:0/2479965308 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f857406c9c0 con 0x7f85740d05a0 2024-03-18T16:24:42.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.412+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85740ccdb0 con 0x7f85740d05a0 2024-03-18T16:24:42.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.425+0000 7f856be00640 1 -- 172.21.15.102:0/2479965308 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f85600594b0 con 0x7f85740d05a0 2024-03-18T16:24:42.427 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:42.428 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:42.428 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:42.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f855406bc20 msgr2=0x7f855406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:42.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f855406bc20 0x7f855406e0e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8564036000 tx=0x7f856403c040 comp rx=0 tx=0).stop 2024-03-18T16:24:42.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85740d05a0 msgr2=0x7f8574159ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:42.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85740d05a0 0x7f8574159ca0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f856000e870 tx=0x7f856000ed30 comp rx=0 tx=0).stop 2024-03-18T16:24:42.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 shutdown_connections 2024-03-18T16:24:42.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f855406bc20 0x7f855406e0e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:42.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f857415ae70 0x7f857414b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:42.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85740d05a0 0x7f8574159ca0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:42.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 --2- 172.21.15.102:0/2479965308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85740c9be0 0x7f85740b5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:42.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 >> 172.21.15.102:0/2479965308 conn(0x7f85740673a0 msgr2=0x7f85740cdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:42.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 shutdown_connections 2024-03-18T16:24:42.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:42.428+0000 7f8579600640 1 -- 172.21.15.102:0/2479965308 wait complete. 2024-03-18T16:24:43.051 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:24:43.051 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:24:44.052 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:44.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.226+0000 7f8005200640 1 Processor -- start 2024-03-18T16:24:44.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.227+0000 7f8005200640 1 -- start start 2024-03-18T16:24:44.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.228+0000 7f8005200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f800015ae70 0x7f80000ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:44.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.228+0000 7f8005200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f80000c9be0 0x7f80000d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:44.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.228+0000 7f8005200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8000154ab0 con 0x7f80000d05a0 2024-03-18T16:24:44.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.229+0000 7f8005200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f800006d2f0 con 0x7f80000c9be0 2024-03-18T16:24:44.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.229+0000 7f8005200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f800006d5d0 con 0x7f800015ae70 2024-03-18T16:24:44.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.229+0000 7f7fff400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f800015ae70 0x7f80000ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:44.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.229+0000 7f7ffea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f80000c9be0 0x7f80000d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:44.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.230+0000 7f7fff400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f800015ae70 0x7f80000ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:35862/0 (socket says 172.21.15.102:35862) 2024-03-18T16:24:44.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.230+0000 7f7fff400640 1 -- 172.21.15.102:0/793597845 learned_addr learned my addr 172.21.15.102:0/793597845 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:44.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.232+0000 7f7fff400640 1 -- 172.21.15.102:0/793597845 >> v2:172.21.15.154:3300/0 conn(0x7f80000c9be0 msgr2=0x7f80000d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:44.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.232+0000 7f7fff400640 1 --2- 172.21.15.102:0/793597845 >> v2:172.21.15.154:3300/0 conn(0x7f80000c9be0 0x7f80000d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.232+0000 7f7fff400640 1 -- 172.21.15.102:0/793597845 >> v1:172.21.15.102:6789/0 conn(0x7f80000d05a0 legacy=0x7f80000cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:44.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.232+0000 7f7fff400640 1 -- 172.21.15.102:0/793597845 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800015b250 con 0x7f800015ae70 2024-03-18T16:24:44.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.234+0000 7f7fff400640 1 --2- 172.21.15.102:0/793597845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f800015ae70 0x7f80000ca490 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009870 tx=0x7f7ff002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b499725039dd0336 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:44.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.238+0000 7f7ffe000640 1 -- 172.21.15.102:0/793597845 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0034070 con 0x7f800015ae70 2024-03-18T16:24:44.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.238+0000 7f7ffe000640 1 -- 172.21.15.102:0/793597845 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff0030840 con 0x7f800015ae70 2024-03-18T16:24:44.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.238+0000 7f7ffe000640 1 -- 172.21.15.102:0/793597845 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0030c10 con 0x7f800015ae70 2024-03-18T16:24:44.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.239+0000 7f8005200640 1 -- 172.21.15.102:0/793597845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f800015ae70 msgr2=0x7f80000ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:44.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.239+0000 7f8005200640 1 --2- 172.21.15.102:0/793597845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f800015ae70 0x7f80000ca490 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009870 tx=0x7f7ff002b980 comp rx=0 tx=0).stop 2024-03-18T16:24:44.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.239+0000 7f8005200640 1 -- 172.21.15.102:0/793597845 shutdown_connections 2024-03-18T16:24:44.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.239+0000 7f8005200640 1 --2- 172.21.15.102:0/793597845 >> v2:172.21.15.154:3300/0 conn(0x7f80000c9be0 0x7f80000d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.239+0000 7f8005200640 1 --2- 172.21.15.102:0/793597845 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f800015ae70 0x7f80000ca490 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.239+0000 7f8005200640 1 -- 172.21.15.102:0/793597845 >> 172.21.15.102:0/793597845 conn(0x7f80000673a0 msgr2=0x7f80000677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:44.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.239+0000 7f8005200640 1 -- 172.21.15.102:0/793597845 shutdown_connections 2024-03-18T16:24:44.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.239+0000 7f8005200640 1 -- 172.21.15.102:0/793597845 wait complete. 2024-03-18T16:24:44.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.241+0000 7f8005200640 1 Processor -- start 2024-03-18T16:24:44.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.242+0000 7f8005200640 1 -- start start 2024-03-18T16:24:44.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.243+0000 7f8005200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80000c9be0 0x7f80001513e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:44.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.243+0000 7f8005200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000d05a0 0x7f80000c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:44.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.243+0000 7f8005200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f800015ae70 0x7f80000c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:44.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.243+0000 7f8005200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f800006de70 con 0x7f800015ae70 2024-03-18T16:24:44.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.243+0000 7f8005200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f800006db90 con 0x7f80000d05a0 2024-03-18T16:24:44.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.243+0000 7f8005200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f800006e150 con 0x7f80000c9be0 2024-03-18T16:24:44.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.244+0000 7f7ffea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000d05a0 0x7f80000c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:44.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.244+0000 7f7fff400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80000c9be0 0x7f80001513e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:44.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.244+0000 7f7fffe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f800015ae70 0x7f80000c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:44.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.244+0000 7f7ffea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000d05a0 0x7f80000c7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48030/0 (socket says 172.21.15.102:48030) 2024-03-18T16:24:44.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.244+0000 7f7ffea00640 1 -- 172.21.15.102:0/1687996310 learned_addr learned my addr 172.21.15.102:0/1687996310 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:44.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.248+0000 7f7fff400640 1 -- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000d05a0 msgr2=0x7f80000c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:44.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.248+0000 7f7fff400640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000d05a0 0x7f80000c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.248+0000 7f7fff400640 1 -- 172.21.15.102:0/1687996310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f800015ae70 msgr2=0x7f80000c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:44.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.248+0000 7f7fff400640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f800015ae70 0x7f80000c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.248+0000 7f7fff400640 1 -- 172.21.15.102:0/1687996310 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8000157290 con 0x7f80000c9be0 2024-03-18T16:24:44.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.250+0000 7f7fff400640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80000c9be0 0x7f80001513e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0033000 tx=0x7f7ff00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:44.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.254+0000 7f7ff7400640 1 -- 172.21.15.102:0/1687996310 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0034040 con 0x7f80000c9be0 2024-03-18T16:24:44.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.254+0000 7f7ff7400640 1 -- 172.21.15.102:0/1687996310 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff0030640 con 0x7f80000c9be0 2024-03-18T16:24:44.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.254+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80001557a0 con 0x7f80000c9be0 2024-03-18T16:24:44.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.254+0000 7f7ff7400640 1 -- 172.21.15.102:0/1687996310 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff00423f0 con 0x7f80000c9be0 2024-03-18T16:24:44.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.254+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8000155d40 con 0x7f80000c9be0 2024-03-18T16:24:44.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.256+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8000074fd0 con 0x7f80000c9be0 2024-03-18T16:24:44.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.275+0000 7f7ff7400640 1 -- 172.21.15.102:0/1687996310 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7ff0037070 con 0x7f80000c9be0 2024-03-18T16:24:44.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.277+0000 7f7ff7400640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fe006bc20 0x7f7fe006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:44.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.277+0000 7f7ffea00640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fe006bc20 0x7f7fe006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:44.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.278+0000 7f7ff7400640 1 -- 172.21.15.102:0/1687996310 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7ff00307f0 con 0x7f80000c9be0 2024-03-18T16:24:44.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.278+0000 7f7ffea00640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fe006bc20 0x7f7fe006e0e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f80000c8ba0 tx=0x7f7fec009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:44.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.527+0000 7f7ff7400640 1 -- 172.21.15.102:0/1687996310 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7ff007bc20 con 0x7f80000c9be0 2024-03-18T16:24:44.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.765+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8000151920 con 0x7f80000c9be0 2024-03-18T16:24:44.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.778+0000 7f7ff7400640 1 -- 172.21.15.102:0/1687996310 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7ff007fcc0 con 0x7f80000c9be0 2024-03-18T16:24:44.780 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:44.780 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:44.781 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:44.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fe006bc20 msgr2=0x7f7fe006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:44.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fe006bc20 0x7f7fe006e0e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f80000c8ba0 tx=0x7f7fec009190 comp rx=0 tx=0).stop 2024-03-18T16:24:44.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80000c9be0 msgr2=0x7f80001513e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:44.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80000c9be0 0x7f80001513e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0033000 tx=0x7f7ff00046c0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 shutdown_connections 2024-03-18T16:24:44.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7fe006bc20 0x7f7fe006e0e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f800015ae70 0x7f80000c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000d05a0 0x7f80000c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 --2- 172.21.15.102:0/1687996310 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80000c9be0 0x7f80001513e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:44.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.780+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 >> 172.21.15.102:0/1687996310 conn(0x7f80000673a0 msgr2=0x7f8000067780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:44.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.781+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 shutdown_connections 2024-03-18T16:24:44.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:44.781+0000 7f8005200640 1 -- 172.21.15.102:0/1687996310 wait complete. 2024-03-18T16:24:45.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:24:45.405 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:24:46.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:46.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.576+0000 7fda50a00640 1 Processor -- start 2024-03-18T16:24:46.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.576+0000 7fda50a00640 1 -- start start 2024-03-18T16:24:46.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.577+0000 7fda50a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fda4c0cfe70 0x7fda4c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:46.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.577+0000 7fda50a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0cc2f0 0x7fda4c0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:46.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.577+0000 7fda50a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4c0cabd0 con 0x7fda4c0d0250 2024-03-18T16:24:46.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.577+0000 7fda50a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fda4c0caea0 con 0x7fda4c0cfe70 2024-03-18T16:24:46.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.577+0000 7fda50a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fda4c0cb180 con 0x7fda4c0cc2f0 2024-03-18T16:24:46.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.578+0000 7fda4a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fda4c0cfe70 0x7fda4c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:46.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.578+0000 7fda49e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fda4c0d0250 0x7fda4c0c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:52280/0 (socket says 172.21.15.102:52280) 2024-03-18T16:24:46.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.578+0000 7fda49e00640 1 -- 172.21.15.102:0/376196435 learned_addr learned my addr 172.21.15.102:0/376196435 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:24:46.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.578+0000 7fda4b200640 1 --2- 172.21.15.102:0/376196435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0cc2f0 0x7fda4c0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:46.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.581+0000 7fda49400640 1 -- 172.21.15.102:0/376196435 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834408613 0 0) 0x7fda4c0cabd0 con 0x7fda4c0d0250 2024-03-18T16:24:46.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.581+0000 7fda49400640 1 -- 172.21.15.102:0/376196435 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda30003520 con 0x7fda4c0d0250 2024-03-18T16:24:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.581+0000 7fda4a800640 1 -- 172.21.15.102:0/376196435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0cc2f0 msgr2=0x7fda4c0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.581+0000 7fda4a800640 1 --2- 172.21.15.102:0/376196435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0cc2f0 0x7fda4c0cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.581+0000 7fda4a800640 1 -- 172.21.15.102:0/376196435 >> v1:172.21.15.154:6789/0 conn(0x7fda4c0d0250 legacy=0x7fda4c0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.581+0000 7fda4a800640 1 -- 172.21.15.102:0/376196435 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c0cb460 con 0x7fda4c0cfe70 2024-03-18T16:24:46.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.584+0000 7fda4a800640 1 --2- 172.21.15.102:0/376196435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda4c0cfe70 0x7fda4c15ae40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c06e6db6222d56a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:46.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.586+0000 7fda49400640 1 -- 172.21.15.102:0/376196435 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c034070 con 0x7fda4c0cfe70 2024-03-18T16:24:46.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.589+0000 7fda49400640 1 -- 172.21.15.102:0/376196435 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda3c030800 con 0x7fda4c0cfe70 2024-03-18T16:24:46.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.589+0000 7fda49400640 1 -- 172.21.15.102:0/376196435 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c030bb0 con 0x7fda4c0cfe70 2024-03-18T16:24:46.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.590+0000 7fda50a00640 1 -- 172.21.15.102:0/376196435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda4c0cfe70 msgr2=0x7fda4c15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:46.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.590+0000 7fda50a00640 1 --2- 172.21.15.102:0/376196435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda4c0cfe70 0x7fda4c15ae40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0097a0 tx=0x7fda3c02b940 comp rx=0 tx=0).stop 2024-03-18T16:24:46.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.590+0000 7fda50a00640 1 -- 172.21.15.102:0/376196435 shutdown_connections 2024-03-18T16:24:46.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.590+0000 7fda50a00640 1 --2- 172.21.15.102:0/376196435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0cc2f0 0x7fda4c0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:46.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.590+0000 7fda50a00640 1 --2- 172.21.15.102:0/376196435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda4c0cfe70 0x7fda4c15ae40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:46.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.590+0000 7fda50a00640 1 -- 172.21.15.102:0/376196435 >> 172.21.15.102:0/376196435 conn(0x7fda4c08c3e0 msgr2=0x7fda4c06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:46.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.590+0000 7fda50a00640 1 -- 172.21.15.102:0/376196435 shutdown_connections 2024-03-18T16:24:46.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.590+0000 7fda50a00640 1 -- 172.21.15.102:0/376196435 wait complete. 2024-03-18T16:24:46.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.592+0000 7fda50a00640 1 Processor -- start 2024-03-18T16:24:46.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.592+0000 7fda50a00640 1 -- start start 2024-03-18T16:24:46.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.593+0000 7fda50a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda4c0cc2f0 0x7fda4c1e2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:46.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.593+0000 7fda50a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda4c0cfe70 0x7fda4c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:46.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.593+0000 7fda50a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0d0250 0x7fda4c156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:46.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.593+0000 7fda50a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c0cbd00 con 0x7fda4c0cc2f0 2024-03-18T16:24:46.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.593+0000 7fda50a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c0cba20 con 0x7fda4c0cfe70 2024-03-18T16:24:46.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.593+0000 7fda50a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fda4c0ccc50 con 0x7fda4c0d0250 2024-03-18T16:24:46.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.593+0000 7fda49e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda4c0cfe70 0x7fda4c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:46.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.593+0000 7fda4a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda4c0cc2f0 0x7fda4c1e2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:46.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.594+0000 7fda4b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0d0250 0x7fda4c156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:46.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.594+0000 7fda49e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda4c0cfe70 0x7fda4c155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48038/0 (socket says 172.21.15.102:48038) 2024-03-18T16:24:46.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.594+0000 7fda49e00640 1 -- 172.21.15.102:0/4076618389 learned_addr learned my addr 172.21.15.102:0/4076618389 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:46.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.597+0000 7fda49e00640 1 -- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0d0250 msgr2=0x7fda4c156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:46.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.597+0000 7fda49e00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0d0250 0x7fda4c156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:46.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.597+0000 7fda49e00640 1 -- 172.21.15.102:0/4076618389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda4c0cc2f0 msgr2=0x7fda4c1e2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:46.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.597+0000 7fda49e00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda4c0cc2f0 0x7fda4c1e2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:46.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.597+0000 7fda49e00640 1 -- 172.21.15.102:0/4076618389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c1e1d30 con 0x7fda4c0cfe70 2024-03-18T16:24:46.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.600+0000 7fda49e00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda4c0cfe70 0x7fda4c155e40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c390 tx=0x7fda3800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:46.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.603+0000 7fda42a00640 1 -- 172.21.15.102:0/4076618389 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38017070 con 0x7fda4c0cfe70 2024-03-18T16:24:46.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.603+0000 7fda42a00640 1 -- 172.21.15.102:0/4076618389 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda38005d00 con 0x7fda4c0cfe70 2024-03-18T16:24:46.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.603+0000 7fda42a00640 1 -- 172.21.15.102:0/4076618389 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38007310 con 0x7fda4c0cfe70 2024-03-18T16:24:46.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.603+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4c159370 con 0x7fda4c0cfe70 2024-03-18T16:24:46.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.604+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4c159930 con 0x7fda4c0cfe70 2024-03-18T16:24:46.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.605+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda4c159f40 con 0x7fda4c0cfe70 2024-03-18T16:24:46.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.863+0000 7fda42a00640 1 -- 172.21.15.102:0/4076618389 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fda3801a070 con 0x7fda4c0cfe70 2024-03-18T16:24:46.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.865+0000 7fda42a00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fda2c06bc50 0x7fda2c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:46.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.866+0000 7fda42a00640 1 -- 172.21.15.102:0/4076618389 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fda3808a510 con 0x7fda4c0cfe70 2024-03-18T16:24:46.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.866+0000 7fda4a800640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fda2c06bc50 0x7fda2c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:46.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.867+0000 7fda4a800640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fda2c06bc50 0x7fda2c06e110 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fda3c036000 tx=0x7fda3c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:46.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:46.877+0000 7fda42a00640 1 -- 172.21.15.102:0/4076618389 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fda38021020 con 0x7fda4c0cfe70 2024-03-18T16:24:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.165+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda4c04f630 con 0x7fda4c0cfe70 2024-03-18T16:24:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.234+0000 7fda42a00640 1 -- 172.21.15.102:0/4076618389 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fda4c04f630 con 0x7fda4c0cfe70 2024-03-18T16:24:47.236 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:47.236 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.236+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fda2c06bc50 msgr2=0x7fda2c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.236+0000 7fda50a00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fda2c06bc50 0x7fda2c06e110 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fda3c036000 tx=0x7fda3c008030 comp rx=0 tx=0).stop 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.236+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda4c0cfe70 msgr2=0x7fda4c155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.236+0000 7fda50a00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda4c0cfe70 0x7fda4c155e40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c390 tx=0x7fda3800c850 comp rx=0 tx=0).stop 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.237+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 shutdown_connections 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.237+0000 7fda50a00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fda2c06bc50 0x7fda2c06e110 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.237+0000 7fda50a00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fda4c0d0250 0x7fda4c156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.237+0000 7fda50a00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda4c0cfe70 0x7fda4c155e40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.237+0000 7fda50a00640 1 --2- 172.21.15.102:0/4076618389 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fda4c0cc2f0 0x7fda4c1e2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:47.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.237+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 >> 172.21.15.102:0/4076618389 conn(0x7fda4c08c3e0 msgr2=0x7fda4c0c6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:47.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.237+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 shutdown_connections 2024-03-18T16:24:47.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:47.237+0000 7fda50a00640 1 -- 172.21.15.102:0/4076618389 wait complete. 2024-03-18T16:24:47.860 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:24:47.860 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:24:48.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:49.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.034+0000 7f563fe00640 1 Processor -- start 2024-03-18T16:24:49.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.034+0000 7f563fe00640 1 -- start start 2024-03-18T16:24:49.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.035+0000 7f563fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 0x7f564015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:49.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.035+0000 7f563fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56400c72d0 con 0x7f56400c4fa0 2024-03-18T16:24:49.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.036+0000 7f563fe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5640158840 con 0x7f5640141650 2024-03-18T16:24:49.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.036+0000 7f563fe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f564015c8e0 con 0x7f5640158420 2024-03-18T16:24:49.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.036+0000 7f563e000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 0x7f564015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:49.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.036+0000 7f563ea00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f5640141650 0x7f56400c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:52294/0 (socket says 172.21.15.102:52294) 2024-03-18T16:24:49.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.036+0000 7f563ea00640 1 -- 172.21.15.102:0/3126496329 learned_addr learned my addr 172.21.15.102:0/3126496329 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:24:49.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.040+0000 7f563e000640 1 -- 172.21.15.102:0/3126496329 >> v1:172.21.15.154:6789/0 conn(0x7f5640141650 legacy=0x7f56400c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:49.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.040+0000 7f563e000640 1 -- 172.21.15.102:0/3126496329 >> v1:172.21.15.102:6789/0 conn(0x7f56400c4fa0 legacy=0x7f5640162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:49.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.040+0000 7f563e000640 1 -- 172.21.15.102:0/3126496329 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5640158ad0 con 0x7f5640158420 2024-03-18T16:24:49.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.042+0000 7f563e000640 1 --2- 172.21.15.102:0/3126496329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 0x7f564015c3a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f562c0097d0 tx=0x7f562c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=821e40b517dda455 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.046+0000 7f563d600640 1 -- 172.21.15.102:0/3126496329 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562c035020 con 0x7f5640158420 2024-03-18T16:24:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.046+0000 7f563d600640 1 -- 172.21.15.102:0/3126496329 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f562c031890 con 0x7f5640158420 2024-03-18T16:24:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.046+0000 7f563d600640 1 -- 172.21.15.102:0/3126496329 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562c031c60 con 0x7f5640158420 2024-03-18T16:24:49.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.047+0000 7f563fe00640 1 -- 172.21.15.102:0/3126496329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 msgr2=0x7f564015c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:49.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.047+0000 7f563fe00640 1 --2- 172.21.15.102:0/3126496329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 0x7f564015c3a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f562c0097d0 tx=0x7f562c02b930 comp rx=0 tx=0).stop 2024-03-18T16:24:49.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.047+0000 7f563fe00640 1 -- 172.21.15.102:0/3126496329 shutdown_connections 2024-03-18T16:24:49.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.047+0000 7f563fe00640 1 --2- 172.21.15.102:0/3126496329 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 0x7f564015c3a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:49.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.047+0000 7f563fe00640 1 -- 172.21.15.102:0/3126496329 >> 172.21.15.102:0/3126496329 conn(0x7f56400cb890 msgr2=0x7f56400cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:49.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.047+0000 7f563fe00640 1 -- 172.21.15.102:0/3126496329 shutdown_connections 2024-03-18T16:24:49.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.047+0000 7f563fe00640 1 -- 172.21.15.102:0/3126496329 wait complete. 2024-03-18T16:24:49.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.049+0000 7f563fe00640 1 Processor -- start 2024-03-18T16:24:49.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.049+0000 7f563fe00640 1 -- start start 2024-03-18T16:24:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.050+0000 7f563fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56400c4fa0 0x7f5640153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.050+0000 7f563fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5640141650 0x7f5640152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.050+0000 7f563fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 0x7f56401f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.050+0000 7f563fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5640159090 con 0x7f56400c4fa0 2024-03-18T16:24:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.050+0000 7f563fe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5640159370 con 0x7f5640141650 2024-03-18T16:24:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.050+0000 7f563fe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f56400c63c0 con 0x7f5640158420 2024-03-18T16:24:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.051+0000 7f563e000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5640141650 0x7f5640152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.051+0000 7f563ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56400c4fa0 0x7f5640153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.051+0000 7f563f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 0x7f56401f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.051+0000 7f563ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56400c4fa0 0x7f5640153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34108/0 (socket says 172.21.15.102:34108) 2024-03-18T16:24:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.051+0000 7f563ea00640 1 -- 172.21.15.102:0/1608073644 learned_addr learned my addr 172.21.15.102:0/1608073644 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.055+0000 7f563e000640 1 -- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 msgr2=0x7f56401f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.055+0000 7f563e000640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 0x7f56401f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.055+0000 7f563e000640 1 -- 172.21.15.102:0/1608073644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56400c4fa0 msgr2=0x7f5640153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:49.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.055+0000 7f563e000640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56400c4fa0 0x7f5640153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:49.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.055+0000 7f563e000640 1 -- 172.21.15.102:0/1608073644 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564015afb0 con 0x7f5640141650 2024-03-18T16:24:49.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.057+0000 7f563e000640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5640141650 0x7f5640152d90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f562c033000 tx=0x7f562c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:49.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.060+0000 7f5636a00640 1 -- 172.21.15.102:0/1608073644 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562c040070 con 0x7f5640141650 2024-03-18T16:24:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.061+0000 7f5636a00640 1 -- 172.21.15.102:0/1608073644 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f562c03a420 con 0x7f5640141650 2024-03-18T16:24:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.061+0000 7f5636a00640 1 -- 172.21.15.102:0/1608073644 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562c035040 con 0x7f5640141650 2024-03-18T16:24:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.061+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f564015abb0 con 0x7f5640141650 2024-03-18T16:24:49.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.061+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5640159990 con 0x7f5640141650 2024-03-18T16:24:49.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.062+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5640067c20 con 0x7f5640141650 2024-03-18T16:24:49.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.081+0000 7f5636a00640 1 -- 172.21.15.102:0/1608073644 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f562c038020 con 0x7f5640141650 2024-03-18T16:24:49.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.082+0000 7f5636a00640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f562006bc50 0x7f562006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:49.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.083+0000 7f563ea00640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f562006bc50 0x7f562006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:49.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.083+0000 7f563ea00640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f562006bc50 0x7f562006e110 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5630005880 tx=0x7f5630009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:49.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.323+0000 7f5636a00640 1 -- 172.21.15.102:0/1608073644 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f562c079240 con 0x7f5640141650 2024-03-18T16:24:49.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.335+0000 7f5636a00640 1 -- 172.21.15.102:0/1608073644 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5640067c20 con 0x7f5640141650 2024-03-18T16:24:49.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.627+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5640074fd0 con 0x7f5640141650 2024-03-18T16:24:49.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.656+0000 7f5636a00640 1 -- 172.21.15.102:0/1608073644 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f564015afb0 con 0x7f5640141650 2024-03-18T16:24:49.658 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:49.659 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:49.659 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:49.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.658+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f562006bc50 msgr2=0x7f562006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:49.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.658+0000 7f563fe00640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f562006bc50 0x7f562006e110 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5630005880 tx=0x7f5630009040 comp rx=0 tx=0).stop 2024-03-18T16:24:49.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.658+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5640141650 msgr2=0x7f5640152d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:49.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.658+0000 7f563fe00640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5640141650 0x7f5640152d90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f562c033000 tx=0x7f562c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:24:49.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.658+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 shutdown_connections 2024-03-18T16:24:49.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.659+0000 7f563fe00640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f562006bc50 0x7f562006e110 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:49.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.659+0000 7f563fe00640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5640158420 0x7f56401f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:49.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.659+0000 7f563fe00640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5640141650 0x7f5640152d90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:49.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.659+0000 7f563fe00640 1 --2- 172.21.15.102:0/1608073644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56400c4fa0 0x7f5640153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:49.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.659+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 >> 172.21.15.102:0/1608073644 conn(0x7f56400cb890 msgr2=0x7f56400cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:49.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.659+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 shutdown_connections 2024-03-18T16:24:49.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:49.659+0000 7f563fe00640 1 -- 172.21.15.102:0/1608073644 wait complete. 2024-03-18T16:24:50.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:24:50.282 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:24:51.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:24:51.207 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:24:51.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.453+0000 7f38ede00640 1 Processor -- start 2024-03-18T16:24:51.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.453+0000 7f38ede00640 1 -- start start 2024-03-18T16:24:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.454+0000 7f38ede00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f38e8158420 0x7f38e815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.454+0000 7f38ede00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e80c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.454+0000 7f38ede00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38e80c72d0 con 0x7f38e8141650 2024-03-18T16:24:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.454+0000 7f38ede00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f38e815c8e0 con 0x7f38e8158420 2024-03-18T16:24:51.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.454+0000 7f38ede00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f38e8159370 con 0x7f38e80c4fa0 2024-03-18T16:24:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.455+0000 7f38e7e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f38e8158420 0x7f38e815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.455+0000 7f38ed400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e80c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.456+0000 7f38ed400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e80c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:54944/0 (socket says 172.21.15.102:54944) 2024-03-18T16:24:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.456+0000 7f38ed400640 1 -- 172.21.15.102:0/2300454740 learned_addr learned my addr 172.21.15.102:0/2300454740 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:51.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.459+0000 7f38e7e00640 1 -- 172.21.15.102:0/2300454740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 msgr2=0x7f38e80c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:51.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.459+0000 7f38e7e00640 1 --2- 172.21.15.102:0/2300454740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e80c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:51.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.459+0000 7f38e7e00640 1 -- 172.21.15.102:0/2300454740 >> v1:172.21.15.154:6789/0 conn(0x7f38e8141650 legacy=0x7f38e80c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:51.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.459+0000 7f38e7e00640 1 -- 172.21.15.102:0/2300454740 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e8158840 con 0x7f38e8158420 2024-03-18T16:24:51.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.463+0000 7f38e7e00640 1 --2- 172.21.15.102:0/2300454740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38e8158420 0x7f38e815c3a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f38d40097a0 tx=0x7f38d402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41cf0886f736947a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.468+0000 7f38e7400640 1 -- 172.21.15.102:0/2300454740 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4035020 con 0x7f38e8158420 2024-03-18T16:24:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.468+0000 7f38e7400640 1 -- 172.21.15.102:0/2300454740 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d4031830 con 0x7f38e8158420 2024-03-18T16:24:51.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.468+0000 7f38e7400640 1 -- 172.21.15.102:0/2300454740 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4031c00 con 0x7f38e8158420 2024-03-18T16:24:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.468+0000 7f38ede00640 1 -- 172.21.15.102:0/2300454740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38e8158420 msgr2=0x7f38e815c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.468+0000 7f38ede00640 1 --2- 172.21.15.102:0/2300454740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38e8158420 0x7f38e815c3a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f38d40097a0 tx=0x7f38d402b8d0 comp rx=0 tx=0).stop 2024-03-18T16:24:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.469+0000 7f38ede00640 1 -- 172.21.15.102:0/2300454740 shutdown_connections 2024-03-18T16:24:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.469+0000 7f38ede00640 1 --2- 172.21.15.102:0/2300454740 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e80c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.469+0000 7f38ede00640 1 --2- 172.21.15.102:0/2300454740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38e8158420 0x7f38e815c3a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.469+0000 7f38ede00640 1 -- 172.21.15.102:0/2300454740 >> 172.21.15.102:0/2300454740 conn(0x7f38e80cb890 msgr2=0x7f38e80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.469+0000 7f38ede00640 1 -- 172.21.15.102:0/2300454740 shutdown_connections 2024-03-18T16:24:51.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.469+0000 7f38ede00640 1 -- 172.21.15.102:0/2300454740 wait complete. 2024-03-18T16:24:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.471+0000 7f38ede00640 1 Processor -- start 2024-03-18T16:24:51.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.472+0000 7f38ede00640 1 -- start start 2024-03-18T16:24:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.473+0000 7f38ede00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e813be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.473+0000 7f38ede00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e8141650 0x7f38e81f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.473+0000 7f38ede00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38e8158420 0x7f38e81f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.473+0000 7f38ede00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f38e8159090 con 0x7f38e8141650 2024-03-18T16:24:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.473+0000 7f38ede00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f38e8158db0 con 0x7f38e8158420 2024-03-18T16:24:51.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.473+0000 7f38ede00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f38e80c63c0 con 0x7f38e80c4fa0 2024-03-18T16:24:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.474+0000 7f38ed400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38e8158420 0x7f38e81f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.474+0000 7f38eca00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e813be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.474+0000 7f38e7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e8141650 0x7f38e81f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.474+0000 7f38eca00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e813be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:54948/0 (socket says 172.21.15.102:54948) 2024-03-18T16:24:51.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.474+0000 7f38eca00640 1 -- 172.21.15.102:0/3653154987 learned_addr learned my addr 172.21.15.102:0/3653154987 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.478+0000 7f38e7e00640 1 -- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 msgr2=0x7f38e813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.478+0000 7f38e7e00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e813be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.478+0000 7f38e7e00640 1 -- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38e8158420 msgr2=0x7f38e81f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.478+0000 7f38e7e00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38e8158420 0x7f38e81f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:51.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.478+0000 7f38e7e00640 1 -- 172.21.15.102:0/3653154987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e815a180 con 0x7f38e8141650 2024-03-18T16:24:51.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.481+0000 7f38e7e00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e8141650 0x7f38e81f3960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f38d4033000 tx=0x7f38d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:51.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.484+0000 7f38e4c00640 1 -- 172.21.15.102:0/3653154987 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4040070 con 0x7f38e8141650 2024-03-18T16:24:51.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.484+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e8140540 con 0x7f38e8141650 2024-03-18T16:24:51.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.484+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e815a700 con 0x7f38e8141650 2024-03-18T16:24:51.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.485+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38e8074fd0 con 0x7f38e8141650 2024-03-18T16:24:51.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.486+0000 7f38e4c00640 1 -- 172.21.15.102:0/3653154987 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d403e070 con 0x7f38e8141650 2024-03-18T16:24:51.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.486+0000 7f38e4c00640 1 -- 172.21.15.102:0/3653154987 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4035040 con 0x7f38e8141650 2024-03-18T16:24:51.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.764+0000 7f38e4c00640 1 -- 172.21.15.102:0/3653154987 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f38d4038020 con 0x7f38e8141650 2024-03-18T16:24:51.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.765+0000 7f38e4c00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38c806bbd0 0x7f38c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:51.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.765+0000 7f38eca00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38c806bbd0 0x7f38c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:51.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.766+0000 7f38eca00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38c806bbd0 0x7f38c806e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f38dc001840 tx=0x7f38dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:51.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.788+0000 7f38e4c00640 1 -- 172.21.15.102:0/3653154987 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f38d4078dd0 con 0x7f38e8141650 2024-03-18T16:24:51.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:51.788+0000 7f38e4c00640 1 -- 172.21.15.102:0/3653154987 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f38d4053020 con 0x7f38e8141650 2024-03-18T16:24:52.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.203+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38e80b7f00 con 0x7f38e8141650 2024-03-18T16:24:52.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.229+0000 7f38e4c00640 1 -- 172.21.15.102:0/3653154987 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f38d4031800 con 0x7f38e8141650 2024-03-18T16:24:52.231 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:52.232 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:52.232 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:52.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.232+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38c806bbd0 msgr2=0x7f38c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:52.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.232+0000 7f38ede00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38c806bbd0 0x7f38c806e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f38dc001840 tx=0x7f38dc009040 comp rx=0 tx=0).stop 2024-03-18T16:24:52.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.233+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e8141650 msgr2=0x7f38e81f3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:52.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.233+0000 7f38ede00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e8141650 0x7f38e81f3960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f38d4033000 tx=0x7f38d40046c0 comp rx=0 tx=0).stop 2024-03-18T16:24:52.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.233+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 shutdown_connections 2024-03-18T16:24:52.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.233+0000 7f38ede00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f38c806bbd0 0x7f38c806e090 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:52.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.233+0000 7f38ede00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38e8158420 0x7f38e81f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:52.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.233+0000 7f38ede00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e8141650 0x7f38e81f3960 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:52.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.233+0000 7f38ede00640 1 --2- 172.21.15.102:0/3653154987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f38e80c4fa0 0x7f38e813be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:52.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.233+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 >> 172.21.15.102:0/3653154987 conn(0x7f38e80cb890 msgr2=0x7f38e8071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:52.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.234+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 shutdown_connections 2024-03-18T16:24:52.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:52.234+0000 7f38ede00640 1 -- 172.21.15.102:0/3653154987 wait complete. 2024-03-18T16:24:52.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:24:52.865 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:24:53.866 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:54.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.040+0000 7fb407e00640 1 Processor -- start 2024-03-18T16:24:54.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.040+0000 7fb407e00640 1 -- start start 2024-03-18T16:24:54.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.041+0000 7fb407e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb408141650 0x7fb408158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:54.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.041+0000 7fb407e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb40815c3a0 0x7fb4080c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:54.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.041+0000 7fb407e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb4080c4fa0 0x7fb4080c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:54.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.041+0000 7fb407e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb408159370 con 0x7fb408141650 2024-03-18T16:24:54.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.041+0000 7fb407e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb408158ad0 con 0x7fb4080c4fa0 2024-03-18T16:24:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.041+0000 7fb407e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb408158db0 con 0x7fb40815c3a0 2024-03-18T16:24:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.042+0000 7fb406a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb408141650 0x7fb408158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.042+0000 7fb406000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb40815c3a0 0x7fb4080c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.042+0000 7fb407400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb4080c4fa0 0x7fb4080c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.042+0000 7fb406a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb408141650 0x7fb408158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39110/0 (socket says 172.21.15.102:39110) 2024-03-18T16:24:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.042+0000 7fb406a00640 1 -- 172.21.15.102:0/4112396033 learned_addr learned my addr 172.21.15.102:0/4112396033 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:54.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.045+0000 7fb407400640 1 -- 172.21.15.102:0/4112396033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb40815c3a0 msgr2=0x7fb4080c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:54.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.045+0000 7fb407400640 1 --2- 172.21.15.102:0/4112396033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb40815c3a0 0x7fb4080c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.045+0000 7fb407400640 1 -- 172.21.15.102:0/4112396033 >> v2:172.21.15.102:3300/0 conn(0x7fb408141650 msgr2=0x7fb408158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:54.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.045+0000 7fb407400640 1 --2- 172.21.15.102:0/4112396033 >> v2:172.21.15.102:3300/0 conn(0x7fb408141650 0x7fb408158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.045+0000 7fb407400640 1 -- 172.21.15.102:0/4112396033 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb408159090 con 0x7fb4080c4fa0 2024-03-18T16:24:54.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.047+0000 7fb407400640 1 --2- 172.21.15.102:0/4112396033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4080c4fa0 0x7fb4080c5430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb4080c5d00 tx=0x7fb40002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a60f8ce9f59e37d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:54.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.051+0000 7fb405600640 1 -- 172.21.15.102:0/4112396033 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400039070 con 0x7fb4080c4fa0 2024-03-18T16:24:54.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.051+0000 7fb405600640 1 -- 172.21.15.102:0/4112396033 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb40002cbd0 con 0x7fb4080c4fa0 2024-03-18T16:24:54.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.051+0000 7fb405600640 1 -- 172.21.15.102:0/4112396033 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400034440 con 0x7fb4080c4fa0 2024-03-18T16:24:54.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.052+0000 7fb407e00640 1 -- 172.21.15.102:0/4112396033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4080c4fa0 msgr2=0x7fb4080c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:54.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.052+0000 7fb407e00640 1 --2- 172.21.15.102:0/4112396033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4080c4fa0 0x7fb4080c5430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb4080c5d00 tx=0x7fb40002be20 comp rx=0 tx=0).stop 2024-03-18T16:24:54.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.052+0000 7fb407e00640 1 -- 172.21.15.102:0/4112396033 shutdown_connections 2024-03-18T16:24:54.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.052+0000 7fb407e00640 1 --2- 172.21.15.102:0/4112396033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4080c4fa0 0x7fb4080c5430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.052+0000 7fb407e00640 1 --2- 172.21.15.102:0/4112396033 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb40815c3a0 0x7fb4080c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.052+0000 7fb407e00640 1 --2- 172.21.15.102:0/4112396033 >> v2:172.21.15.102:3300/0 conn(0x7fb408141650 0x7fb408158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.052+0000 7fb407e00640 1 -- 172.21.15.102:0/4112396033 >> 172.21.15.102:0/4112396033 conn(0x7fb4080cb890 msgr2=0x7fb4080cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.052+0000 7fb407e00640 1 -- 172.21.15.102:0/4112396033 shutdown_connections 2024-03-18T16:24:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.052+0000 7fb407e00640 1 -- 172.21.15.102:0/4112396033 wait complete. 2024-03-18T16:24:54.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.054+0000 7fb407e00640 1 Processor -- start 2024-03-18T16:24:54.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.054+0000 7fb407e00640 1 -- start start 2024-03-18T16:24:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.055+0000 7fb407e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4080c4fa0 0x7fb4080c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.055+0000 7fb407e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb408141650 0x7fb40813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.055+0000 7fb407e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb40815c3a0 0x7fb40813f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.055+0000 7fb407e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb40815c7c0 con 0x7fb408141650 2024-03-18T16:24:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.055+0000 7fb407e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4080c66a0 con 0x7fb40815c3a0 2024-03-18T16:24:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.055+0000 7fb407e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb408141a30 con 0x7fb4080c4fa0 2024-03-18T16:24:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.056+0000 7fb407400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb40815c3a0 0x7fb40813f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.056+0000 7fb406000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb408141650 0x7fb40813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.056+0000 7fb406a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4080c4fa0 0x7fb4080c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.056+0000 7fb406000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb408141650 0x7fb40813d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39124/0 (socket says 172.21.15.102:39124) 2024-03-18T16:24:54.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.056+0000 7fb406000640 1 -- 172.21.15.102:0/995785853 learned_addr learned my addr 172.21.15.102:0/995785853 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.060+0000 7fb406000640 1 -- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4080c4fa0 msgr2=0x7fb4080c2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.060+0000 7fb406000640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4080c4fa0 0x7fb4080c2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.060+0000 7fb406000640 1 -- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb40815c3a0 msgr2=0x7fb40813f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.060+0000 7fb406000640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb40815c3a0 0x7fb40813f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.060+0000 7fb406000640 1 -- 172.21.15.102:0/995785853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4081eff30 con 0x7fb408141650 2024-03-18T16:24:54.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.063+0000 7fb406000640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb408141650 0x7fb40813d3f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400e870 tx=0x7fb3f400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.067+0000 7fb3ff400640 1 -- 172.21.15.102:0/995785853 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4017070 con 0x7fb408141650 2024-03-18T16:24:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.068+0000 7fb3ff400640 1 -- 172.21.15.102:0/995785853 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f400c940 con 0x7fb408141650 2024-03-18T16:24:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.068+0000 7fb3ff400640 1 -- 172.21.15.102:0/995785853 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f400cd30 con 0x7fb408141650 2024-03-18T16:24:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.068+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4081f01f0 con 0x7fb408141650 2024-03-18T16:24:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.068+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4081ec4a0 con 0x7fb408141650 2024-03-18T16:24:54.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.069+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4081ebfb0 con 0x7fb408141650 2024-03-18T16:24:54.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.102+0000 7fb3ff400640 1 -- 172.21.15.102:0/995785853 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb3f401a070 con 0x7fb408141650 2024-03-18T16:24:54.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.103+0000 7fb3ff400640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb3e806bc50 0x7fb3e806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:54.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.103+0000 7fb406a00640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb3e806bc50 0x7fb3e806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:54.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.104+0000 7fb406a00640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb3e806bc50 0x7fb3e806e110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8006cc0 tx=0x7fb3f8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:54.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.379+0000 7fb3ff400640 1 -- 172.21.15.102:0/995785853 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb3f4057540 con 0x7fb408141650 2024-03-18T16:24:54.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.380+0000 7fb3ff400640 1 -- 172.21.15.102:0/995785853 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb3f40594a0 con 0x7fb408141650 2024-03-18T16:24:54.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.618+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb408074fd0 con 0x7fb408141650 2024-03-18T16:24:54.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.633+0000 7fb3ff400640 1 -- 172.21.15.102:0/995785853 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb3f405d540 con 0x7fb408141650 2024-03-18T16:24:54.635 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:54.635 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:54.635 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:54.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.635+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb3e806bc50 msgr2=0x7fb3e806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:54.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.635+0000 7fb407e00640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb3e806bc50 0x7fb3e806e110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8006cc0 tx=0x7fb3f8009240 comp rx=0 tx=0).stop 2024-03-18T16:24:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.635+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb408141650 msgr2=0x7fb40813d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.635+0000 7fb407e00640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb408141650 0x7fb40813d3f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb3f400e870 tx=0x7fb3f400ed30 comp rx=0 tx=0).stop 2024-03-18T16:24:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.635+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 shutdown_connections 2024-03-18T16:24:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.635+0000 7fb407e00640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb3e806bc50 0x7fb3e806e110 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.635+0000 7fb407e00640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb40815c3a0 0x7fb40813f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.635+0000 7fb407e00640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb408141650 0x7fb40813d3f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.636+0000 7fb407e00640 1 --2- 172.21.15.102:0/995785853 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb4080c4fa0 0x7fb4080c2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.636+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 >> 172.21.15.102:0/995785853 conn(0x7fb4080cb890 msgr2=0x7fb408072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:54.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.636+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 shutdown_connections 2024-03-18T16:24:54.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:54.637+0000 7fb407e00640 1 -- 172.21.15.102:0/995785853 wait complete. 2024-03-18T16:24:55.260 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:24:55.260 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:24:56.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.433+0000 7fbe0b200640 1 Processor -- start 2024-03-18T16:24:56.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.433+0000 7fbe0b200640 1 -- start start 2024-03-18T16:24:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.434+0000 7fbe0b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c0c9be0 0x7fbe0c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.434+0000 7fbe0b200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbe0c0d05a0 0x7fbe0c0cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.434+0000 7fbe0b200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0c154ab0 con 0x7fbe0c15ae70 2024-03-18T16:24:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.434+0000 7fbe0b200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbe0c0caba0 con 0x7fbe0c0d05a0 2024-03-18T16:24:56.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.434+0000 7fbe0b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0c06d2f0 con 0x7fbe0c0c9be0 2024-03-18T16:24:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.435+0000 7fbe03e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c0c9be0 0x7fbe0c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.435+0000 7fbe0a800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbe0c0d05a0 0x7fbe0c0cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.435+0000 7fbe03e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c0c9be0 0x7fbe0c0d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:54988/0 (socket says 172.21.15.102:54988) 2024-03-18T16:24:56.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.435+0000 7fbe03e00640 1 -- 172.21.15.102:0/149259856 learned_addr learned my addr 172.21.15.102:0/149259856 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:56.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.438+0000 7fbe03e00640 1 -- 172.21.15.102:0/149259856 >> v2:172.21.15.154:3300/0 conn(0x7fbe0c0d05a0 msgr2=0x7fbe0c0cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.438+0000 7fbe03e00640 1 --2- 172.21.15.102:0/149259856 >> v2:172.21.15.154:3300/0 conn(0x7fbe0c0d05a0 0x7fbe0c0cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.438+0000 7fbe03e00640 1 -- 172.21.15.102:0/149259856 >> v1:172.21.15.154:6789/0 conn(0x7fbe0c15ae70 legacy=0x7fbe0c0ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:56.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.438+0000 7fbe03e00640 1 -- 172.21.15.102:0/149259856 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c15b250 con 0x7fbe0c0c9be0 2024-03-18T16:24:56.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.440+0000 7fbe03e00640 1 --2- 172.21.15.102:0/149259856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c0c9be0 0x7fbe0c0d0060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009870 tx=0x7fbdf402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2528ed0e2f235976 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.444+0000 7fbe09400640 1 -- 172.21.15.102:0/149259856 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4034070 con 0x7fbe0c0c9be0 2024-03-18T16:24:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.444+0000 7fbe09400640 1 -- 172.21.15.102:0/149259856 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdf4030840 con 0x7fbe0c0c9be0 2024-03-18T16:24:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.444+0000 7fbe09400640 1 -- 172.21.15.102:0/149259856 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4030c10 con 0x7fbe0c0c9be0 2024-03-18T16:24:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.445+0000 7fbe0b200640 1 -- 172.21.15.102:0/149259856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c0c9be0 msgr2=0x7fbe0c0d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.445+0000 7fbe0b200640 1 --2- 172.21.15.102:0/149259856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c0c9be0 0x7fbe0c0d0060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009870 tx=0x7fbdf402b980 comp rx=0 tx=0).stop 2024-03-18T16:24:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.445+0000 7fbe0b200640 1 -- 172.21.15.102:0/149259856 shutdown_connections 2024-03-18T16:24:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.445+0000 7fbe0b200640 1 --2- 172.21.15.102:0/149259856 >> v2:172.21.15.154:3300/0 conn(0x7fbe0c0d05a0 0x7fbe0c0cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.445+0000 7fbe0b200640 1 --2- 172.21.15.102:0/149259856 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c0c9be0 0x7fbe0c0d0060 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.445+0000 7fbe0b200640 1 -- 172.21.15.102:0/149259856 >> 172.21.15.102:0/149259856 conn(0x7fbe0c0673a0 msgr2=0x7fbe0c0677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.445+0000 7fbe0b200640 1 -- 172.21.15.102:0/149259856 shutdown_connections 2024-03-18T16:24:56.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.445+0000 7fbe0b200640 1 -- 172.21.15.102:0/149259856 wait complete. 2024-03-18T16:24:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.447+0000 7fbe0b200640 1 Processor -- start 2024-03-18T16:24:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.447+0000 7fbe0b200640 1 -- start start 2024-03-18T16:24:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.448+0000 7fbe0b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe0c0c9be0 0x7fbe0c14e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.448+0000 7fbe0b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0d05a0 0x7fbe0c0b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.448+0000 7fbe0b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c15ae70 0x7fbe0c0b45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.448+0000 7fbe0b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c06db90 con 0x7fbe0c0c9be0 2024-03-18T16:24:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.448+0000 7fbe0b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c06d8b0 con 0x7fbe0c0d05a0 2024-03-18T16:24:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.448+0000 7fbe0b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0c06de70 con 0x7fbe0c15ae70 2024-03-18T16:24:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.448+0000 7fbe0a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c15ae70 0x7fbe0c0b45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.448+0000 7fbe03e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0d05a0 0x7fbe0c0b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.449+0000 7fbe09e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe0c0c9be0 0x7fbe0c14e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.449+0000 7fbe0a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c15ae70 0x7fbe0c0b45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:55002/0 (socket says 172.21.15.102:55002) 2024-03-18T16:24:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.449+0000 7fbe0a800640 1 -- 172.21.15.102:0/2303076619 learned_addr learned my addr 172.21.15.102:0/2303076619 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.452+0000 7fbe0a800640 1 -- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0d05a0 msgr2=0x7fbe0c0b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.452+0000 7fbe0a800640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0d05a0 0x7fbe0c0b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.452+0000 7fbe0a800640 1 -- 172.21.15.102:0/2303076619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe0c0c9be0 msgr2=0x7fbe0c14e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.452+0000 7fbe0a800640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe0c0c9be0 0x7fbe0c14e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.452+0000 7fbe0a800640 1 -- 172.21.15.102:0/2303076619 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c14d1b0 con 0x7fbe0c15ae70 2024-03-18T16:24:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.455+0000 7fbe0a800640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c15ae70 0x7fbe0c0b45a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009d30 tx=0x7fbe0400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.459+0000 7fbe02000640 1 -- 172.21.15.102:0/2303076619 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04018020 con 0x7fbe0c15ae70 2024-03-18T16:24:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.459+0000 7fbe02000640 1 -- 172.21.15.102:0/2303076619 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe04014480 con 0x7fbe0c15ae70 2024-03-18T16:24:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.459+0000 7fbe02000640 1 -- 172.21.15.102:0/2303076619 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04014870 con 0x7fbe0c15ae70 2024-03-18T16:24:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.459+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0c14d450 con 0x7fbe0c15ae70 2024-03-18T16:24:56.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.459+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe0c158160 con 0x7fbe0c15ae70 2024-03-18T16:24:56.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.460+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe0c074fd0 con 0x7fbe0c15ae70 2024-03-18T16:24:56.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.719+0000 7fbe02000640 1 -- 172.21.15.102:0/2303076619 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbe04005d10 con 0x7fbe0c15ae70 2024-03-18T16:24:56.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.721+0000 7fbe02000640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbde806bc20 0x7fbde806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:56.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.721+0000 7fbe09e00640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbde806bc20 0x7fbde806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:56.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.722+0000 7fbe02000640 1 -- 172.21.15.102:0/2303076619 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fbe040581d0 con 0x7fbe0c15ae70 2024-03-18T16:24:56.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.722+0000 7fbe09e00640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbde806bc20 0x7fbde806e0e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0016e0 tx=0x7fbdfc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:56.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:56.734+0000 7fbe02000640 1 -- 172.21.15.102:0/2303076619 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbe04022030 con 0x7fbe0c15ae70 2024-03-18T16:24:57.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.011+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe0c15a0a0 con 0x7fbe0c15ae70 2024-03-18T16:24:57.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.068+0000 7fbe02000640 1 -- 172.21.15.102:0/2303076619 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fbe04010040 con 0x7fbe0c15ae70 2024-03-18T16:24:57.070 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:57.070 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:57.070 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.070+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbde806bc20 msgr2=0x7fbde806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.070+0000 7fbe0b200640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbde806bc20 0x7fbde806e0e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc0016e0 tx=0x7fbdfc009040 comp rx=0 tx=0).stop 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.070+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c15ae70 msgr2=0x7fbe0c0b45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.070+0000 7fbe0b200640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c15ae70 0x7fbe0c0b45a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbe04009d30 tx=0x7fbe0400c490 comp rx=0 tx=0).stop 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.071+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 shutdown_connections 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.071+0000 7fbe0b200640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbde806bc20 0x7fbde806e0e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.071+0000 7fbe0b200640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe0c15ae70 0x7fbe0c0b45a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.071+0000 7fbe0b200640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0d05a0 0x7fbe0c0b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.071+0000 7fbe0b200640 1 --2- 172.21.15.102:0/2303076619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe0c0c9be0 0x7fbe0c14e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.071+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 >> 172.21.15.102:0/2303076619 conn(0x7fbe0c0673a0 msgr2=0x7fbe0c159290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.071+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 shutdown_connections 2024-03-18T16:24:57.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:57.071+0000 7fbe0b200640 1 -- 172.21.15.102:0/2303076619 wait complete. 2024-03-18T16:24:57.702 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:24:57.702 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:24:58.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:24:58.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.881+0000 7f413f200640 1 Processor -- start 2024-03-18T16:24:58.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.881+0000 7f413f200640 1 -- start start 2024-03-18T16:24:58.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.883+0000 7f413f200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f41400cfe70 0x7f414015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:58.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.883+0000 7f413f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f41400cc2f0 0x7f41400cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:58.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.883+0000 7f413f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41400ccc80 con 0x7f41400d0250 2024-03-18T16:24:58.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.883+0000 7f413f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f41400cabd0 con 0x7f41400cc2f0 2024-03-18T16:24:58.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.883+0000 7f413f200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f41400caea0 con 0x7f41400cfe70 2024-03-18T16:24:58.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.883+0000 7f413de00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f41400cfe70 0x7f414015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:58.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.883+0000 7f413de00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f41400cfe70 0x7f414015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59584/0 (socket says 172.21.15.102:59584) 2024-03-18T16:24:58.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.883+0000 7f413de00640 1 -- 172.21.15.102:0/3275611353 learned_addr learned my addr 172.21.15.102:0/3275611353 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.886+0000 7f413de00640 1 -- 172.21.15.102:0/3275611353 >> v2:172.21.15.102:3300/0 conn(0x7f41400cc2f0 msgr2=0x7f41400cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.886+0000 7f413de00640 1 --2- 172.21.15.102:0/3275611353 >> v2:172.21.15.102:3300/0 conn(0x7f41400cc2f0 0x7f41400cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:58.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.886+0000 7f413de00640 1 -- 172.21.15.102:0/3275611353 >> v1:172.21.15.102:6789/0 conn(0x7f41400d0250 legacy=0x7f41400c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:24:58.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.886+0000 7f413de00640 1 -- 172.21.15.102:0/3275611353 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41400cb180 con 0x7f41400cfe70 2024-03-18T16:24:58.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.888+0000 7f413de00640 1 --2- 172.21.15.102:0/3275611353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cfe70 0x7f414015ae40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f41400cd800 tx=0x7f413002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffc2bbd1946a9930 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.891+0000 7f413ca00640 1 -- 172.21.15.102:0/3275611353 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130034070 con 0x7f41400cfe70 2024-03-18T16:24:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.891+0000 7f413ca00640 1 -- 172.21.15.102:0/3275611353 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4130030860 con 0x7f41400cfe70 2024-03-18T16:24:58.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.891+0000 7f413ca00640 1 -- 172.21.15.102:0/3275611353 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130030c30 con 0x7f41400cfe70 2024-03-18T16:24:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.892+0000 7f413f200640 1 -- 172.21.15.102:0/3275611353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cfe70 msgr2=0x7f414015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.892+0000 7f413f200640 1 --2- 172.21.15.102:0/3275611353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cfe70 0x7f414015ae40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f41400cd800 tx=0x7f413002b9a0 comp rx=0 tx=0).stop 2024-03-18T16:24:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.892+0000 7f413f200640 1 -- 172.21.15.102:0/3275611353 shutdown_connections 2024-03-18T16:24:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.892+0000 7f413f200640 1 --2- 172.21.15.102:0/3275611353 >> v2:172.21.15.102:3300/0 conn(0x7f41400cc2f0 0x7f41400cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.892+0000 7f413f200640 1 --2- 172.21.15.102:0/3275611353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cfe70 0x7f414015ae40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.892+0000 7f413f200640 1 -- 172.21.15.102:0/3275611353 >> 172.21.15.102:0/3275611353 conn(0x7f414008c3e0 msgr2=0x7f414006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.892+0000 7f413f200640 1 -- 172.21.15.102:0/3275611353 shutdown_connections 2024-03-18T16:24:58.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.892+0000 7f413f200640 1 -- 172.21.15.102:0/3275611353 wait complete. 2024-03-18T16:24:58.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.895+0000 7f413f200640 1 Processor -- start 2024-03-18T16:24:58.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.895+0000 7f413f200640 1 -- start start 2024-03-18T16:24:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.896+0000 7f413f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cc2f0 0x7f41400c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.896+0000 7f413f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41400cfe70 0x7f41401533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.896+0000 7f413f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41400d0250 0x7f41401f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.896+0000 7f413f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f41400cba20 con 0x7f41400cfe70 2024-03-18T16:24:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.896+0000 7f413f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41400cb740 con 0x7f41400cc2f0 2024-03-18T16:24:58.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.896+0000 7f413f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f41400cbd00 con 0x7f41400d0250 2024-03-18T16:24:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.897+0000 7f413de00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cc2f0 0x7f41400c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.897+0000 7f413de00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cc2f0 0x7f41400c5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59586/0 (socket says 172.21.15.102:59586) 2024-03-18T16:24:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.897+0000 7f413de00640 1 -- 172.21.15.102:0/554186238 learned_addr learned my addr 172.21.15.102:0/554186238 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:24:58.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.897+0000 7f413e800640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41400d0250 0x7f41401f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.900+0000 7f413de00640 1 -- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41400d0250 msgr2=0x7f41401f0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.900+0000 7f413de00640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41400d0250 0x7f41401f0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.900+0000 7f413de00640 1 -- 172.21.15.102:0/554186238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41400cfe70 msgr2=0x7f41401533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:58.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.900+0000 7f413de00640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41400cfe70 0x7f41401533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:58.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.900+0000 7f413de00640 1 -- 172.21.15.102:0/554186238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41401f0030 con 0x7f41400cc2f0 2024-03-18T16:24:58.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.904+0000 7f413de00640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cc2f0 0x7f41400c5510 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4130000c00 tx=0x7f4130004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:58.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.907+0000 7f4136000640 1 -- 172.21.15.102:0/554186238 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130048070 con 0x7f41400cc2f0 2024-03-18T16:24:58.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.908+0000 7f4136000640 1 -- 172.21.15.102:0/554186238 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4130030660 con 0x7f41400cc2f0 2024-03-18T16:24:58.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.908+0000 7f4136000640 1 -- 172.21.15.102:0/554186238 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130034040 con 0x7f41400cc2f0 2024-03-18T16:24:58.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.908+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41400c4e60 con 0x7f41400cc2f0 2024-03-18T16:24:58.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.908+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41401f4140 con 0x7f41400cc2f0 2024-03-18T16:24:58.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.909+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41401efe40 con 0x7f41400cc2f0 2024-03-18T16:24:58.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.943+0000 7f4136000640 1 -- 172.21.15.102:0/554186238 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f413004b070 con 0x7f41400cc2f0 2024-03-18T16:24:58.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.944+0000 7f4136000640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f412006bc50 0x7f412006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:24:58.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.945+0000 7f413d400640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f412006bc50 0x7f412006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:24:58.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:58.946+0000 7f413d400640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f412006bc50 0x7f412006e110 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f41400c3600 tx=0x7f412c006ba0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:24:59.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.190+0000 7f4136000640 1 -- 172.21.15.102:0/554186238 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f4130030810 con 0x7f41400cc2f0 2024-03-18T16:24:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.202+0000 7f4136000640 1 -- 172.21.15.102:0/554186238 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4130052020 con 0x7f41400cc2f0 2024-03-18T16:24:59.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.576+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4140074fd0 con 0x7f41400cc2f0 2024-03-18T16:24:59.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.590+0000 7f4136000640 1 -- 172.21.15.102:0/554186238 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f413007b550 con 0x7f41400cc2f0 2024-03-18T16:24:59.592 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:24:59.593 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:24:59.593 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:24:59.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f412006bc50 msgr2=0x7f412006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:59.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f412006bc50 0x7f412006e110 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f41400c3600 tx=0x7f412c006ba0 comp rx=0 tx=0).stop 2024-03-18T16:24:59.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cc2f0 msgr2=0x7f41400c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:24:59.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cc2f0 0x7f41400c5510 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4130000c00 tx=0x7f4130004730 comp rx=0 tx=0).stop 2024-03-18T16:24:59.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 shutdown_connections 2024-03-18T16:24:59.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f412006bc50 0x7f412006e110 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:59.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f41400d0250 0x7f41401f0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:59.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41400cfe70 0x7f41401533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:59.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 --2- 172.21.15.102:0/554186238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41400cc2f0 0x7f41400c5510 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:24:59.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 >> 172.21.15.102:0/554186238 conn(0x7f414008c3e0 msgr2=0x7f4140158960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:24:59.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 shutdown_connections 2024-03-18T16:24:59.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:24:59.594+0000 7f413f200640 1 -- 172.21.15.102:0/554186238 wait complete. 2024-03-18T16:25:00.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:00.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:01.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:01.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.404+0000 7f24b7200640 1 Processor -- start 2024-03-18T16:25:01.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.404+0000 7f24b7200640 1 -- start start 2024-03-18T16:25:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.405+0000 7f24b7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b814dad0 0x7f24b8157a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.406+0000 7f24b7200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24b815b1c0 0x7f24b806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.406+0000 7f24b7200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24b80c75b0 con 0x7f24b80733b0 2024-03-18T16:25:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.406+0000 7f24b7200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f24b814deb0 con 0x7f24b815b1c0 2024-03-18T16:25:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.406+0000 7f24b7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24b815ad30 con 0x7f24b814dad0 2024-03-18T16:25:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.406+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b814dad0 0x7f24b8157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.407+0000 7f24b5e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b814dad0 0x7f24b8157a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57444/0 (socket says 172.21.15.102:57444) 2024-03-18T16:25:01.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.407+0000 7f24b5e00640 1 -- 172.21.15.102:0/1120568798 learned_addr learned my addr 172.21.15.102:0/1120568798 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.409+0000 7f24b5e00640 1 -- 172.21.15.102:0/1120568798 >> v2:172.21.15.154:3300/0 conn(0x7f24b815b1c0 msgr2=0x7f24b806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.409+0000 7f24b5e00640 1 --2- 172.21.15.102:0/1120568798 >> v2:172.21.15.154:3300/0 conn(0x7f24b815b1c0 0x7f24b806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.409+0000 7f24b5e00640 1 -- 172.21.15.102:0/1120568798 >> v1:172.21.15.154:6789/0 conn(0x7f24b80733b0 legacy=0x7f24b80c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:01.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.409+0000 7f24b5e00640 1 -- 172.21.15.102:0/1120568798 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b80c7850 con 0x7f24b814dad0 2024-03-18T16:25:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.412+0000 7f24b5e00640 1 --2- 172.21.15.102:0/1120568798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b814dad0 0x7f24b8157a40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009870 tx=0x7f24a802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98df8cdb95b5724d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:01.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.414+0000 7f24b4a00640 1 -- 172.21.15.102:0/1120568798 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a8035020 con 0x7f24b814dad0 2024-03-18T16:25:01.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.416+0000 7f24b4a00640 1 -- 172.21.15.102:0/1120568798 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a8031640 con 0x7f24b814dad0 2024-03-18T16:25:01.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.416+0000 7f24b4a00640 1 -- 172.21.15.102:0/1120568798 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a802d040 con 0x7f24b814dad0 2024-03-18T16:25:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.417+0000 7f24b7200640 1 -- 172.21.15.102:0/1120568798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b814dad0 msgr2=0x7f24b8157a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.417+0000 7f24b7200640 1 --2- 172.21.15.102:0/1120568798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b814dad0 0x7f24b8157a40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009870 tx=0x7f24a802b8f0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.417+0000 7f24b7200640 1 -- 172.21.15.102:0/1120568798 shutdown_connections 2024-03-18T16:25:01.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.417+0000 7f24b7200640 1 --2- 172.21.15.102:0/1120568798 >> v2:172.21.15.154:3300/0 conn(0x7f24b815b1c0 0x7f24b806de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.417+0000 7f24b7200640 1 --2- 172.21.15.102:0/1120568798 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b814dad0 0x7f24b8157a40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.417+0000 7f24b7200640 1 -- 172.21.15.102:0/1120568798 >> 172.21.15.102:0/1120568798 conn(0x7f24b80727e0 msgr2=0x7f24b8072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:01.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.417+0000 7f24b7200640 1 -- 172.21.15.102:0/1120568798 shutdown_connections 2024-03-18T16:25:01.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.417+0000 7f24b7200640 1 -- 172.21.15.102:0/1120568798 wait complete. 2024-03-18T16:25:01.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.419+0000 7f24b7200640 1 Processor -- start 2024-03-18T16:25:01.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.419+0000 7f24b7200640 1 -- start start 2024-03-18T16:25:01.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.420+0000 7f24b7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b80733b0 0x7f24b813f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:01.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.420+0000 7f24b7200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24b814dad0 0x7f24b813d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:01.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.420+0000 7f24b7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b815b1c0 0x7f24b813df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:01.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.420+0000 7f24b7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24b815a6e0 con 0x7f24b80733b0 2024-03-18T16:25:01.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.420+0000 7f24b7200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24b806e390 con 0x7f24b814dad0 2024-03-18T16:25:01.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.420+0000 7f24b7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24b815a9c0 con 0x7f24b815b1c0 2024-03-18T16:25:01.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.421+0000 7f24b6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b815b1c0 0x7f24b813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:01.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.421+0000 7f24b6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b815b1c0 0x7f24b813df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57448/0 (socket says 172.21.15.102:57448) 2024-03-18T16:25:01.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.421+0000 7f24b6800640 1 -- 172.21.15.102:0/784376683 learned_addr learned my addr 172.21.15.102:0/784376683 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:01.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.427+0000 7f24b6800640 1 -- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24b814dad0 msgr2=0x7f24b813d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:01.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.427+0000 7f24b6800640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24b814dad0 0x7f24b813d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.427+0000 7f24b6800640 1 -- 172.21.15.102:0/784376683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b80733b0 msgr2=0x7f24b813f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:01.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.427+0000 7f24b6800640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b80733b0 0x7f24b813f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.427+0000 7f24b6800640 1 -- 172.21.15.102:0/784376683 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b8155490 con 0x7f24b815b1c0 2024-03-18T16:25:01.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.429+0000 7f24b6800640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b815b1c0 0x7f24b813df30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f24b000b9a0 tx=0x7f24b000be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.433+0000 7f24ae000640 1 -- 172.21.15.102:0/784376683 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0019070 con 0x7f24b815b1c0 2024-03-18T16:25:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.433+0000 7f24ae000640 1 -- 172.21.15.102:0/784376683 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b000cbc0 con 0x7f24b815b1c0 2024-03-18T16:25:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.433+0000 7f24ae000640 1 -- 172.21.15.102:0/784376683 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0014440 con 0x7f24b815b1c0 2024-03-18T16:25:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.433+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b813d640 con 0x7f24b815b1c0 2024-03-18T16:25:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.434+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b81551b0 con 0x7f24b815b1c0 2024-03-18T16:25:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.435+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b8074fd0 con 0x7f24b815b1c0 2024-03-18T16:25:01.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.455+0000 7f24ae000640 1 -- 172.21.15.102:0/784376683 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f24b001c070 con 0x7f24b815b1c0 2024-03-18T16:25:01.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.456+0000 7f24ae000640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f249806bc20 0x7f249806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:01.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.456+0000 7f24b5e00640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f249806bc20 0x7f249806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:01.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.457+0000 7f24b5e00640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f249806bc20 0x7f249806e0e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f24a8036000 tx=0x7f24a8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:01.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.458+0000 7f24ae000640 1 -- 172.21.15.102:0/784376683 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f24b0057090 con 0x7f24b815b1c0 2024-03-18T16:25:01.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.712+0000 7f24ae000640 1 -- 172.21.15.102:0/784376683 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f24b0023020 con 0x7f24b815b1c0 2024-03-18T16:25:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.957+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24b806e170 con 0x7f24b815b1c0 2024-03-18T16:25:01.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.971+0000 7f24ae000640 1 -- 172.21.15.102:0/784376683 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f24b00596f0 con 0x7f24b815b1c0 2024-03-18T16:25:01.973 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:01.973 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:01.974 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f249806bc20 msgr2=0x7f249806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f249806bc20 0x7f249806e0e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f24a8036000 tx=0x7f24a8030000 comp rx=0 tx=0).stop 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b815b1c0 msgr2=0x7f24b813df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b815b1c0 0x7f24b813df30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f24b000b9a0 tx=0x7f24b000be60 comp rx=0 tx=0).stop 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 shutdown_connections 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f249806bc20 0x7f249806e0e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24b815b1c0 0x7f24b813df30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24b814dad0 0x7f24b813d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 --2- 172.21.15.102:0/784376683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b80733b0 0x7f24b813f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:01.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 >> 172.21.15.102:0/784376683 conn(0x7f24b80727e0 msgr2=0x7f24b80c3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:01.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 shutdown_connections 2024-03-18T16:25:01.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:01.974+0000 7f24b7200640 1 -- 172.21.15.102:0/784376683 wait complete. 2024-03-18T16:25:02.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:02.596 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:03.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:03.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.781+0000 7fde40400640 1 Processor -- start 2024-03-18T16:25:03.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.781+0000 7fde40400640 1 -- start start 2024-03-18T16:25:03.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.783+0000 7fde40400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fde381524c0 0x7fde38072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:03.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.783+0000 7fde40400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde38158ee0 0x7fde3806e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:03.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.783+0000 7fde40400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3815c200 con 0x7fde3815b8e0 2024-03-18T16:25:03.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.783+0000 7fde40400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fde38159300 con 0x7fde381524c0 2024-03-18T16:25:03.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.783+0000 7fde40400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde3815c4e0 con 0x7fde38158ee0 2024-03-18T16:25:03.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.784+0000 7fde3d000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde38158ee0 0x7fde3806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:03.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.784+0000 7fde3da00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fde381524c0 0x7fde38072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:03.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.784+0000 7fde3d000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde38158ee0 0x7fde3806e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57454/0 (socket says 172.21.15.102:57454) 2024-03-18T16:25:03.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.784+0000 7fde3d000640 1 -- 172.21.15.102:0/4285854619 learned_addr learned my addr 172.21.15.102:0/4285854619 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:03.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.786+0000 7fde3d000640 1 -- 172.21.15.102:0/4285854619 >> v2:172.21.15.102:3300/0 conn(0x7fde381524c0 msgr2=0x7fde38072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:03.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.786+0000 7fde3d000640 1 --2- 172.21.15.102:0/4285854619 >> v2:172.21.15.102:3300/0 conn(0x7fde381524c0 0x7fde38072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:03.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.786+0000 7fde3d000640 1 -- 172.21.15.102:0/4285854619 >> v1:172.21.15.102:6789/0 conn(0x7fde3815b8e0 legacy=0x7fde380c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:03.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.786+0000 7fde3d000640 1 -- 172.21.15.102:0/4285854619 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3815c7c0 con 0x7fde38158ee0 2024-03-18T16:25:03.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.789+0000 7fde3d000640 1 --2- 172.21.15.102:0/4285854619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde38158ee0 0x7fde3806e340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009870 tx=0x7fde2c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a04dd166a8a8acc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:03.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.792+0000 7fde37e00640 1 -- 172.21.15.102:0/4285854619 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c034070 con 0x7fde38158ee0 2024-03-18T16:25:03.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.792+0000 7fde37e00640 1 -- 172.21.15.102:0/4285854619 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde2c030840 con 0x7fde38158ee0 2024-03-18T16:25:03.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.792+0000 7fde37e00640 1 -- 172.21.15.102:0/4285854619 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c030c10 con 0x7fde38158ee0 2024-03-18T16:25:03.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.793+0000 7fde40400640 1 -- 172.21.15.102:0/4285854619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde38158ee0 msgr2=0x7fde3806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:03.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.793+0000 7fde40400640 1 --2- 172.21.15.102:0/4285854619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde38158ee0 0x7fde3806e340 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009870 tx=0x7fde2c02b980 comp rx=0 tx=0).stop 2024-03-18T16:25:03.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.794+0000 7fde40400640 1 -- 172.21.15.102:0/4285854619 shutdown_connections 2024-03-18T16:25:03.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.794+0000 7fde40400640 1 --2- 172.21.15.102:0/4285854619 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde38158ee0 0x7fde3806e340 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:03.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.794+0000 7fde40400640 1 --2- 172.21.15.102:0/4285854619 >> v2:172.21.15.102:3300/0 conn(0x7fde381524c0 0x7fde38072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:03.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.794+0000 7fde40400640 1 -- 172.21.15.102:0/4285854619 >> 172.21.15.102:0/4285854619 conn(0x7fde380700d0 msgr2=0x7fde3806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:03.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.794+0000 7fde40400640 1 -- 172.21.15.102:0/4285854619 shutdown_connections 2024-03-18T16:25:03.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.794+0000 7fde40400640 1 -- 172.21.15.102:0/4285854619 wait complete. 2024-03-18T16:25:03.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.796+0000 7fde40400640 1 Processor -- start 2024-03-18T16:25:03.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.797+0000 7fde40400640 1 -- start start 2024-03-18T16:25:03.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.798+0000 7fde40400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde381524c0 0x7fde380c2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:03.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.798+0000 7fde40400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde38158ee0 0x7fde380b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:03.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.798+0000 7fde40400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde3815b8e0 0x7fde3814f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:03.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.798+0000 7fde40400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fde381589c0 con 0x7fde38158ee0 2024-03-18T16:25:03.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.798+0000 7fde40400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fde381586e0 con 0x7fde381524c0 2024-03-18T16:25:03.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.798+0000 7fde40400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fde3806e880 con 0x7fde3815b8e0 2024-03-18T16:25:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.799+0000 7fde3e400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde3815b8e0 0x7fde3814f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.799+0000 7fde3da00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde381524c0 0x7fde380c2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.799+0000 7fde3e400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde3815b8e0 0x7fde3814f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57460/0 (socket says 172.21.15.102:57460) 2024-03-18T16:25:03.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.799+0000 7fde3e400640 1 -- 172.21.15.102:0/4206722546 learned_addr learned my addr 172.21.15.102:0/4206722546 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:03.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.802+0000 7fde3d000640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde38158ee0 0x7fde380b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:03.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.803+0000 7fde3e400640 1 -- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde381524c0 msgr2=0x7fde380c2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:03.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.803+0000 7fde3e400640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde381524c0 0x7fde380c2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:03.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.803+0000 7fde3e400640 1 -- 172.21.15.102:0/4206722546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde38158ee0 msgr2=0x7fde380b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:03.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.803+0000 7fde3e400640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde38158ee0 0x7fde380b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:03.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.803+0000 7fde3e400640 1 -- 172.21.15.102:0/4206722546 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde381efb80 con 0x7fde3815b8e0 2024-03-18T16:25:03.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.807+0000 7fde3e400640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde3815b8e0 0x7fde3814f140 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde3000c6b0 tx=0x7fde3000cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.812+0000 7fde35600640 1 -- 172.21.15.102:0/4206722546 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30003b70 con 0x7fde3815b8e0 2024-03-18T16:25:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.812+0000 7fde35600640 1 -- 172.21.15.102:0/4206722546 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde30003d30 con 0x7fde3815b8e0 2024-03-18T16:25:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.812+0000 7fde35600640 1 -- 172.21.15.102:0/4206722546 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30005310 con 0x7fde3815b8e0 2024-03-18T16:25:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.812+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde381efe60 con 0x7fde3815b8e0 2024-03-18T16:25:03.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.812+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde381f0420 con 0x7fde3815b8e0 2024-03-18T16:25:03.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.814+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde38074fd0 con 0x7fde3815b8e0 2024-03-18T16:25:03.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.838+0000 7fde35600640 1 -- 172.21.15.102:0/4206722546 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fde300054b0 con 0x7fde3815b8e0 2024-03-18T16:25:03.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.838+0000 7fde35600640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde1806bc20 0x7fde1806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:03.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.839+0000 7fde3da00640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde1806bc20 0x7fde1806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:03.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.840+0000 7fde3da00640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde1806bc20 0x7fde1806e0e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fde24009b40 tx=0x7fde24009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:03.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:03.844+0000 7fde35600640 1 -- 172.21.15.102:0/4206722546 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fde300057a0 con 0x7fde3815b8e0 2024-03-18T16:25:04.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.148+0000 7fde35600640 1 -- 172.21.15.102:0/4206722546 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fde30059e50 con 0x7fde3815b8e0 2024-03-18T16:25:04.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.445+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde38151c80 con 0x7fde3815b8e0 2024-03-18T16:25:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.468+0000 7fde35600640 1 -- 172.21.15.102:0/4206722546 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fde30059750 con 0x7fde3815b8e0 2024-03-18T16:25:04.470 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:04.470 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:04.470 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde1806bc20 msgr2=0x7fde1806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde1806bc20 0x7fde1806e0e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fde24009b40 tx=0x7fde24009190 comp rx=0 tx=0).stop 2024-03-18T16:25:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde3815b8e0 msgr2=0x7fde3814f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde3815b8e0 0x7fde3814f140 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fde3000c6b0 tx=0x7fde3000cb70 comp rx=0 tx=0).stop 2024-03-18T16:25:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 shutdown_connections 2024-03-18T16:25:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fde1806bc20 0x7fde1806e0e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fde3815b8e0 0x7fde3814f140 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde38158ee0 0x7fde380b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 --2- 172.21.15.102:0/4206722546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fde381524c0 0x7fde380c2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 >> 172.21.15.102:0/4206722546 conn(0x7fde380700d0 msgr2=0x7fde38071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 shutdown_connections 2024-03-18T16:25:04.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:04.471+0000 7fde40400640 1 -- 172.21.15.102:0/4206722546 wait complete. 2024-03-18T16:25:05.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:05.102 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:06.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:06.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.275+0000 7f5e75000640 1 Processor -- start 2024-03-18T16:25:06.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.275+0000 7f5e75000640 1 -- start start 2024-03-18T16:25:06.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.276+0000 7f5e75000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e7015ae70 0x7f5e700ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:06.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.276+0000 7f5e75000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5e700c9be0 0x7f5e700d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:06.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.276+0000 7f5e75000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e70154ab0 con 0x7f5e700d05a0 2024-03-18T16:25:06.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.276+0000 7f5e75000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5e7006d2f0 con 0x7f5e7015ae70 2024-03-18T16:25:06.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.276+0000 7f5e75000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5e7006d5d0 con 0x7f5e700c9be0 2024-03-18T16:25:06.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.277+0000 7f5e6ea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5e700c9be0 0x7f5e700d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:06.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.277+0000 7f5e6f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e7015ae70 0x7f5e700ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:06.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.277+0000 7f5e6ea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5e700c9be0 0x7f5e700d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42650/0 (socket says 172.21.15.102:42650) 2024-03-18T16:25:06.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.277+0000 7f5e6ea00640 1 -- 172.21.15.102:0/3997399616 learned_addr learned my addr 172.21.15.102:0/3997399616 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:06.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.280+0000 7f5e6f400640 1 -- 172.21.15.102:0/3997399616 >> v2:172.21.15.154:3300/0 conn(0x7f5e700c9be0 msgr2=0x7f5e700d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:06.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.280+0000 7f5e6f400640 1 --2- 172.21.15.102:0/3997399616 >> v2:172.21.15.154:3300/0 conn(0x7f5e700c9be0 0x7f5e700d0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.280+0000 7f5e6f400640 1 -- 172.21.15.102:0/3997399616 >> v1:172.21.15.154:6789/0 conn(0x7f5e700d05a0 legacy=0x7f5e700cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:06.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.280+0000 7f5e6f400640 1 -- 172.21.15.102:0/3997399616 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e7015b250 con 0x7f5e7015ae70 2024-03-18T16:25:06.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.283+0000 7f5e6f400640 1 --2- 172.21.15.102:0/3997399616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e7015ae70 0x7f5e700ca490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5e70157f40 tx=0x7f5e6002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c083e9c126cef9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:06.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.287+0000 7f5e6e000640 1 -- 172.21.15.102:0/3997399616 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e600315a0 con 0x7f5e7015ae70 2024-03-18T16:25:06.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.287+0000 7f5e6e000640 1 -- 172.21.15.102:0/3997399616 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e60037070 con 0x7f5e7015ae70 2024-03-18T16:25:06.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.287+0000 7f5e6e000640 1 -- 172.21.15.102:0/3997399616 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60031960 con 0x7f5e7015ae70 2024-03-18T16:25:06.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.288+0000 7f5e75000640 1 -- 172.21.15.102:0/3997399616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e7015ae70 msgr2=0x7f5e700ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:06.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.288+0000 7f5e75000640 1 --2- 172.21.15.102:0/3997399616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e7015ae70 0x7f5e700ca490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5e70157f40 tx=0x7f5e6002b840 comp rx=0 tx=0).stop 2024-03-18T16:25:06.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.289+0000 7f5e75000640 1 -- 172.21.15.102:0/3997399616 shutdown_connections 2024-03-18T16:25:06.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.289+0000 7f5e75000640 1 --2- 172.21.15.102:0/3997399616 >> v2:172.21.15.154:3300/0 conn(0x7f5e700c9be0 0x7f5e700d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.289+0000 7f5e75000640 1 --2- 172.21.15.102:0/3997399616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e7015ae70 0x7f5e700ca490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.289+0000 7f5e75000640 1 -- 172.21.15.102:0/3997399616 >> 172.21.15.102:0/3997399616 conn(0x7f5e700673a0 msgr2=0x7f5e700677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:06.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.289+0000 7f5e75000640 1 -- 172.21.15.102:0/3997399616 shutdown_connections 2024-03-18T16:25:06.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.289+0000 7f5e75000640 1 -- 172.21.15.102:0/3997399616 wait complete. 2024-03-18T16:25:06.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.291+0000 7f5e75000640 1 Processor -- start 2024-03-18T16:25:06.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.292+0000 7f5e75000640 1 -- start start 2024-03-18T16:25:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.293+0000 7f5e75000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e700c9be0 0x7f5e700c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.293+0000 7f5e75000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e700d05a0 0x7f5e701f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.293+0000 7f5e75000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e7015ae70 0x7f5e70159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.293+0000 7f5e75000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7006de70 con 0x7f5e7015ae70 2024-03-18T16:25:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.293+0000 7f5e75000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7006db90 con 0x7f5e700d05a0 2024-03-18T16:25:06.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.293+0000 7f5e75000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5e7006e150 con 0x7f5e700c9be0 2024-03-18T16:25:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.294+0000 7f5e6f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e700c9be0 0x7f5e700c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.294+0000 7f5e6ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e700d05a0 0x7f5e701f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.294+0000 7f5e6fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e7015ae70 0x7f5e70159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.294+0000 7f5e6f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e700c9be0 0x7f5e700c87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57470/0 (socket says 172.21.15.102:57470) 2024-03-18T16:25:06.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.294+0000 7f5e6f400640 1 -- 172.21.15.102:0/550507791 learned_addr learned my addr 172.21.15.102:0/550507791 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:06.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.298+0000 7f5e6f400640 1 -- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e700d05a0 msgr2=0x7f5e701f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:06.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.298+0000 7f5e6f400640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e700d05a0 0x7f5e701f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.298+0000 7f5e6f400640 1 -- 172.21.15.102:0/550507791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e7015ae70 msgr2=0x7f5e70159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:06.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.298+0000 7f5e6f400640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e7015ae70 0x7f5e70159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.298+0000 7f5e6f400640 1 -- 172.21.15.102:0/550507791 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e701f12a0 con 0x7f5e700c9be0 2024-03-18T16:25:06.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.301+0000 7f5e6f400640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e700c9be0 0x7f5e700c87d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009890 tx=0x7f5e600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:06.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.305+0000 7f5e67400640 1 -- 172.21.15.102:0/550507791 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60032630 con 0x7f5e700c9be0 2024-03-18T16:25:06.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.305+0000 7f5e67400640 1 -- 172.21.15.102:0/550507791 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e600327d0 con 0x7f5e700c9be0 2024-03-18T16:25:06.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.305+0000 7f5e67400640 1 -- 172.21.15.102:0/550507791 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e60032bc0 con 0x7f5e700c9be0 2024-03-18T16:25:06.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.305+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e700c8ed0 con 0x7f5e700c9be0 2024-03-18T16:25:06.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.305+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e701f07c0 con 0x7f5e700c9be0 2024-03-18T16:25:06.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.307+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e700c91b0 con 0x7f5e700c9be0 2024-03-18T16:25:06.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.566+0000 7f5e67400640 1 -- 172.21.15.102:0/550507791 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5e6004c060 con 0x7f5e700c9be0 2024-03-18T16:25:06.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.567+0000 7f5e67400640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5e5006bc20 0x7f5e5006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:06.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.568+0000 7f5e6ea00640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5e5006bc20 0x7f5e5006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:06.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.569+0000 7f5e67400640 1 -- 172.21.15.102:0/550507791 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5e6002f030 con 0x7f5e700c9be0 2024-03-18T16:25:06.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.569+0000 7f5e6ea00640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5e5006bc20 0x7f5e5006e0e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5e700cd1d0 tx=0x7f5e5c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:06.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.580+0000 7f5e67400640 1 -- 172.21.15.102:0/550507791 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5e60040070 con 0x7f5e700c9be0 2024-03-18T16:25:06.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.815+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e700caac0 con 0x7f5e700c9be0 2024-03-18T16:25:06.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.836+0000 7f5e67400640 1 -- 172.21.15.102:0/550507791 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f5e700caac0 con 0x7f5e700c9be0 2024-03-18T16:25:06.838 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:06.839 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:06.839 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:06.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5e5006bc20 msgr2=0x7f5e5006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:06.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5e5006bc20 0x7f5e5006e0e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5e700cd1d0 tx=0x7f5e5c009240 comp rx=0 tx=0).stop 2024-03-18T16:25:06.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e700c9be0 msgr2=0x7f5e700c87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e700c9be0 0x7f5e700c87d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e60009890 tx=0x7f5e600046c0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 shutdown_connections 2024-03-18T16:25:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5e5006bc20 0x7f5e5006e0e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e7015ae70 0x7f5e70159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e700d05a0 0x7f5e701f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 --2- 172.21.15.102:0/550507791 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5e700c9be0 0x7f5e700c87d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.840+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 >> 172.21.15.102:0/550507791 conn(0x7f5e700673a0 msgr2=0x7f5e70159b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.841+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 shutdown_connections 2024-03-18T16:25:06.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:06.841+0000 7f5e75000640 1 -- 172.21.15.102:0/550507791 wait complete. 2024-03-18T16:25:07.463 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:07.464 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:08.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:08.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.637+0000 7fcc1e200640 1 Processor -- start 2024-03-18T16:25:08.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.637+0000 7fcc1e200640 1 -- start start 2024-03-18T16:25:08.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.638+0000 7fcc1e200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcc18141650 0x7fcc18158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:08.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.638+0000 7fcc1e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcc1815c3a0 0x7fcc180c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:08.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.638+0000 7fcc1e200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc18158ad0 con 0x7fcc180c4fa0 2024-03-18T16:25:08.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.638+0000 7fcc1e200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fcc18159370 con 0x7fcc1815c3a0 2024-03-18T16:25:08.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.638+0000 7fcc1e200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcc18158db0 con 0x7fcc18141650 2024-03-18T16:25:08.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.639+0000 7fcc1ce00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcc18141650 0x7fcc18158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:08.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.639+0000 7fcc17e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcc1815c3a0 0x7fcc180c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:08.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.639+0000 7fcc1ce00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcc18141650 0x7fcc18158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42662/0 (socket says 172.21.15.102:42662) 2024-03-18T16:25:08.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.639+0000 7fcc1ce00640 1 -- 172.21.15.102:0/4275686612 learned_addr learned my addr 172.21.15.102:0/4275686612 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:08.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.642+0000 7fcc1ce00640 1 -- 172.21.15.102:0/4275686612 >> v2:172.21.15.102:3300/0 conn(0x7fcc1815c3a0 msgr2=0x7fcc180c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:08.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.642+0000 7fcc1ce00640 1 --2- 172.21.15.102:0/4275686612 >> v2:172.21.15.102:3300/0 conn(0x7fcc1815c3a0 0x7fcc180c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:08.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.642+0000 7fcc1ce00640 1 -- 172.21.15.102:0/4275686612 >> v1:172.21.15.154:6789/0 conn(0x7fcc180c4fa0 legacy=0x7fcc1806f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:08.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.642+0000 7fcc1ce00640 1 -- 172.21.15.102:0/4275686612 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc18159090 con 0x7fcc18141650 2024-03-18T16:25:08.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.644+0000 7fcc1ce00640 1 --2- 172.21.15.102:0/4275686612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc18141650 0x7fcc18158420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcc18068ac0 tx=0x7fcc0c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65714c164cb0af1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:08.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.649+0000 7fcc17400640 1 -- 172.21.15.102:0/4275686612 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c035020 con 0x7fcc18141650 2024-03-18T16:25:08.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.650+0000 7fcc17400640 1 -- 172.21.15.102:0/4275686612 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c031870 con 0x7fcc18141650 2024-03-18T16:25:08.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.650+0000 7fcc17400640 1 -- 172.21.15.102:0/4275686612 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c031c40 con 0x7fcc18141650 2024-03-18T16:25:08.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.651+0000 7fcc1e200640 1 -- 172.21.15.102:0/4275686612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc18141650 msgr2=0x7fcc18158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:08.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.651+0000 7fcc1e200640 1 --2- 172.21.15.102:0/4275686612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc18141650 0x7fcc18158420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcc18068ac0 tx=0x7fcc0c02b910 comp rx=0 tx=0).stop 2024-03-18T16:25:08.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.651+0000 7fcc1e200640 1 -- 172.21.15.102:0/4275686612 shutdown_connections 2024-03-18T16:25:08.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.651+0000 7fcc1e200640 1 --2- 172.21.15.102:0/4275686612 >> v2:172.21.15.102:3300/0 conn(0x7fcc1815c3a0 0x7fcc180c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:08.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.651+0000 7fcc1e200640 1 --2- 172.21.15.102:0/4275686612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc18141650 0x7fcc18158420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:08.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.651+0000 7fcc1e200640 1 -- 172.21.15.102:0/4275686612 >> 172.21.15.102:0/4275686612 conn(0x7fcc180cb890 msgr2=0x7fcc180cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:08.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.651+0000 7fcc1e200640 1 -- 172.21.15.102:0/4275686612 shutdown_connections 2024-03-18T16:25:08.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.651+0000 7fcc1e200640 1 -- 172.21.15.102:0/4275686612 wait complete. 2024-03-18T16:25:08.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.652+0000 7fcc1e200640 1 Processor -- start 2024-03-18T16:25:08.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.653+0000 7fcc1e200640 1 -- start start 2024-03-18T16:25:08.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.653+0000 7fcc1e200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc180c4fa0 0x7fcc18153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:08.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.653+0000 7fcc1e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18141650 0x7fcc18137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:08.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.654+0000 7fcc1e200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1815c3a0 0x7fcc181352e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:08.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.654+0000 7fcc1e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcc18141a30 con 0x7fcc18141650 2024-03-18T16:25:08.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.654+0000 7fcc1e200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcc180c66a0 con 0x7fcc1815c3a0 2024-03-18T16:25:08.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.654+0000 7fcc1e200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcc18141bb0 con 0x7fcc180c4fa0 2024-03-18T16:25:08.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.654+0000 7fcc1d800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1815c3a0 0x7fcc181352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:08.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.654+0000 7fcc1ce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc180c4fa0 0x7fcc18153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:08.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.654+0000 7fcc1d800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1815c3a0 0x7fcc181352e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42676/0 (socket says 172.21.15.102:42676) 2024-03-18T16:25:08.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.655+0000 7fcc1d800640 1 -- 172.21.15.102:0/4290642605 learned_addr learned my addr 172.21.15.102:0/4290642605 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:08.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.655+0000 7fcc17e00640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18141650 0x7fcc18137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:08.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.658+0000 7fcc1d800640 1 -- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc180c4fa0 msgr2=0x7fcc18153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:08.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.658+0000 7fcc1d800640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc180c4fa0 0x7fcc18153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:08.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.658+0000 7fcc1d800640 1 -- 172.21.15.102:0/4290642605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18141650 msgr2=0x7fcc18137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:08.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.658+0000 7fcc1d800640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18141650 0x7fcc18137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:08.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.658+0000 7fcc1d800640 1 -- 172.21.15.102:0/4290642605 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc181558a0 con 0x7fcc1815c3a0 2024-03-18T16:25:08.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.660+0000 7fcc1d800640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1815c3a0 0x7fcc181352e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c780 tx=0x7fcc1000cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:08.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.664+0000 7fcc14c00640 1 -- 172.21.15.102:0/4290642605 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10017070 con 0x7fcc1815c3a0 2024-03-18T16:25:08.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.664+0000 7fcc14c00640 1 -- 172.21.15.102:0/4290642605 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10003da0 con 0x7fcc1815c3a0 2024-03-18T16:25:08.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.664+0000 7fcc14c00640 1 -- 172.21.15.102:0/4290642605 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100052b0 con 0x7fcc1815c3a0 2024-03-18T16:25:08.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.664+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc18155b80 con 0x7fcc1815c3a0 2024-03-18T16:25:08.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.664+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc18156c80 con 0x7fcc1815c3a0 2024-03-18T16:25:08.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.666+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc18159ea0 con 0x7fcc1815c3a0 2024-03-18T16:25:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.685+0000 7fcc14c00640 1 -- 172.21.15.102:0/4290642605 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcc1001a070 con 0x7fcc1815c3a0 2024-03-18T16:25:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.686+0000 7fcc14c00640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf806bc20 0x7fcbf806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.687+0000 7fcc1ce00640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf806bc20 0x7fcbf806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:08.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.687+0000 7fcc1ce00640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf806bc20 0x7fcbf806e0e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c036000 tx=0x7fcc0c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:08.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.928+0000 7fcc14c00640 1 -- 172.21.15.102:0/4290642605 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fcc100569e0 con 0x7fcc1815c3a0 2024-03-18T16:25:08.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:08.939+0000 7fcc14c00640 1 -- 172.21.15.102:0/4290642605 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcc18159ea0 con 0x7fcc1815c3a0 2024-03-18T16:25:09.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.226+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc180b7f00 con 0x7fcc1815c3a0 2024-03-18T16:25:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.338+0000 7fcc14c00640 1 -- 172.21.15.102:0/4290642605 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fcc10021020 con 0x7fcc1815c3a0 2024-03-18T16:25:09.340 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:09.340 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:09.340 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf806bc20 msgr2=0x7fcbf806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf806bc20 0x7fcbf806e0e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c036000 tx=0x7fcc0c03c040 comp rx=0 tx=0).stop 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1815c3a0 msgr2=0x7fcc181352e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1815c3a0 0x7fcc181352e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcc1000c780 tx=0x7fcc1000cc40 comp rx=0 tx=0).stop 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 shutdown_connections 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf806bc20 0x7fcbf806e0e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1815c3a0 0x7fcc181352e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc18141650 0x7fcc18137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 --2- 172.21.15.102:0/4290642605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc180c4fa0 0x7fcc18153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:09.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 >> 172.21.15.102:0/4290642605 conn(0x7fcc180cb890 msgr2=0x7fcc18159790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:09.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 shutdown_connections 2024-03-18T16:25:09.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:09.342+0000 7fcc1e200640 1 -- 172.21.15.102:0/4290642605 wait complete. 2024-03-18T16:25:09.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:09.971 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:10.972 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:11.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.146+0000 7faee1600640 1 Processor -- start 2024-03-18T16:25:11.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.146+0000 7faee1600640 1 -- start start 2024-03-18T16:25:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.147+0000 7faee1600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faedc141650 0x7faedc158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.148+0000 7faee1600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faedc0c4fa0 0x7faedc0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.148+0000 7faee1600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faedc0c72d0 con 0x7faedc15c3a0 2024-03-18T16:25:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.148+0000 7faee1600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7faedc159370 con 0x7faedc141650 2024-03-18T16:25:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.148+0000 7faee1600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faedc158ad0 con 0x7faedc0c4fa0 2024-03-18T16:25:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.148+0000 7faee0c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faedc0c4fa0 0x7faedc0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.149+0000 7faedbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faedc141650 0x7faedc158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:11.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.149+0000 7faee0c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faedc0c4fa0 0x7faedc0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59070/0 (socket says 172.21.15.102:59070) 2024-03-18T16:25:11.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.149+0000 7faee0c00640 1 -- 172.21.15.102:0/74598912 learned_addr learned my addr 172.21.15.102:0/74598912 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.151+0000 7faee0c00640 1 -- 172.21.15.102:0/74598912 >> v2:172.21.15.102:3300/0 conn(0x7faedc141650 msgr2=0x7faedc158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.151+0000 7faee0c00640 1 --2- 172.21.15.102:0/74598912 >> v2:172.21.15.102:3300/0 conn(0x7faedc141650 0x7faedc158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.151+0000 7faee0c00640 1 -- 172.21.15.102:0/74598912 >> v1:172.21.15.102:6789/0 conn(0x7faedc15c3a0 legacy=0x7faedc0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:11.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.151+0000 7faee0c00640 1 -- 172.21.15.102:0/74598912 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faedc158db0 con 0x7faedc0c4fa0 2024-03-18T16:25:11.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.154+0000 7faee0c00640 1 --2- 172.21.15.102:0/74598912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc0c4fa0 0x7faedc0c5410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faedc0c9720 tx=0x7faed402bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b018054231a50ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.157+0000 7faedaa00640 1 -- 172.21.15.102:0/74598912 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed402cb10 con 0x7faedc0c4fa0 2024-03-18T16:25:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.157+0000 7faedaa00640 1 -- 172.21.15.102:0/74598912 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed402ccb0 con 0x7faedc0c4fa0 2024-03-18T16:25:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.158+0000 7faedaa00640 1 -- 172.21.15.102:0/74598912 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed402ce60 con 0x7faedc0c4fa0 2024-03-18T16:25:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.158+0000 7faee1600640 1 -- 172.21.15.102:0/74598912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc0c4fa0 msgr2=0x7faedc0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.158+0000 7faee1600640 1 --2- 172.21.15.102:0/74598912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc0c4fa0 0x7faedc0c5410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faedc0c9720 tx=0x7faed402bf70 comp rx=0 tx=0).stop 2024-03-18T16:25:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.159+0000 7faee1600640 1 -- 172.21.15.102:0/74598912 shutdown_connections 2024-03-18T16:25:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.159+0000 7faee1600640 1 --2- 172.21.15.102:0/74598912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc0c4fa0 0x7faedc0c5410 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.159+0000 7faee1600640 1 --2- 172.21.15.102:0/74598912 >> v2:172.21.15.102:3300/0 conn(0x7faedc141650 0x7faedc158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.159+0000 7faee1600640 1 -- 172.21.15.102:0/74598912 >> 172.21.15.102:0/74598912 conn(0x7faedc0cb890 msgr2=0x7faedc0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:11.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.159+0000 7faee1600640 1 -- 172.21.15.102:0/74598912 shutdown_connections 2024-03-18T16:25:11.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.159+0000 7faee1600640 1 -- 172.21.15.102:0/74598912 wait complete. 2024-03-18T16:25:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.161+0000 7faee1600640 1 Processor -- start 2024-03-18T16:25:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.161+0000 7faee1600640 1 -- start start 2024-03-18T16:25:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.163+0000 7faee1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faedc0c4fa0 0x7faedc1531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.163+0000 7faee1600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc141650 0x7faedc1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.163+0000 7faee1600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faedc15c3a0 0x7faedc1597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.163+0000 7faee1600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faedc0c66a0 con 0x7faedc0c4fa0 2024-03-18T16:25:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.163+0000 7faee1600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faedc0c63c0 con 0x7faedc141650 2024-03-18T16:25:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.163+0000 7faee1600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faedc15c7c0 con 0x7faedc15c3a0 2024-03-18T16:25:11.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.163+0000 7faedb400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc141650 0x7faedc1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:11.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.164+0000 7faedb400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc141650 0x7faedc1536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59076/0 (socket says 172.21.15.102:59076) 2024-03-18T16:25:11.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.164+0000 7faedb400640 1 -- 172.21.15.102:0/2489217716 learned_addr learned my addr 172.21.15.102:0/2489217716 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:11.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.164+0000 7faee0c00640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faedc15c3a0 0x7faedc1597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:11.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.164+0000 7faedbe00640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faedc0c4fa0 0x7faedc1531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.167+0000 7faedb400640 1 -- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faedc15c3a0 msgr2=0x7faedc1597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.167+0000 7faedb400640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faedc15c3a0 0x7faedc1597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.167+0000 7faedb400640 1 -- 172.21.15.102:0/2489217716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faedc0c4fa0 msgr2=0x7faedc1531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.167+0000 7faedb400640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faedc0c4fa0 0x7faedc1531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.167+0000 7faedb400640 1 -- 172.21.15.102:0/2489217716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faedc1f0190 con 0x7faedc141650 2024-03-18T16:25:11.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.169+0000 7faedb400640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc141650 0x7faedc1536f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7faec800c3f0 tx=0x7faec800c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.173+0000 7faed3e00640 1 -- 172.21.15.102:0/2489217716 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec80052d0 con 0x7faedc141650 2024-03-18T16:25:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.173+0000 7faed3e00640 1 -- 172.21.15.102:0/2489217716 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec8005490 con 0x7faedc141650 2024-03-18T16:25:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.173+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faedc1f0470 con 0x7faedc141650 2024-03-18T16:25:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.173+0000 7faed3e00640 1 -- 172.21.15.102:0/2489217716 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8007260 con 0x7faedc141650 2024-03-18T16:25:11.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.173+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faedc1f0a30 con 0x7faedc141650 2024-03-18T16:25:11.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.175+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faedc06e3d0 con 0x7faedc141650 2024-03-18T16:25:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.194+0000 7faed3e00640 1 -- 172.21.15.102:0/2489217716 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faec8007400 con 0x7faedc141650 2024-03-18T16:25:11.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.195+0000 7faed3e00640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faebc06bc20 0x7faebc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:11.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.196+0000 7faedbe00640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faebc06bc20 0x7faebc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:11.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.196+0000 7faedbe00640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faebc06bc20 0x7faebc06e0e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faecc009b40 tx=0x7faecc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:11.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.197+0000 7faed3e00640 1 -- 172.21.15.102:0/2489217716 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7faec80568b0 con 0x7faedc141650 2024-03-18T16:25:11.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.449+0000 7faed3e00640 1 -- 172.21.15.102:0/2489217716 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faedc06e3d0 con 0x7faedc141650 2024-03-18T16:25:11.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.761+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faedc074fd0 con 0x7faedc141650 2024-03-18T16:25:11.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.774+0000 7faed3e00640 1 -- 172.21.15.102:0/2489217716 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7faec8058810 con 0x7faedc141650 2024-03-18T16:25:11.776 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:11.776 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:11.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faebc06bc20 msgr2=0x7faebc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:11.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faebc06bc20 0x7faebc06e0e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faecc009b40 tx=0x7faecc009190 comp rx=0 tx=0).stop 2024-03-18T16:25:11.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc141650 msgr2=0x7faedc1536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:11.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc141650 0x7faedc1536f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7faec800c3f0 tx=0x7faec800c8b0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 shutdown_connections 2024-03-18T16:25:11.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faebc06bc20 0x7faebc06e0e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faedc15c3a0 0x7faedc1597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faedc141650 0x7faedc1536f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 --2- 172.21.15.102:0/2489217716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faedc0c4fa0 0x7faedc1531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 >> 172.21.15.102:0/2489217716 conn(0x7faedc0cb890 msgr2=0x7faedc072d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 shutdown_connections 2024-03-18T16:25:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:11.777+0000 7faee1600640 1 -- 172.21.15.102:0/2489217716 wait complete. 2024-03-18T16:25:12.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:12.409 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:13.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:13.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.584+0000 7f3bcbe00640 1 Processor -- start 2024-03-18T16:25:13.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.585+0000 7f3bcbe00640 1 -- start start 2024-03-18T16:25:13.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.587+0000 7f3bcbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3bcc0ca960 0x7f3bcc06c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:13.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.587+0000 7f3bcbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc0cad40 0x7f3bcc15e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:13.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.587+0000 7f3bcbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3bcc15a1a0 0x7f3bcc0705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:13.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.587+0000 7f3bcbe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3bcc0c8680 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.587+0000 7f3bcbe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3bcc15e670 con 0x7f3bcc0ca960 2024-03-18T16:25:13.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.587+0000 7f3bcbe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bcc06c9c0 con 0x7f3bcc0cad40 2024-03-18T16:25:13.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.587+0000 7f3bca000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc0cad40 0x7f3bcc15e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:13.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.588+0000 7f3bcaa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3bcc0ca960 0x7f3bcc06c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:13.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.588+0000 7f3bca000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc0cad40 0x7f3bcc15e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50638/0 (socket says 172.21.15.102:50638) 2024-03-18T16:25:13.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.588+0000 7f3bca000640 1 -- 172.21.15.102:0/1760523054 learned_addr learned my addr 172.21.15.102:0/1760523054 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:13.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.588+0000 7f3bcb400640 1 --2- 172.21.15.102:0/1760523054 >> v2:172.21.15.102:3300/0 conn(0x7f3bcc15a1a0 0x7f3bcc0705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:13.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.590+0000 7f3bca000640 1 -- 172.21.15.102:0/1760523054 >> v2:172.21.15.154:3300/0 conn(0x7f3bcc0ca960 msgr2=0x7f3bcc06c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:13.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.590+0000 7f3bca000640 1 --2- 172.21.15.102:0/1760523054 >> v2:172.21.15.154:3300/0 conn(0x7f3bcc0ca960 0x7f3bcc06c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:13.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.590+0000 7f3bca000640 1 -- 172.21.15.102:0/1760523054 >> v2:172.21.15.102:3300/0 conn(0x7f3bcc15a1a0 msgr2=0x7f3bcc0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:13.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.590+0000 7f3bca000640 1 --2- 172.21.15.102:0/1760523054 >> v2:172.21.15.102:3300/0 conn(0x7f3bcc15a1a0 0x7f3bcc0705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:13.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.590+0000 7f3bca000640 1 -- 172.21.15.102:0/1760523054 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bcc06cca0 con 0x7f3bcc0cad40 2024-03-18T16:25:13.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.592+0000 7f3bca000640 1 --2- 172.21.15.102:0/1760523054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc0cad40 0x7f3bcc15e130 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0cb250 tx=0x7f3bb802b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92f532b873d7751e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:13.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.597+0000 7f3bc9600640 1 -- 172.21.15.102:0/1760523054 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8034070 con 0x7f3bcc0cad40 2024-03-18T16:25:13.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.597+0000 7f3bc9600640 1 -- 172.21.15.102:0/1760523054 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb8030860 con 0x7f3bcc0cad40 2024-03-18T16:25:13.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.597+0000 7f3bc9600640 1 -- 172.21.15.102:0/1760523054 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8030c30 con 0x7f3bcc0cad40 2024-03-18T16:25:13.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.598+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1760523054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc0cad40 msgr2=0x7f3bcc15e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:13.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.598+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1760523054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc0cad40 0x7f3bcc15e130 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc0cb250 tx=0x7f3bb802b9a0 comp rx=0 tx=0).stop 2024-03-18T16:25:13.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.598+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1760523054 shutdown_connections 2024-03-18T16:25:13.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.598+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1760523054 >> v2:172.21.15.102:3300/0 conn(0x7f3bcc15a1a0 0x7f3bcc0705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:13.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.598+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1760523054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc0cad40 0x7f3bcc15e130 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:13.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.598+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1760523054 >> v2:172.21.15.154:3300/0 conn(0x7f3bcc0ca960 0x7f3bcc06c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:13.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.598+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1760523054 >> 172.21.15.102:0/1760523054 conn(0x7f3bcc067190 msgr2=0x7f3bcc0675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:13.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.598+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1760523054 shutdown_connections 2024-03-18T16:25:13.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.598+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1760523054 wait complete. 2024-03-18T16:25:13.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.601+0000 7f3bcbe00640 1 Processor -- start 2024-03-18T16:25:13.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.601+0000 7f3bcbe00640 1 -- start start 2024-03-18T16:25:13.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.602+0000 7f3bcbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc0ca960 0x7f3bcc0cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:13.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.602+0000 7f3bcbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bcc0cad40 0x7f3bcc1e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:13.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.602+0000 7f3bcbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc15a1a0 0x7f3bcc1e7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:13.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.602+0000 7f3bcbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc15dc10 con 0x7f3bcc0ca960 2024-03-18T16:25:13.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.602+0000 7f3bcbe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bcc15d930 con 0x7f3bcc0cad40 2024-03-18T16:25:13.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.602+0000 7f3bcbe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bcc070ae0 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.603+0000 7f3bcb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc15a1a0 0x7f3bcc1e7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:13.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.603+0000 7f3bcaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc0ca960 0x7f3bcc0cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:13.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.603+0000 7f3bca000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bcc0cad40 0x7f3bcc1e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:13.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.603+0000 7f3bcb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc15a1a0 0x7f3bcc1e7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50654/0 (socket says 172.21.15.102:50654) 2024-03-18T16:25:13.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.603+0000 7f3bcb400640 1 -- 172.21.15.102:0/1908053507 learned_addr learned my addr 172.21.15.102:0/1908053507 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:13.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.607+0000 7f3bcb400640 1 -- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bcc0cad40 msgr2=0x7f3bcc1e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:13.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.607+0000 7f3bcb400640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bcc0cad40 0x7f3bcc1e6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:13.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.607+0000 7f3bcb400640 1 -- 172.21.15.102:0/1908053507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc0ca960 msgr2=0x7f3bcc0cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:13.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.607+0000 7f3bcb400640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc0ca960 0x7f3bcc0cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:13.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.607+0000 7f3bcb400640 1 -- 172.21.15.102:0/1908053507 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bcc0cc5b0 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.609+0000 7f3bcb400640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc15a1a0 0x7f3bcc1e7110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400efc0 tx=0x7f3bc400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:13.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.612+0000 7f3bc2a00640 1 -- 172.21.15.102:0/1908053507 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4018020 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.613+0000 7f3bc2a00640 1 -- 172.21.15.102:0/1908053507 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc4014480 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.613+0000 7f3bc2a00640 1 -- 172.21.15.102:0/1908053507 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4014870 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.613+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bcc1f0db0 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.613+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bcc1f1090 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.614+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bcc0cca90 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.634+0000 7f3bc2a00640 1 -- 172.21.15.102:0/1908053507 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3bc4005d10 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.635+0000 7f3bc2a00640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bac06bc20 0x7f3bac06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:13.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.635+0000 7f3bcaa00640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bac06bc20 0x7f3bac06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:13.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.636+0000 7f3bcaa00640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bac06bc20 0x7f3bac06e0e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009a20 tx=0x7f3bbc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:13.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.637+0000 7f3bc2a00640 1 -- 172.21.15.102:0/1908053507 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3bc4058550 con 0x7f3bcc15a1a0 2024-03-18T16:25:13.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:13.888+0000 7f3bc2a00640 1 -- 172.21.15.102:0/1908053507 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3bcc0cca90 con 0x7f3bcc15a1a0 2024-03-18T16:25:14.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.185+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bcc06dd30 con 0x7f3bcc15a1a0 2024-03-18T16:25:14.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.199+0000 7f3bc2a00640 1 -- 172.21.15.102:0/1908053507 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3bcc06dd30 con 0x7f3bcc15a1a0 2024-03-18T16:25:14.201 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:14.201 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:14.201 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bac06bc20 msgr2=0x7f3bac06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bac06bc20 0x7f3bac06e0e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009a20 tx=0x7f3bbc009190 comp rx=0 tx=0).stop 2024-03-18T16:25:14.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc15a1a0 msgr2=0x7f3bcc1e7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc15a1a0 0x7f3bcc1e7110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400efc0 tx=0x7f3bc400c490 comp rx=0 tx=0).stop 2024-03-18T16:25:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 shutdown_connections 2024-03-18T16:25:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bac06bc20 0x7f3bac06e0e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bcc15a1a0 0x7f3bcc1e7110 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bcc0cad40 0x7f3bcc1e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 --2- 172.21.15.102:0/1908053507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc0ca960 0x7f3bcc0cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 >> 172.21.15.102:0/1908053507 conn(0x7f3bcc067190 msgr2=0x7f3bcc141270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 shutdown_connections 2024-03-18T16:25:14.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:14.201+0000 7f3bcbe00640 1 -- 172.21.15.102:0/1908053507 wait complete. 2024-03-18T16:25:14.824 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:14.825 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:15.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:16.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:15.999+0000 7fd9de000640 1 Processor -- start 2024-03-18T16:25:16.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:15.999+0000 7fd9de000640 1 -- start start 2024-03-18T16:25:16.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.001+0000 7fd9de000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d80c9be0 0x7fd9d80d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:16.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.001+0000 7fd9de000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d8154ab0 con 0x7fd9d815ae70 2024-03-18T16:25:16.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.001+0000 7fd9de000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d815b250 con 0x7fd9d80d05a0 2024-03-18T16:25:16.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.001+0000 7fd9de000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d80caba0 con 0x7fd9d80c9be0 2024-03-18T16:25:16.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.001+0000 7fd9dd600640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd9d80d05a0 0x7fd9d80cc4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33556/0 (socket says 172.21.15.102:33556) 2024-03-18T16:25:16.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.001+0000 7fd9dd600640 1 -- 172.21.15.102:0/3155292095 learned_addr learned my addr 172.21.15.102:0/3155292095 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:25:16.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.001+0000 7fd9d7e00640 1 --2- 172.21.15.102:0/3155292095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d80c9be0 0x7fd9d80d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.004+0000 7fd9d7e00640 1 -- 172.21.15.102:0/3155292095 >> v1:172.21.15.154:6789/0 conn(0x7fd9d80d05a0 legacy=0x7fd9d80cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.004+0000 7fd9d7e00640 1 -- 172.21.15.102:0/3155292095 >> v1:172.21.15.102:6789/0 conn(0x7fd9d815ae70 legacy=0x7fd9d80ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.004+0000 7fd9d7e00640 1 -- 172.21.15.102:0/3155292095 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d806d2f0 con 0x7fd9d80c9be0 2024-03-18T16:25:16.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.007+0000 7fd9d7e00640 1 --2- 172.21.15.102:0/3155292095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d80c9be0 0x7fd9d80d0060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40097d0 tx=0x7fd9c402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2572e4f29e893f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:16.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.009+0000 7fd9d7400640 1 -- 172.21.15.102:0/3155292095 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4034070 con 0x7fd9d80c9be0 2024-03-18T16:25:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.011+0000 7fd9d7400640 1 -- 172.21.15.102:0/3155292095 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c4030800 con 0x7fd9d80c9be0 2024-03-18T16:25:16.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.011+0000 7fd9d7400640 1 -- 172.21.15.102:0/3155292095 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4030bd0 con 0x7fd9d80c9be0 2024-03-18T16:25:16.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.012+0000 7fd9de000640 1 -- 172.21.15.102:0/3155292095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d80c9be0 msgr2=0x7fd9d80d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:16.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.012+0000 7fd9de000640 1 --2- 172.21.15.102:0/3155292095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d80c9be0 0x7fd9d80d0060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40097d0 tx=0x7fd9c402b940 comp rx=0 tx=0).stop 2024-03-18T16:25:16.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.012+0000 7fd9de000640 1 -- 172.21.15.102:0/3155292095 shutdown_connections 2024-03-18T16:25:16.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.012+0000 7fd9de000640 1 --2- 172.21.15.102:0/3155292095 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d80c9be0 0x7fd9d80d0060 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.012+0000 7fd9de000640 1 -- 172.21.15.102:0/3155292095 >> 172.21.15.102:0/3155292095 conn(0x7fd9d80673a0 msgr2=0x7fd9d80677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.012+0000 7fd9de000640 1 -- 172.21.15.102:0/3155292095 shutdown_connections 2024-03-18T16:25:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.012+0000 7fd9de000640 1 -- 172.21.15.102:0/3155292095 wait complete. 2024-03-18T16:25:16.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.014+0000 7fd9de000640 1 Processor -- start 2024-03-18T16:25:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.014+0000 7fd9de000640 1 -- start start 2024-03-18T16:25:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.015+0000 7fd9de000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d80c9be0 0x7fd9d80ac310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.015+0000 7fd9de000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d80d05a0 0x7fd9d81f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.015+0000 7fd9de000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d815ae70 0x7fd9d80c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.015+0000 7fd9de000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d806db90 con 0x7fd9d80d05a0 2024-03-18T16:25:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.015+0000 7fd9de000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d806d8b0 con 0x7fd9d80c9be0 2024-03-18T16:25:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.015+0000 7fd9de000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d806de70 con 0x7fd9d815ae70 2024-03-18T16:25:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.015+0000 7fd9dd600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d815ae70 0x7fd9d80c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.016+0000 7fd9dcc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d80c9be0 0x7fd9d80ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.016+0000 7fd9d7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d80d05a0 0x7fd9d81f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.016+0000 7fd9dd600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d815ae70 0x7fd9d80c4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50676/0 (socket says 172.21.15.102:50676) 2024-03-18T16:25:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.016+0000 7fd9dd600640 1 -- 172.21.15.102:0/1707465171 learned_addr learned my addr 172.21.15.102:0/1707465171 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:16.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.019+0000 7fd9dd600640 1 -- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d80c9be0 msgr2=0x7fd9d80ac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:16.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.019+0000 7fd9dd600640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d80c9be0 0x7fd9d80ac310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:16.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.019+0000 7fd9dd600640 1 -- 172.21.15.102:0/1707465171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d80d05a0 msgr2=0x7fd9d81f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:16.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.019+0000 7fd9dd600640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d80d05a0 0x7fd9d81f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:16.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.019+0000 7fd9dd600640 1 -- 172.21.15.102:0/1707465171 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d80add10 con 0x7fd9d815ae70 2024-03-18T16:25:16.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.019+0000 7fd9dcc00640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d80c9be0 0x7fd9d80ac310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:25:16.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.022+0000 7fd9dd600640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d815ae70 0x7fd9d80c4b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd9d000cab0 tx=0x7fd9d000cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:16.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.024+0000 7fd9d4c00640 1 -- 172.21.15.102:0/1707465171 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0017070 con 0x7fd9d815ae70 2024-03-18T16:25:16.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.025+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d80adf80 con 0x7fd9d815ae70 2024-03-18T16:25:16.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.025+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d80ae540 con 0x7fd9d815ae70 2024-03-18T16:25:16.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.026+0000 7fd9d4c00640 1 -- 172.21.15.102:0/1707465171 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d000ac30 con 0x7fd9d815ae70 2024-03-18T16:25:16.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.026+0000 7fd9d4c00640 1 -- 172.21.15.102:0/1707465171 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0021060 con 0x7fd9d815ae70 2024-03-18T16:25:16.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.026+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d8074fd0 con 0x7fd9d815ae70 2024-03-18T16:25:16.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.294+0000 7fd9d4c00640 1 -- 172.21.15.102:0/1707465171 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd9d001a070 con 0x7fd9d815ae70 2024-03-18T16:25:16.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.296+0000 7fd9d4c00640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd9b806bbd0 0x7fd9b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:16.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.296+0000 7fd9dcc00640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd9b806bbd0 0x7fd9b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:16.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.296+0000 7fd9d4c00640 1 -- 172.21.15.102:0/1707465171 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd9d0057790 con 0x7fd9d815ae70 2024-03-18T16:25:16.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.297+0000 7fd9dcc00640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd9b806bbd0 0x7fd9b806e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc001b70 tx=0x7fd9cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:16.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.313+0000 7fd9d4c00640 1 -- 172.21.15.102:0/1707465171 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd9d00596f0 con 0x7fd9d815ae70 2024-03-18T16:25:16.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.576+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9d81592a0 con 0x7fd9d815ae70 2024-03-18T16:25:16.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.589+0000 7fd9d4c00640 1 -- 172.21.15.102:0/1707465171 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd9d005d790 con 0x7fd9d815ae70 2024-03-18T16:25:16.592 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:16.592 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:16.592 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:16.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd9b806bbd0 msgr2=0x7fd9b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd9b806bbd0 0x7fd9b806e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc001b70 tx=0x7fd9cc009040 comp rx=0 tx=0).stop 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d815ae70 msgr2=0x7fd9d80c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d815ae70 0x7fd9d80c4b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd9d000cab0 tx=0x7fd9d000cf70 comp rx=0 tx=0).stop 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 shutdown_connections 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd9b806bbd0 0x7fd9b806e090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd9d815ae70 0x7fd9d80c4b50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d80d05a0 0x7fd9d81f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 --2- 172.21.15.102:0/1707465171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d80c9be0 0x7fd9d80ac310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.591+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 >> 172.21.15.102:0/1707465171 conn(0x7fd9d80673a0 msgr2=0x7fd9d80c62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.592+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 shutdown_connections 2024-03-18T16:25:16.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:16.592+0000 7fd9de000640 1 -- 172.21.15.102:0/1707465171 wait complete. 2024-03-18T16:25:17.216 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:17.216 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:18.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:18.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.384+0000 7f3a43000640 1 Processor -- start 2024-03-18T16:25:18.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.384+0000 7f3a43000640 1 -- start start 2024-03-18T16:25:18.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.385+0000 7f3a43000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3a3c0d0250 0x7f3a3c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:18.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.385+0000 7f3a43000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a3c0cabd0 con 0x7f3a3c0cfe70 2024-03-18T16:25:18.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.385+0000 7f3a43000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a3c0caea0 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.385+0000 7f3a43000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3a3c0cb180 con 0x7f3a3c0d0250 2024-03-18T16:25:18.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.386+0000 7f3a42600640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3a3c0cc2f0 0x7f3a3c0cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33568/0 (socket says 172.21.15.102:33568) 2024-03-18T16:25:18.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.386+0000 7f3a42600640 1 -- 172.21.15.102:0/1373840997 learned_addr learned my addr 172.21.15.102:0/1373840997 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:25:18.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.386+0000 7f3a41200640 1 --2- 172.21.15.102:0/1373840997 >> v2:172.21.15.102:3300/0 conn(0x7f3a3c0d0250 0x7f3a3c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:18.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.389+0000 7f3a3be00640 1 -- 172.21.15.102:0/1373840997 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1257980035 0 0) 0x7f3a3c0caea0 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.389+0000 7f3a3be00640 1 -- 172.21.15.102:0/1373840997 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a24003520 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.391+0000 7f3a41200640 1 -- 172.21.15.102:0/1373840997 >> v1:172.21.15.154:6789/0 conn(0x7f3a3c0cc2f0 legacy=0x7f3a3c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:18.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.391+0000 7f3a41200640 1 -- 172.21.15.102:0/1373840997 >> v1:172.21.15.102:6789/0 conn(0x7f3a3c0cfe70 legacy=0x7f3a3c0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:18.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.391+0000 7f3a41200640 1 -- 172.21.15.102:0/1373840997 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3c0cb460 con 0x7f3a3c0d0250 2024-03-18T16:25:18.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.398+0000 7f3a41200640 1 --2- 172.21.15.102:0/1373840997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a3c0d0250 0x7f3a3c15ae40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3a280097a0 tx=0x7f3a2802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92dd4fddb400480 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:18.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.403+0000 7f3a3be00640 1 -- 172.21.15.102:0/1373840997 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2802f810 con 0x7f3a3c0d0250 2024-03-18T16:25:18.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.403+0000 7f3a3be00640 1 -- 172.21.15.102:0/1373840997 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a2802f9b0 con 0x7f3a3c0d0250 2024-03-18T16:25:18.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.403+0000 7f3a3be00640 1 -- 172.21.15.102:0/1373840997 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2802fd80 con 0x7f3a3c0d0250 2024-03-18T16:25:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.403+0000 7f3a43000640 1 -- 172.21.15.102:0/1373840997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a3c0d0250 msgr2=0x7f3a3c15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.403+0000 7f3a43000640 1 --2- 172.21.15.102:0/1373840997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a3c0d0250 0x7f3a3c15ae40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3a280097a0 tx=0x7f3a2802b9f0 comp rx=0 tx=0).stop 2024-03-18T16:25:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.404+0000 7f3a43000640 1 -- 172.21.15.102:0/1373840997 shutdown_connections 2024-03-18T16:25:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.404+0000 7f3a43000640 1 --2- 172.21.15.102:0/1373840997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a3c0d0250 0x7f3a3c15ae40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:18.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.404+0000 7f3a43000640 1 -- 172.21.15.102:0/1373840997 >> 172.21.15.102:0/1373840997 conn(0x7f3a3c08c3e0 msgr2=0x7f3a3c06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:18.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.404+0000 7f3a43000640 1 -- 172.21.15.102:0/1373840997 shutdown_connections 2024-03-18T16:25:18.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.404+0000 7f3a43000640 1 -- 172.21.15.102:0/1373840997 wait complete. 2024-03-18T16:25:18.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.406+0000 7f3a43000640 1 Processor -- start 2024-03-18T16:25:18.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.407+0000 7f3a43000640 1 -- start start 2024-03-18T16:25:18.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.408+0000 7f3a43000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a3c0cc2f0 0x7f3a3c0c4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:18.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.408+0000 7f3a43000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a3c0cfe70 0x7f3a3c159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:18.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.408+0000 7f3a43000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a3c0d0250 0x7f3a3c150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:18.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.408+0000 7f3a43000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c0cbd00 con 0x7f3a3c0cfe70 2024-03-18T16:25:18.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.408+0000 7f3a43000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3c0cba20 con 0x7f3a3c0d0250 2024-03-18T16:25:18.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.408+0000 7f3a43000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3a3c0c8330 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.409+0000 7f3a42600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a3c0d0250 0x7f3a3c150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:18.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.409+0000 7f3a41c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a3c0cc2f0 0x7f3a3c0c4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:18.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.409+0000 7f3a41200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a3c0cfe70 0x7f3a3c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:18.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.409+0000 7f3a42600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a3c0d0250 0x7f3a3c150cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59126/0 (socket says 172.21.15.102:59126) 2024-03-18T16:25:18.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.409+0000 7f3a42600640 1 -- 172.21.15.102:0/1967341366 learned_addr learned my addr 172.21.15.102:0/1967341366 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:18.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.409+0000 7f3a41c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a3c0cc2f0 0x7f3a3c0c4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50682/0 (socket says 172.21.15.102:50682) 2024-03-18T16:25:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.413+0000 7f3a41c00640 1 -- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a3c0d0250 msgr2=0x7f3a3c150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:18.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.413+0000 7f3a41c00640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a3c0d0250 0x7f3a3c150cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.413+0000 7f3a41c00640 1 -- 172.21.15.102:0/1967341366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a3c0cfe70 msgr2=0x7f3a3c159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.413+0000 7f3a41c00640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a3c0cfe70 0x7f3a3c159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:18.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.413+0000 7f3a41c00640 1 -- 172.21.15.102:0/1967341366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3c1f3550 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.415+0000 7f3a41c00640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a3c0cc2f0 0x7f3a3c0c4450 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000c340 tx=0x7f3a3000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:18.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.419+0000 7f3a39600640 1 -- 172.21.15.102:0/1967341366 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a300052d0 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.419+0000 7f3a39600640 1 -- 172.21.15.102:0/1967341366 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a30005490 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.419+0000 7f3a39600640 1 -- 172.21.15.102:0/1967341366 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30007260 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.419+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3c1f3830 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.419+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3c1f3df0 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.421+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a3c0ce130 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.447+0000 7f3a39600640 1 -- 172.21.15.102:0/1967341366 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3a30007400 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.449+0000 7f3a39600640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a1c06bc50 0x7f3a1c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:18.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.450+0000 7f3a41200640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a1c06bc50 0x7f3a1c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:18.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.451+0000 7f3a41200640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a1c06bc50 0x7f3a1c06e110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3a28036000 tx=0x7f3a28008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:18.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.730+0000 7f3a39600640 1 -- 172.21.15.102:0/1967341366 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3a30056b80 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.741+0000 7f3a39600640 1 -- 172.21.15.102:0/1967341366 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3a30020070 con 0x7f3a3c0cc2f0 2024-03-18T16:25:18.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:18.995+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a3c0cde50 con 0x7f3a3c0cc2f0 2024-03-18T16:25:19.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.115+0000 7f3a39600640 1 -- 172.21.15.102:0/1967341366 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3a30058ae0 con 0x7f3a3c0cc2f0 2024-03-18T16:25:19.118 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:19.118 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:19.118 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:19.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.118+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a1c06bc50 msgr2=0x7f3a1c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:19.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a1c06bc50 0x7f3a1c06e110 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3a28036000 tx=0x7f3a28008030 comp rx=0 tx=0).stop 2024-03-18T16:25:19.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a3c0cc2f0 msgr2=0x7f3a3c0c4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:19.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a3c0cc2f0 0x7f3a3c0c4450 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000c340 tx=0x7f3a3000c800 comp rx=0 tx=0).stop 2024-03-18T16:25:19.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 shutdown_connections 2024-03-18T16:25:19.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a1c06bc50 0x7f3a1c06e110 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:19.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a3c0d0250 0x7f3a3c150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:19.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a3c0cfe70 0x7f3a3c159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:19.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 --2- 172.21.15.102:0/1967341366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a3c0cc2f0 0x7f3a3c0c4450 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:19.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 >> 172.21.15.102:0/1967341366 conn(0x7f3a3c08c3e0 msgr2=0x7f3a3c0c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:19.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 shutdown_connections 2024-03-18T16:25:19.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:19.119+0000 7f3a43000640 1 -- 172.21.15.102:0/1967341366 wait complete. 2024-03-18T16:25:19.745 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:19.745 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:20.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:20.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.937+0000 7f2024a00640 1 Processor -- start 2024-03-18T16:25:20.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.937+0000 7f2024a00640 1 -- start start 2024-03-18T16:25:20.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.938+0000 7f2024a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f20201524c0 0x7f2020072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:20.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.938+0000 7f2024a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f202015b8e0 0x7f20200c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:20.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.938+0000 7f2024a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f202015c200 con 0x7f2020158ee0 2024-03-18T16:25:20.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.938+0000 7f2024a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2020159300 con 0x7f20201524c0 2024-03-18T16:25:20.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.938+0000 7f2024a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f202015c4e0 con 0x7f202015b8e0 2024-03-18T16:25:20.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.939+0000 7f201e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f202015b8e0 0x7f20200c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:20.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.939+0000 7f201de00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f20201524c0 0x7f2020072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:20.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.939+0000 7f201e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f202015b8e0 0x7f20200c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48904/0 (socket says 172.21.15.102:48904) 2024-03-18T16:25:20.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.939+0000 7f201e800640 1 -- 172.21.15.102:0/1035811078 learned_addr learned my addr 172.21.15.102:0/1035811078 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:20.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.939+0000 7f201de00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f20201524c0 0x7f2020072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41406/0 (socket says 172.21.15.102:41406) 2024-03-18T16:25:20.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.942+0000 7f201e800640 1 -- 172.21.15.102:0/1035811078 >> v2:172.21.15.154:3300/0 conn(0x7f20201524c0 msgr2=0x7f2020072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:20.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.942+0000 7f201e800640 1 --2- 172.21.15.102:0/1035811078 >> v2:172.21.15.154:3300/0 conn(0x7f20201524c0 0x7f2020072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:20.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.942+0000 7f201e800640 1 -- 172.21.15.102:0/1035811078 >> v1:172.21.15.154:6789/0 conn(0x7f2020158ee0 legacy=0x7f202006e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:20.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.943+0000 7f201e800640 1 -- 172.21.15.102:0/1035811078 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202015c7c0 con 0x7f202015b8e0 2024-03-18T16:25:20.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.945+0000 7f201e800640 1 --2- 172.21.15.102:0/1035811078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f202015b8e0 0x7f20200c51e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2018009e50 tx=0x7f201802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50ffc6fa4cc30d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:20.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.948+0000 7f201ca00640 1 -- 172.21.15.102:0/1035811078 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201802cb00 con 0x7f202015b8e0 2024-03-18T16:25:20.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.948+0000 7f201ca00640 1 -- 172.21.15.102:0/1035811078 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f201802cca0 con 0x7f202015b8e0 2024-03-18T16:25:20.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.948+0000 7f201ca00640 1 -- 172.21.15.102:0/1035811078 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018034610 con 0x7f202015b8e0 2024-03-18T16:25:20.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.949+0000 7f2024a00640 1 -- 172.21.15.102:0/1035811078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f202015b8e0 msgr2=0x7f20200c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:20.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.949+0000 7f2024a00640 1 --2- 172.21.15.102:0/1035811078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f202015b8e0 0x7f20200c51e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2018009e50 tx=0x7f201802bf60 comp rx=0 tx=0).stop 2024-03-18T16:25:20.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.949+0000 7f2024a00640 1 -- 172.21.15.102:0/1035811078 shutdown_connections 2024-03-18T16:25:20.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.949+0000 7f2024a00640 1 --2- 172.21.15.102:0/1035811078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f202015b8e0 0x7f20200c51e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:20.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.949+0000 7f2024a00640 1 --2- 172.21.15.102:0/1035811078 >> v2:172.21.15.154:3300/0 conn(0x7f20201524c0 0x7f2020072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:20.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.949+0000 7f2024a00640 1 -- 172.21.15.102:0/1035811078 >> 172.21.15.102:0/1035811078 conn(0x7f20200700d0 msgr2=0x7f202006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:20.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.949+0000 7f2024a00640 1 -- 172.21.15.102:0/1035811078 shutdown_connections 2024-03-18T16:25:20.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.949+0000 7f2024a00640 1 -- 172.21.15.102:0/1035811078 wait complete. 2024-03-18T16:25:20.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.951+0000 7f2024a00640 1 Processor -- start 2024-03-18T16:25:20.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.951+0000 7f2024a00640 1 -- start start 2024-03-18T16:25:20.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.952+0000 7f2024a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201524c0 0x7f202014cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:20.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.952+0000 7f2024a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2020158ee0 0x7f2020146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:20.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.952+0000 7f2024a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f202015b8e0 0x7f202015ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:20.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.952+0000 7f2024a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f20201589c0 con 0x7f202015b8e0 2024-03-18T16:25:20.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.952+0000 7f2024a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20201586e0 con 0x7f20201524c0 2024-03-18T16:25:20.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.952+0000 7f2024a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f20201528a0 con 0x7f2020158ee0 2024-03-18T16:25:20.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.952+0000 7f201d400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2020158ee0 0x7f2020146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:20.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.953+0000 7f201de00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201524c0 0x7f202014cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:20.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.953+0000 7f201d400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2020158ee0 0x7f2020146e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48910/0 (socket says 172.21.15.102:48910) 2024-03-18T16:25:20.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.953+0000 7f201d400640 1 -- 172.21.15.102:0/3751439586 learned_addr learned my addr 172.21.15.102:0/3751439586 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:20.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.953+0000 7f201e800640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f202015b8e0 0x7f202015ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:20.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.956+0000 7f201d400640 1 -- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201524c0 msgr2=0x7f202014cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:20.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.956+0000 7f201d400640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201524c0 0x7f202014cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:20.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.956+0000 7f201d400640 1 -- 172.21.15.102:0/3751439586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f202015b8e0 msgr2=0x7f202015ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:20.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.956+0000 7f201d400640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f202015b8e0 0x7f202015ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:20.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.956+0000 7f201d400640 1 -- 172.21.15.102:0/3751439586 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20201f0bd0 con 0x7f2020158ee0 2024-03-18T16:25:20.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.958+0000 7f201d400640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2020158ee0 0x7f2020146e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f200c00c430 tx=0x7f200c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:20.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.960+0000 7f2016000640 1 -- 172.21.15.102:0/3751439586 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c0052d0 con 0x7f2020158ee0 2024-03-18T16:25:20.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.960+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2020159b10 con 0x7f2020158ee0 2024-03-18T16:25:20.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.961+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2020159830 con 0x7f2020158ee0 2024-03-18T16:25:20.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.962+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2020074fd0 con 0x7f2020158ee0 2024-03-18T16:25:20.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.981+0000 7f2016000640 1 -- 172.21.15.102:0/3751439586 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f200c005470 con 0x7f2020158ee0 2024-03-18T16:25:20.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.981+0000 7f2016000640 1 -- 172.21.15.102:0/3751439586 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c007250 con 0x7f2020158ee0 2024-03-18T16:25:20.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.983+0000 7f2016000640 1 -- 172.21.15.102:0/3751439586 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f200c0074b0 con 0x7f2020158ee0 2024-03-18T16:25:20.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.985+0000 7f2016000640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f200006bbd0 0x7f200006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:20.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.986+0000 7f201de00640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f200006bbd0 0x7f200006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:20.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.986+0000 7f201de00640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f200006bbd0 0x7f200006e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2010009aa0 tx=0x7f2010009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:20.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:20.987+0000 7f2016000640 1 -- 172.21.15.102:0/3751439586 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f200c056850 con 0x7f2020158ee0 2024-03-18T16:25:21.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.237+0000 7f2016000640 1 -- 172.21.15.102:0/3751439586 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f200c020020 con 0x7f2020158ee0 2024-03-18T16:25:21.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:25:21.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:25:21.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.549+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f202015a8d0 con 0x7f2020158ee0 2024-03-18T16:25:21.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.562+0000 7f2016000640 1 -- 172.21.15.102:0/3751439586 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f200c0587b0 con 0x7f2020158ee0 2024-03-18T16:25:21.565 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:21.565 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:21.566 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f200006bbd0 msgr2=0x7f200006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f200006bbd0 0x7f200006e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2010009aa0 tx=0x7f2010009190 comp rx=0 tx=0).stop 2024-03-18T16:25:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2020158ee0 msgr2=0x7f2020146e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2020158ee0 0x7f2020146e70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f200c00c430 tx=0x7f200c00c8f0 comp rx=0 tx=0).stop 2024-03-18T16:25:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 shutdown_connections 2024-03-18T16:25:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f200006bbd0 0x7f200006e090 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f202015b8e0 0x7f202015ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:21.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2020158ee0 0x7f2020146e70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:21.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 --2- 172.21.15.102:0/3751439586 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20201524c0 0x7f202014cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:21.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 >> 172.21.15.102:0/3751439586 conn(0x7f20200700d0 msgr2=0x7f2020071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:21.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 shutdown_connections 2024-03-18T16:25:21.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:21.565+0000 7f2024a00640 1 -- 172.21.15.102:0/3751439586 wait complete. 2024-03-18T16:25:22.188 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:22.188 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:23.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:23.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.369+0000 7f319ca00640 1 Processor -- start 2024-03-18T16:25:23.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.369+0000 7f319ca00640 1 -- start start 2024-03-18T16:25:23.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.371+0000 7f319ca00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f3198158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:23.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.371+0000 7f319ca00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31980c72d0 con 0x7f31980c4fa0 2024-03-18T16:25:23.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.371+0000 7f319ca00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3198158ad0 con 0x7f319815c3a0 2024-03-18T16:25:23.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.371+0000 7f319ca00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3198159370 con 0x7f3198141650 2024-03-18T16:25:23.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.373+0000 7f3196a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f319815c3a0 0x7f31980c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:57670/0 (socket says 172.21.15.102:57670) 2024-03-18T16:25:23.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.373+0000 7f3196a00640 1 -- 172.21.15.102:0/3042589009 learned_addr learned my addr 172.21.15.102:0/3042589009 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:25:23.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.373+0000 7f3197400640 1 --2- 172.21.15.102:0/3042589009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f3198158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:23.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.376+0000 7f3196000640 1 -- 172.21.15.102:0/3042589009 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3199898996 0 0) 0x7f3198158ad0 con 0x7f319815c3a0 2024-03-18T16:25:23.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.376+0000 7f3196000640 1 -- 172.21.15.102:0/3042589009 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f317c003520 con 0x7f319815c3a0 2024-03-18T16:25:23.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.376+0000 7f3197400640 1 -- 172.21.15.102:0/3042589009 >> v1:172.21.15.154:6789/0 conn(0x7f319815c3a0 legacy=0x7f31980c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:23.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.376+0000 7f3197400640 1 -- 172.21.15.102:0/3042589009 >> v1:172.21.15.102:6789/0 conn(0x7f31980c4fa0 legacy=0x7f3198162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:23.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.376+0000 7f3197400640 1 -- 172.21.15.102:0/3042589009 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3198158db0 con 0x7f3198141650 2024-03-18T16:25:23.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.378+0000 7f3197400640 1 --2- 172.21.15.102:0/3042589009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f3198158420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f31880097d0 tx=0x7f318802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b3ae754c3692438 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:23.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.382+0000 7f3196000640 1 -- 172.21.15.102:0/3042589009 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188035020 con 0x7f3198141650 2024-03-18T16:25:23.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.382+0000 7f3196000640 1 -- 172.21.15.102:0/3042589009 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3188031810 con 0x7f3198141650 2024-03-18T16:25:23.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.382+0000 7f3196000640 1 -- 172.21.15.102:0/3042589009 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188031be0 con 0x7f3198141650 2024-03-18T16:25:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.384+0000 7f319ca00640 1 -- 172.21.15.102:0/3042589009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 msgr2=0x7f3198158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.384+0000 7f319ca00640 1 --2- 172.21.15.102:0/3042589009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f3198158420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f31880097d0 tx=0x7f318802b8b0 comp rx=0 tx=0).stop 2024-03-18T16:25:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.384+0000 7f319ca00640 1 -- 172.21.15.102:0/3042589009 shutdown_connections 2024-03-18T16:25:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.384+0000 7f319ca00640 1 --2- 172.21.15.102:0/3042589009 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f3198158420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.384+0000 7f319ca00640 1 -- 172.21.15.102:0/3042589009 >> 172.21.15.102:0/3042589009 conn(0x7f31980cb890 msgr2=0x7f31980cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.384+0000 7f319ca00640 1 -- 172.21.15.102:0/3042589009 shutdown_connections 2024-03-18T16:25:23.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.384+0000 7f319ca00640 1 -- 172.21.15.102:0/3042589009 wait complete. 2024-03-18T16:25:23.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.385+0000 7f319ca00640 1 Processor -- start 2024-03-18T16:25:23.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.385+0000 7f319ca00640 1 -- start start 2024-03-18T16:25:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.387+0000 7f319ca00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31980c4fa0 0x7f3198152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.387+0000 7f319ca00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f31980c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.387+0000 7f319ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f319815c3a0 0x7f3198156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.387+0000 7f319ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f31980c66a0 con 0x7f319815c3a0 2024-03-18T16:25:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.387+0000 7f319ca00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31980c63c0 con 0x7f31980c4fa0 2024-03-18T16:25:23.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.387+0000 7f319ca00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f319815c7c0 con 0x7f3198141650 2024-03-18T16:25:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.387+0000 7f3196a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f31980c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.388+0000 7f3197400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31980c4fa0 0x7f3198152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.388+0000 7f3197e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f319815c3a0 0x7f3198156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.388+0000 7f3196a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f31980c4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48918/0 (socket says 172.21.15.102:48918) 2024-03-18T16:25:23.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.388+0000 7f3196a00640 1 -- 172.21.15.102:0/3071096100 learned_addr learned my addr 172.21.15.102:0/3071096100 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.392+0000 7f3197e00640 1 -- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 msgr2=0x7f31980c4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.392+0000 7f3197e00640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f31980c4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.392+0000 7f3197e00640 1 -- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31980c4fa0 msgr2=0x7f3198152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.392+0000 7f3197e00640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31980c4fa0 0x7f3198152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:23.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.392+0000 7f3197e00640 1 -- 172.21.15.102:0/3071096100 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3198072120 con 0x7f319815c3a0 2024-03-18T16:25:23.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.393+0000 7f3197e00640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f319815c3a0 0x7f3198156d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f319000c690 tx=0x7f319000cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.397+0000 7f318f400640 1 -- 172.21.15.102:0/3071096100 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190003b70 con 0x7f319815c3a0 2024-03-18T16:25:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.397+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31981558a0 con 0x7f319815c3a0 2024-03-18T16:25:23.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.397+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31980cb130 con 0x7f319815c3a0 2024-03-18T16:25:23.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.399+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3198155d60 con 0x7f319815c3a0 2024-03-18T16:25:23.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.400+0000 7f318f400640 1 -- 172.21.15.102:0/3071096100 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3190003d10 con 0x7f319815c3a0 2024-03-18T16:25:23.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.400+0000 7f318f400640 1 -- 172.21.15.102:0/3071096100 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190001780 con 0x7f319815c3a0 2024-03-18T16:25:23.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.681+0000 7f318f400640 1 -- 172.21.15.102:0/3071096100 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f31900052d0 con 0x7f319815c3a0 2024-03-18T16:25:23.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.682+0000 7f318f400640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f317806bbd0 0x7f317806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:23.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.683+0000 7f3197400640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f317806bbd0 0x7f317806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:23.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.683+0000 7f3197400640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f317806bbd0 0x7f317806e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3188036000 tx=0x7f3188008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:23.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.706+0000 7f318f400640 1 -- 172.21.15.102:0/3071096100 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3190057770 con 0x7f319815c3a0 2024-03-18T16:25:23.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:23.706+0000 7f318f400640 1 -- 172.21.15.102:0/3071096100 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f31900596d0 con 0x7f319815c3a0 2024-03-18T16:25:24.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.068+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31980c7af0 con 0x7f319815c3a0 2024-03-18T16:25:24.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.085+0000 7f318f400640 1 -- 172.21.15.102:0/3071096100 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f31980c7af0 con 0x7f319815c3a0 2024-03-18T16:25:24.087 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:24.087 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:24.088 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:24.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f317806bbd0 msgr2=0x7f317806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:24.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f317806bbd0 0x7f317806e090 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3188036000 tx=0x7f3188008030 comp rx=0 tx=0).stop 2024-03-18T16:25:24.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f319815c3a0 msgr2=0x7f3198156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:24.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f319815c3a0 0x7f3198156d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f319000c690 tx=0x7f319000cb50 comp rx=0 tx=0).stop 2024-03-18T16:25:24.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 shutdown_connections 2024-03-18T16:25:24.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f317806bbd0 0x7f317806e090 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:24.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f319815c3a0 0x7f3198156d80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:24.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3198141650 0x7f31980c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:24.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 --2- 172.21.15.102:0/3071096100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31980c4fa0 0x7f3198152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:24.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 >> 172.21.15.102:0/3071096100 conn(0x7f31980cb890 msgr2=0x7f319813ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:24.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 shutdown_connections 2024-03-18T16:25:24.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:24.089+0000 7f319ca00640 1 -- 172.21.15.102:0/3071096100 wait complete. 2024-03-18T16:25:24.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:24.719 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:25.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:25.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.895+0000 7f7db6c00640 1 Processor -- start 2024-03-18T16:25:25.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.895+0000 7f7db6c00640 1 -- start start 2024-03-18T16:25:25.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.897+0000 7f7db6c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:25.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.897+0000 7f7db6c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7db015baa0 0x7f7db015bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:25.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.897+0000 7f7db6c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db015c470 con 0x7f7db015a310 2024-03-18T16:25:25.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.897+0000 7f7db6c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7db00cd410 con 0x7f7db015baa0 2024-03-18T16:25:25.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.897+0000 7f7db6c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7db00ca500 con 0x7f7db015a6f0 2024-03-18T16:25:25.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.898+0000 7f7db4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:25.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.898+0000 7f7db4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db015ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48934/0 (socket says 172.21.15.102:48934) 2024-03-18T16:25:25.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.898+0000 7f7db4e00640 1 -- 172.21.15.102:0/1149749705 learned_addr learned my addr 172.21.15.102:0/1149749705 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:25.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.898+0000 7f7db6200640 1 --2- 172.21.15.102:0/1149749705 >> v2:172.21.15.102:3300/0 conn(0x7f7db015baa0 0x7f7db015bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:25.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.900+0000 7f7db4e00640 1 -- 172.21.15.102:0/1149749705 >> v2:172.21.15.102:3300/0 conn(0x7f7db015baa0 msgr2=0x7f7db015bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:25.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.900+0000 7f7db4e00640 1 --2- 172.21.15.102:0/1149749705 >> v2:172.21.15.102:3300/0 conn(0x7f7db015baa0 0x7f7db015bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:25.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.900+0000 7f7db4e00640 1 -- 172.21.15.102:0/1149749705 >> v1:172.21.15.102:6789/0 conn(0x7f7db015a310 legacy=0x7f7db006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:25.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.900+0000 7f7db4e00640 1 -- 172.21.15.102:0/1149749705 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db015b050 con 0x7f7db015a6f0 2024-03-18T16:25:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.902+0000 7f7db4e00640 1 --2- 172.21.15.102:0/1149749705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db015ab10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009870 tx=0x7f7d9c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0618b65668b048c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:25.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.904+0000 7f7dafe00640 1 -- 172.21.15.102:0/1149749705 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c034070 con 0x7f7db015a6f0 2024-03-18T16:25:25.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.906+0000 7f7dafe00640 1 -- 172.21.15.102:0/1149749705 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d9c030840 con 0x7f7db015a6f0 2024-03-18T16:25:25.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.906+0000 7f7dafe00640 1 -- 172.21.15.102:0/1149749705 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c030c10 con 0x7f7db015a6f0 2024-03-18T16:25:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.907+0000 7f7db6c00640 1 -- 172.21.15.102:0/1149749705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 msgr2=0x7f7db015ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.907+0000 7f7db6c00640 1 --2- 172.21.15.102:0/1149749705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db015ab10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009870 tx=0x7f7d9c02b980 comp rx=0 tx=0).stop 2024-03-18T16:25:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.907+0000 7f7db6c00640 1 -- 172.21.15.102:0/1149749705 shutdown_connections 2024-03-18T16:25:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.907+0000 7f7db6c00640 1 --2- 172.21.15.102:0/1149749705 >> v2:172.21.15.102:3300/0 conn(0x7f7db015baa0 0x7f7db015bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.907+0000 7f7db6c00640 1 --2- 172.21.15.102:0/1149749705 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db015ab10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.907+0000 7f7db6c00640 1 -- 172.21.15.102:0/1149749705 >> 172.21.15.102:0/1149749705 conn(0x7f7db008c1a0 msgr2=0x7f7db0066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.908+0000 7f7db6c00640 1 -- 172.21.15.102:0/1149749705 shutdown_connections 2024-03-18T16:25:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.908+0000 7f7db6c00640 1 -- 172.21.15.102:0/1149749705 wait complete. 2024-03-18T16:25:25.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.909+0000 7f7db6c00640 1 Processor -- start 2024-03-18T16:25:25.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.909+0000 7f7db6c00640 1 -- start start 2024-03-18T16:25:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.910+0000 7f7db6c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db015a310 0x7f7db00c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.910+0000 7f7db6c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db00c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.910+0000 7f7db6c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db015baa0 0x7f7db014bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.910+0000 7f7db6c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7db00cce50 con 0x7f7db015baa0 2024-03-18T16:25:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.910+0000 7f7db6c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7db00caa80 con 0x7f7db015a310 2024-03-18T16:25:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.910+0000 7f7db6c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7db00cd130 con 0x7f7db015a6f0 2024-03-18T16:25:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.911+0000 7f7db4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db00c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.911+0000 7f7db5800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db015a310 0x7f7db00c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:25.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.911+0000 7f7db6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db015baa0 0x7f7db014bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.911+0000 7f7db4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db00c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48948/0 (socket says 172.21.15.102:48948) 2024-03-18T16:25:25.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.911+0000 7f7db4e00640 1 -- 172.21.15.102:0/2101336079 learned_addr learned my addr 172.21.15.102:0/2101336079 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:25.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.915+0000 7f7db5800640 1 -- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 msgr2=0x7f7db00c8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.915+0000 7f7db5800640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db00c8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.915+0000 7f7db5800640 1 -- 172.21.15.102:0/2101336079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db015baa0 msgr2=0x7f7db014bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.915+0000 7f7db5800640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db015baa0 0x7f7db014bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:25.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.915+0000 7f7db5800640 1 -- 172.21.15.102:0/2101336079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db00c66c0 con 0x7f7db015a310 2024-03-18T16:25:25.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.917+0000 7f7db5800640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db015a310 0x7f7db00c7ed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c3c0 tx=0x7f7da400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:25.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.921+0000 7f7dad600640 1 -- 172.21.15.102:0/2101336079 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da40052d0 con 0x7f7db015a310 2024-03-18T16:25:25.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.921+0000 7f7dad600640 1 -- 172.21.15.102:0/2101336079 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da4005490 con 0x7f7db015a310 2024-03-18T16:25:25.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.921+0000 7f7dad600640 1 -- 172.21.15.102:0/2101336079 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da4007260 con 0x7f7db015a310 2024-03-18T16:25:25.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.921+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db01515b0 con 0x7f7db015a310 2024-03-18T16:25:25.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.921+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db00c49a0 con 0x7f7db015a310 2024-03-18T16:25:25.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.922+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db00733c0 con 0x7f7db015a310 2024-03-18T16:25:25.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.942+0000 7f7dad600640 1 -- 172.21.15.102:0/2101336079 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7da4007400 con 0x7f7db015a310 2024-03-18T16:25:25.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.943+0000 7f7dad600640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7d9006bbd0 0x7f7d9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:25.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.943+0000 7f7db4e00640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7d9006bbd0 0x7f7d9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:25.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.943+0000 7f7dad600640 1 -- 172.21.15.102:0/2101336079 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7da4056b10 con 0x7f7db015a310 2024-03-18T16:25:25.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:25.944+0000 7f7db4e00640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7d9006bbd0 0x7f7d9006e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7db00c6c00 tx=0x7f7d9c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:26.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.199+0000 7f7dad600640 1 -- 172.21.15.102:0/2101336079 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7db00733c0 con 0x7f7db015a310 2024-03-18T16:25:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.441+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7db00cba30 con 0x7f7db015a310 2024-03-18T16:25:26.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.464+0000 7f7dad600640 1 -- 172.21.15.102:0/2101336079 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7db00cba30 con 0x7f7db015a310 2024-03-18T16:25:26.467 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:26.467 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:26.467 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.467+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7d9006bbd0 msgr2=0x7f7d9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.467+0000 7f7db6c00640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7d9006bbd0 0x7f7d9006e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7db00c6c00 tx=0x7f7d9c03b040 comp rx=0 tx=0).stop 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.467+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db015a310 msgr2=0x7f7db00c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.467+0000 7f7db6c00640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db015a310 0x7f7db00c7ed0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c3c0 tx=0x7f7da400c880 comp rx=0 tx=0).stop 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.467+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 shutdown_connections 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.467+0000 7f7db6c00640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7d9006bbd0 0x7f7d9006e090 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.467+0000 7f7db6c00640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7db015baa0 0x7f7db014bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.467+0000 7f7db6c00640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7db015a6f0 0x7f7db00c8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.467+0000 7f7db6c00640 1 --2- 172.21.15.102:0/2101336079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7db015a310 0x7f7db00c7ed0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.468+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 >> 172.21.15.102:0/2101336079 conn(0x7f7db008c1a0 msgr2=0x7f7db015d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.468+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 shutdown_connections 2024-03-18T16:25:26.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:26.468+0000 7f7db6c00640 1 -- 172.21.15.102:0/2101336079 wait complete. 2024-03-18T16:25:27.100 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:27.101 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:28.101 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.274+0000 7f0809000640 1 Processor -- start 2024-03-18T16:25:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.274+0000 7f0809000640 1 -- start start 2024-03-18T16:25:28.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.275+0000 7f0809000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f080415a310 0x7f080406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:28.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.275+0000 7f0809000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f080415baa0 0x7f080415bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:28.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.275+0000 7f0809000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f080415c470 con 0x7f080415a6f0 2024-03-18T16:25:28.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.275+0000 7f0809000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f08040cd410 con 0x7f080415a310 2024-03-18T16:25:28.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.275+0000 7f0809000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f08040ca500 con 0x7f080415baa0 2024-03-18T16:25:28.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.276+0000 7f0803e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f080415baa0 0x7f080415bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:28.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.276+0000 7f0803400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f080415a310 0x7f080406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:28.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.276+0000 7f0802a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f080415a6f0 0x7f080415ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39794/0 (socket says 172.21.15.102:39794) 2024-03-18T16:25:28.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.276+0000 7f0802a00640 1 -- 172.21.15.102:0/2915141426 learned_addr learned my addr 172.21.15.102:0/2915141426 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:25:28.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.279+0000 7f0803e00640 1 -- 172.21.15.102:0/2915141426 >> v2:172.21.15.102:3300/0 conn(0x7f080415a310 msgr2=0x7f080406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:28.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.279+0000 7f0803e00640 1 --2- 172.21.15.102:0/2915141426 >> v2:172.21.15.102:3300/0 conn(0x7f080415a310 0x7f080406d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.279+0000 7f0803e00640 1 -- 172.21.15.102:0/2915141426 >> v1:172.21.15.102:6789/0 conn(0x7f080415a6f0 legacy=0x7f080415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:28.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.279+0000 7f0803e00640 1 -- 172.21.15.102:0/2915141426 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08040ca7a0 con 0x7f080415baa0 2024-03-18T16:25:28.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.283+0000 7f0803e00640 1 --2- 172.21.15.102:0/2915141426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080415baa0 0x7f080415bf30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f080406b540 tx=0x7f07fc02bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1453ed18d2d1644 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:28.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.287+0000 7f0802000640 1 -- 172.21.15.102:0/2915141426 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc039070 con 0x7f080415baa0 2024-03-18T16:25:28.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.287+0000 7f0802000640 1 -- 172.21.15.102:0/2915141426 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07fc02cc70 con 0x7f080415baa0 2024-03-18T16:25:28.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.287+0000 7f0802000640 1 -- 172.21.15.102:0/2915141426 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07fc034660 con 0x7f080415baa0 2024-03-18T16:25:28.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.288+0000 7f0809000640 1 -- 172.21.15.102:0/2915141426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080415baa0 msgr2=0x7f080415bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:28.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.288+0000 7f0809000640 1 --2- 172.21.15.102:0/2915141426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080415baa0 0x7f080415bf30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f080406b540 tx=0x7f07fc02bec0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.288+0000 7f0809000640 1 -- 172.21.15.102:0/2915141426 shutdown_connections 2024-03-18T16:25:28.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.288+0000 7f0809000640 1 --2- 172.21.15.102:0/2915141426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080415baa0 0x7f080415bf30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.288+0000 7f0809000640 1 --2- 172.21.15.102:0/2915141426 >> v2:172.21.15.102:3300/0 conn(0x7f080415a310 0x7f080406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.288+0000 7f0809000640 1 -- 172.21.15.102:0/2915141426 >> 172.21.15.102:0/2915141426 conn(0x7f080408c1a0 msgr2=0x7f0804066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:28.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.288+0000 7f0809000640 1 -- 172.21.15.102:0/2915141426 shutdown_connections 2024-03-18T16:25:28.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.289+0000 7f0809000640 1 -- 172.21.15.102:0/2915141426 wait complete. 2024-03-18T16:25:28.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.291+0000 7f0809000640 1 Processor -- start 2024-03-18T16:25:28.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.291+0000 7f0809000640 1 -- start start 2024-03-18T16:25:28.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.293+0000 7f0809000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f080415a310 0x7f08040c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:28.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.293+0000 7f0809000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f080415a6f0 0x7f08041f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:28.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.293+0000 7f0809000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080415baa0 0x7f08040c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:28.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.293+0000 7f0809000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f08040cd130 con 0x7f080415a310 2024-03-18T16:25:28.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.293+0000 7f0809000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08040cce50 con 0x7f080415baa0 2024-03-18T16:25:28.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.293+0000 7f0809000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f080406d920 con 0x7f080415a6f0 2024-03-18T16:25:28.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.293+0000 7f0802a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f080415a6f0 0x7f08041f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:28.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.294+0000 7f0803400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f080415a310 0x7f08040c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:28.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.294+0000 7f0802a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f080415a6f0 0x7f08041f3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48964/0 (socket says 172.21.15.102:48964) 2024-03-18T16:25:28.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.294+0000 7f0802a00640 1 -- 172.21.15.102:0/2145740134 learned_addr learned my addr 172.21.15.102:0/2145740134 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:28.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.294+0000 7f0803e00640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080415baa0 0x7f08040c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:28.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.297+0000 7f0802a00640 1 -- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080415baa0 msgr2=0x7f08040c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:28.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.297+0000 7f0802a00640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080415baa0 0x7f08040c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.297+0000 7f0802a00640 1 -- 172.21.15.102:0/2145740134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f080415a310 msgr2=0x7f08040c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:28.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.297+0000 7f0802a00640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f080415a310 0x7f08040c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.297+0000 7f0802a00640 1 -- 172.21.15.102:0/2145740134 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08041f4340 con 0x7f080415a6f0 2024-03-18T16:25:28.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.300+0000 7f0802a00640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f080415a6f0 0x7f08041f3e00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f07f000c470 tx=0x7f07f0001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:28.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.304+0000 7f07fb400640 1 -- 172.21.15.102:0/2145740134 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0017070 con 0x7f080415a6f0 2024-03-18T16:25:28.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.304+0000 7f07fb400640 1 -- 172.21.15.102:0/2145740134 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07f0005ce0 con 0x7f080415a6f0 2024-03-18T16:25:28.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.304+0000 7f07fb400640 1 -- 172.21.15.102:0/2145740134 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f00072f0 con 0x7f080415a6f0 2024-03-18T16:25:28.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.304+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08041f12f0 con 0x7f080415a6f0 2024-03-18T16:25:28.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.304+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f08040c72c0 con 0x7f080415a6f0 2024-03-18T16:25:28.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.306+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08040cba30 con 0x7f080415a6f0 2024-03-18T16:25:28.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.333+0000 7f07fb400640 1 -- 172.21.15.102:0/2145740134 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f07f001a070 con 0x7f080415a6f0 2024-03-18T16:25:28.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.334+0000 7f07fb400640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f07e406bc20 0x7f07e406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:28.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.335+0000 7f0803400640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f07e406bc20 0x7f07e406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:28.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.336+0000 7f0803400640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f07e406bc20 0x7f07e406e0e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009b40 tx=0x7f07f4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:28.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.336+0000 7f07fb400640 1 -- 172.21.15.102:0/2145740134 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f07f0056760 con 0x7f080415a6f0 2024-03-18T16:25:28.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.583+0000 7f07fb400640 1 -- 172.21.15.102:0/2145740134 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f08040cba30 con 0x7f080415a6f0 2024-03-18T16:25:28.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.848+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0804074fd0 con 0x7f080415a6f0 2024-03-18T16:25:28.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.862+0000 7f07fb400640 1 -- 172.21.15.102:0/2145740134 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f07f0021020 con 0x7f080415a6f0 2024-03-18T16:25:28.864 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:28.864 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:28.865 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:28.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f07e406bc20 msgr2=0x7f07e406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:28.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f07e406bc20 0x7f07e406e0e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009b40 tx=0x7f07f4009190 comp rx=0 tx=0).stop 2024-03-18T16:25:28.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f080415a6f0 msgr2=0x7f08041f3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f080415a6f0 0x7f08041f3e00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f07f000c470 tx=0x7f07f0001e70 comp rx=0 tx=0).stop 2024-03-18T16:25:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 shutdown_connections 2024-03-18T16:25:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f07e406bc20 0x7f07e406e0e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080415baa0 0x7f08040c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f080415a6f0 0x7f08041f3e00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 --2- 172.21.15.102:0/2145740134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f080415a310 0x7f08040c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:28.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.865+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 >> 172.21.15.102:0/2145740134 conn(0x7f080408c1a0 msgr2=0x7f0804153a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:28.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.866+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 shutdown_connections 2024-03-18T16:25:28.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:28.866+0000 7f0809000640 1 -- 172.21.15.102:0/2145740134 wait complete. 2024-03-18T16:25:29.489 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:29.489 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:30.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:30.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.657+0000 7efe39e00640 1 Processor -- start 2024-03-18T16:25:30.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.657+0000 7efe39e00640 1 -- start start 2024-03-18T16:25:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.658+0000 7efe39e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406b470 0x7efe3415ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.658+0000 7efe39e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe3406de50 0x7efe3406b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.658+0000 7efe39e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efe3406bd90 0x7efe34154150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:30.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.659+0000 7efe39e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7efe340ca0a0 con 0x7efe3406bd90 2024-03-18T16:25:30.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.659+0000 7efe39e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efe340c94b0 con 0x7efe3406de50 2024-03-18T16:25:30.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.659+0000 7efe39e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efe340c9770 con 0x7efe3406b470 2024-03-18T16:25:30.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.659+0000 7efe39400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efe3406bd90 0x7efe34154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:30.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.659+0000 7efe33e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe3406de50 0x7efe3406b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:30.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.660+0000 7efe38a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406b470 0x7efe3415ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:30.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.660+0000 7efe33e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe3406de50 0x7efe3406b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41466/0 (socket says 172.21.15.102:41466) 2024-03-18T16:25:30.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.660+0000 7efe33e00640 1 -- 172.21.15.102:0/1938097332 learned_addr learned my addr 172.21.15.102:0/1938097332 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:30.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.662+0000 7efe33e00640 1 -- 172.21.15.102:0/1938097332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406b470 msgr2=0x7efe3415ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:30.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.662+0000 7efe33e00640 1 --2- 172.21.15.102:0/1938097332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406b470 0x7efe3415ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:30.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.662+0000 7efe33e00640 1 -- 172.21.15.102:0/1938097332 >> v2:172.21.15.102:3300/0 conn(0x7efe3406bd90 msgr2=0x7efe34154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:30.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.662+0000 7efe33e00640 1 --2- 172.21.15.102:0/1938097332 >> v2:172.21.15.102:3300/0 conn(0x7efe3406bd90 0x7efe34154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:30.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.662+0000 7efe33e00640 1 -- 172.21.15.102:0/1938097332 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe340c9a50 con 0x7efe3406de50 2024-03-18T16:25:30.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.664+0000 7efe33e00640 1 --2- 172.21.15.102:0/1938097332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406de50 0x7efe3406b850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efe34068fd0 tx=0x7efe2002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44b73a331fc038d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:30.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.668+0000 7efe33400640 1 -- 172.21.15.102:0/1938097332 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20035020 con 0x7efe3406de50 2024-03-18T16:25:30.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.668+0000 7efe33400640 1 -- 172.21.15.102:0/1938097332 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe20031870 con 0x7efe3406de50 2024-03-18T16:25:30.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.668+0000 7efe33400640 1 -- 172.21.15.102:0/1938097332 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20031c40 con 0x7efe3406de50 2024-03-18T16:25:30.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.668+0000 7efe39e00640 1 -- 172.21.15.102:0/1938097332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406de50 msgr2=0x7efe3406b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:30.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.668+0000 7efe39e00640 1 --2- 172.21.15.102:0/1938097332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406de50 0x7efe3406b850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efe34068fd0 tx=0x7efe2002b910 comp rx=0 tx=0).stop 2024-03-18T16:25:30.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.669+0000 7efe39e00640 1 -- 172.21.15.102:0/1938097332 shutdown_connections 2024-03-18T16:25:30.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.669+0000 7efe39e00640 1 --2- 172.21.15.102:0/1938097332 >> v2:172.21.15.102:3300/0 conn(0x7efe3406bd90 0x7efe34154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:30.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.669+0000 7efe39e00640 1 --2- 172.21.15.102:0/1938097332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406de50 0x7efe3406b850 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:30.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.669+0000 7efe39e00640 1 --2- 172.21.15.102:0/1938097332 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406b470 0x7efe3415ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:30.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.669+0000 7efe39e00640 1 -- 172.21.15.102:0/1938097332 >> 172.21.15.102:0/1938097332 conn(0x7efe340727e0 msgr2=0x7efe34072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:30.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.669+0000 7efe39e00640 1 -- 172.21.15.102:0/1938097332 shutdown_connections 2024-03-18T16:25:30.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.669+0000 7efe39e00640 1 -- 172.21.15.102:0/1938097332 wait complete. 2024-03-18T16:25:30.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.670+0000 7efe39e00640 1 Processor -- start 2024-03-18T16:25:30.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.670+0000 7efe39e00640 1 -- start start 2024-03-18T16:25:30.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.671+0000 7efe39e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406b470 0x7efe3415a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:30.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.671+0000 7efe39e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe3406bd90 0x7efe34156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:30.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.671+0000 7efe39e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406de50 0x7efe34158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:30.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.671+0000 7efe39e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efe34154810 con 0x7efe3406bd90 2024-03-18T16:25:30.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.671+0000 7efe39e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe34154690 con 0x7efe3406b470 2024-03-18T16:25:30.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.671+0000 7efe39e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efe34154990 con 0x7efe3406de50 2024-03-18T16:25:30.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.672+0000 7efe38a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406b470 0x7efe3415a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:30.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.672+0000 7efe33e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe3406bd90 0x7efe34156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:30.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.672+0000 7efe38a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406b470 0x7efe3415a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41480/0 (socket says 172.21.15.102:41480) 2024-03-18T16:25:30.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.672+0000 7efe38a00640 1 -- 172.21.15.102:0/3130091313 learned_addr learned my addr 172.21.15.102:0/3130091313 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:30.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.672+0000 7efe39400640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406de50 0x7efe34158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:30.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.676+0000 7efe38a00640 1 -- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406de50 msgr2=0x7efe34158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:30.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.676+0000 7efe38a00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406de50 0x7efe34158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:30.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.676+0000 7efe38a00640 1 -- 172.21.15.102:0/3130091313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe3406bd90 msgr2=0x7efe34156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:30.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.676+0000 7efe38a00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe3406bd90 0x7efe34156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:30.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.676+0000 7efe38a00640 1 -- 172.21.15.102:0/3130091313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3415ab40 con 0x7efe3406b470 2024-03-18T16:25:30.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.678+0000 7efe38a00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406b470 0x7efe3415a550 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efe2800e870 tx=0x7efe2800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.681+0000 7efe30c00640 1 -- 172.21.15.102:0/3130091313 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe28017070 con 0x7efe3406b470 2024-03-18T16:25:30.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.682+0000 7efe30c00640 1 -- 172.21.15.102:0/3130091313 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe2800ee60 con 0x7efe3406b470 2024-03-18T16:25:30.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.682+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe3415ade0 con 0x7efe3406b470 2024-03-18T16:25:30.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.682+0000 7efe30c00640 1 -- 172.21.15.102:0/3130091313 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe2800cb80 con 0x7efe3406b470 2024-03-18T16:25:30.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.682+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe341f0010 con 0x7efe3406b470 2024-03-18T16:25:30.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.683+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe340cd710 con 0x7efe3406b470 2024-03-18T16:25:30.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.945+0000 7efe30c00640 1 -- 172.21.15.102:0/3130091313 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efe2801a070 con 0x7efe3406b470 2024-03-18T16:25:30.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.946+0000 7efe30c00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe1406bc50 0x7efe1406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:30.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.947+0000 7efe33e00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe1406bc50 0x7efe1406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:30.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.948+0000 7efe33e00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe1406bc50 0x7efe1406e110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efe20036000 tx=0x7efe2003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:30.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.949+0000 7efe30c00640 1 -- 172.21.15.102:0/3130091313 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7efe280574f0 con 0x7efe3406b470 2024-03-18T16:25:30.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:30.960+0000 7efe30c00640 1 -- 172.21.15.102:0/3130091313 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efe340cd710 con 0x7efe3406b470 2024-03-18T16:25:31.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.230+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe34074fd0 con 0x7efe3406b470 2024-03-18T16:25:31.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.273+0000 7efe30c00640 1 -- 172.21.15.102:0/3130091313 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7efe28059450 con 0x7efe3406b470 2024-03-18T16:25:31.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:31.276 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:31.276 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe1406bc50 msgr2=0x7efe1406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe1406bc50 0x7efe1406e110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efe20036000 tx=0x7efe2003c040 comp rx=0 tx=0).stop 2024-03-18T16:25:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406b470 msgr2=0x7efe3415a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406b470 0x7efe3415a550 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7efe2800e870 tx=0x7efe2800ed30 comp rx=0 tx=0).stop 2024-03-18T16:25:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 shutdown_connections 2024-03-18T16:25:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe1406bc50 0x7efe1406e110 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe3406de50 0x7efe34158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:31.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe3406bd90 0x7efe34156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 --2- 172.21.15.102:0/3130091313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe3406b470 0x7efe3415a550 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 >> 172.21.15.102:0/3130091313 conn(0x7efe340727e0 msgr2=0x7efe340cb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 shutdown_connections 2024-03-18T16:25:31.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:31.276+0000 7efe39e00640 1 -- 172.21.15.102:0/3130091313 wait complete. 2024-03-18T16:25:31.898 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:31.898 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:32.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:33.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.076+0000 7f252dc00640 1 Processor -- start 2024-03-18T16:25:33.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.077+0000 7f252dc00640 1 -- start start 2024-03-18T16:25:33.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.078+0000 7f252dc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f25280d0250 0x7f252815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:33.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.078+0000 7f252dc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25280cabd0 con 0x7f25280cfe70 2024-03-18T16:25:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.078+0000 7f252dc00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25280caea0 con 0x7f25280cc2f0 2024-03-18T16:25:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.078+0000 7f252dc00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f25280cb180 con 0x7f25280d0250 2024-03-18T16:25:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.079+0000 7f2527e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f25280cfe70 0x7f25280c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52302/0 (socket says 172.21.15.102:52302) 2024-03-18T16:25:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.079+0000 7f2527e00640 1 -- 172.21.15.102:0/585925851 learned_addr learned my addr 172.21.15.102:0/585925851 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:25:33.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.079+0000 7f2527400640 1 --2- 172.21.15.102:0/585925851 >> v2:172.21.15.154:3300/0 conn(0x7f25280d0250 0x7f252815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:33.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.081+0000 7f2527400640 1 -- 172.21.15.102:0/585925851 >> v1:172.21.15.154:6789/0 conn(0x7f25280cc2f0 legacy=0x7f25280cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:33.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.081+0000 7f2527400640 1 -- 172.21.15.102:0/585925851 >> v1:172.21.15.102:6789/0 conn(0x7f25280cfe70 legacy=0x7f25280c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:33.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.081+0000 7f2527400640 1 -- 172.21.15.102:0/585925851 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25280cb460 con 0x7f25280d0250 2024-03-18T16:25:33.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.084+0000 7f2527400640 1 --2- 172.21.15.102:0/585925851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280d0250 0x7f252815ae40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f25140097a0 tx=0x7f251402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=172ef580bb3453b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:33.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.087+0000 7f2526a00640 1 -- 172.21.15.102:0/585925851 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514034070 con 0x7f25280d0250 2024-03-18T16:25:33.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.087+0000 7f2526a00640 1 -- 172.21.15.102:0/585925851 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2514030610 con 0x7f25280d0250 2024-03-18T16:25:33.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.087+0000 7f2526a00640 1 -- 172.21.15.102:0/585925851 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25140309e0 con 0x7f25280d0250 2024-03-18T16:25:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.088+0000 7f252dc00640 1 -- 172.21.15.102:0/585925851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280d0250 msgr2=0x7f252815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.088+0000 7f252dc00640 1 --2- 172.21.15.102:0/585925851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280d0250 0x7f252815ae40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f25140097a0 tx=0x7f251402b960 comp rx=0 tx=0).stop 2024-03-18T16:25:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.088+0000 7f252dc00640 1 -- 172.21.15.102:0/585925851 shutdown_connections 2024-03-18T16:25:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.088+0000 7f252dc00640 1 --2- 172.21.15.102:0/585925851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280d0250 0x7f252815ae40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.088+0000 7f252dc00640 1 -- 172.21.15.102:0/585925851 >> 172.21.15.102:0/585925851 conn(0x7f252808c3e0 msgr2=0x7f252806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.088+0000 7f252dc00640 1 -- 172.21.15.102:0/585925851 shutdown_connections 2024-03-18T16:25:33.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.088+0000 7f252dc00640 1 -- 172.21.15.102:0/585925851 wait complete. 2024-03-18T16:25:33.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.090+0000 7f252dc00640 1 Processor -- start 2024-03-18T16:25:33.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.090+0000 7f252dc00640 1 -- start start 2024-03-18T16:25:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.091+0000 7f252dc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25280cc2f0 0x7f25281e22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.091+0000 7f252dc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280cfe70 0x7f25281f0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.091+0000 7f252dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25280d0250 0x7f25281e0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.091+0000 7f252dc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f25280cbd00 con 0x7f25280d0250 2024-03-18T16:25:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.091+0000 7f252dc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25280cba20 con 0x7f25280cfe70 2024-03-18T16:25:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.091+0000 7f252dc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f25280c8330 con 0x7f25280cc2f0 2024-03-18T16:25:33.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.091+0000 7f2527400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280cfe70 0x7f25281f0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.092+0000 7f2527400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280cfe70 0x7f25281f0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:55064/0 (socket says 172.21.15.102:55064) 2024-03-18T16:25:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.092+0000 7f2527400640 1 -- 172.21.15.102:0/1297906578 learned_addr learned my addr 172.21.15.102:0/1297906578 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.092+0000 7f2527e00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25280cc2f0 0x7f25281e22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:33.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.092+0000 7f252d200640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25280d0250 0x7f25281e0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.095+0000 7f2527400640 1 -- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25280cc2f0 msgr2=0x7f25281e22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.095+0000 7f2527400640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25280cc2f0 0x7f25281e22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.095+0000 7f2527400640 1 -- 172.21.15.102:0/1297906578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25280d0250 msgr2=0x7f25281e0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.095+0000 7f2527400640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25280d0250 0x7f25281e0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:33.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.095+0000 7f2527400640 1 -- 172.21.15.102:0/1297906578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2528157f00 con 0x7f25280cfe70 2024-03-18T16:25:33.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.097+0000 7f2527400640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280cfe70 0x7f25281f0570 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f25140098c0 tx=0x7f25140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.101+0000 7f251fe00640 1 -- 172.21.15.102:0/1297906578 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514034040 con 0x7f25280cfe70 2024-03-18T16:25:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.101+0000 7f251fe00640 1 -- 172.21.15.102:0/1297906578 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251403acc0 con 0x7f25280cfe70 2024-03-18T16:25:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.101+0000 7f251fe00640 1 -- 172.21.15.102:0/1297906578 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514044630 con 0x7f25280cfe70 2024-03-18T16:25:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.101+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25281581e0 con 0x7f25280cfe70 2024-03-18T16:25:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.101+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25281587a0 con 0x7f25280cfe70 2024-03-18T16:25:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.103+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2528074fd0 con 0x7f25280cfe70 2024-03-18T16:25:33.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.124+0000 7f251fe00640 1 -- 172.21.15.102:0/1297906578 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f251404b060 con 0x7f25280cfe70 2024-03-18T16:25:33.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.124+0000 7f251fe00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f250806bc00 0x7f250806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:33.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.125+0000 7f2527e00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f250806bc00 0x7f250806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:33.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.126+0000 7f2527e00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f250806bc00 0x7f250806e0c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2518007fd0 tx=0x7f2518007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:33.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.367+0000 7f251fe00640 1 -- 172.21.15.102:0/1297906578 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f25140305a0 con 0x7f25280cfe70 2024-03-18T16:25:34.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.378+0000 7f251fe00640 1 -- 172.21.15.102:0/1297906578 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f251403f070 con 0x7f25280cfe70 2024-03-18T16:25:34.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.729+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25280c7e50 con 0x7f25280cfe70 2024-03-18T16:25:34.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.743+0000 7f251fe00640 1 -- 172.21.15.102:0/1297906578 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f251407b260 con 0x7f25280cfe70 2024-03-18T16:25:34.358 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:34.358 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:34.360 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:34.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f250806bc00 msgr2=0x7f250806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:34.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f250806bc00 0x7f250806e0c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2518007fd0 tx=0x7f2518007e80 comp rx=0 tx=0).stop 2024-03-18T16:25:34.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280cfe70 msgr2=0x7f25281f0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:34.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280cfe70 0x7f25281f0570 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f25140098c0 tx=0x7f25140046c0 comp rx=0 tx=0).stop 2024-03-18T16:25:34.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 shutdown_connections 2024-03-18T16:25:34.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f250806bc00 0x7f250806e0c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:34.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25280d0250 0x7f25281e0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:34.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25280cfe70 0x7f25281f0570 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:34.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 --2- 172.21.15.102:0/1297906578 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25280cc2f0 0x7f25281e22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:34.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.745+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 >> 172.21.15.102:0/1297906578 conn(0x7f252808c3e0 msgr2=0x7f252806c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:34.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.746+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 shutdown_connections 2024-03-18T16:25:34.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:33.746+0000 7f252dc00640 1 -- 172.21.15.102:0/1297906578 wait complete. 2024-03-18T16:25:34.966 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:34.967 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:35.967 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:36.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.138+0000 7f10ef000640 1 Processor -- start 2024-03-18T16:25:36.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.138+0000 7f10ef000640 1 -- start start 2024-03-18T16:25:36.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.139+0000 7f10ef000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f10e8141650 0x7f10e8158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.140+0000 7f10ef000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e80c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.140+0000 7f10ef000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e80c72d0 con 0x7f10e815c3a0 2024-03-18T16:25:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.140+0000 7f10ef000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f10e8159370 con 0x7f10e8141650 2024-03-18T16:25:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.140+0000 7f10ef000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f10e8158ad0 con 0x7f10e80c4fa0 2024-03-18T16:25:36.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.140+0000 7f10ee600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e80c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:36.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.141+0000 7f10ee600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e80c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50282/0 (socket says 172.21.15.102:50282) 2024-03-18T16:25:36.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.141+0000 7f10ee600640 1 -- 172.21.15.102:0/3102296795 learned_addr learned my addr 172.21.15.102:0/3102296795 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:36.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.141+0000 7f10edc00640 1 --2- 172.21.15.102:0/3102296795 >> v2:172.21.15.102:3300/0 conn(0x7f10e8141650 0x7f10e8158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:36.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.143+0000 7f10ee600640 1 -- 172.21.15.102:0/3102296795 >> v2:172.21.15.102:3300/0 conn(0x7f10e8141650 msgr2=0x7f10e8158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:36.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.143+0000 7f10ee600640 1 --2- 172.21.15.102:0/3102296795 >> v2:172.21.15.102:3300/0 conn(0x7f10e8141650 0x7f10e8158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:36.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.143+0000 7f10ee600640 1 -- 172.21.15.102:0/3102296795 >> v1:172.21.15.102:6789/0 conn(0x7f10e815c3a0 legacy=0x7f10e80c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:36.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.143+0000 7f10ee600640 1 -- 172.21.15.102:0/3102296795 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e8158db0 con 0x7f10e80c4fa0 2024-03-18T16:25:36.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.146+0000 7f10ee600640 1 --2- 172.21.15.102:0/3102296795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e80c5430 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009e50 tx=0x7f10e002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8856e03d5dde6006 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:36.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.150+0000 7f10e7e00640 1 -- 172.21.15.102:0/3102296795 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e002cb00 con 0x7f10e80c4fa0 2024-03-18T16:25:36.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.151+0000 7f10e7e00640 1 -- 172.21.15.102:0/3102296795 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e002cca0 con 0x7f10e80c4fa0 2024-03-18T16:25:36.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.151+0000 7f10e7e00640 1 -- 172.21.15.102:0/3102296795 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0034610 con 0x7f10e80c4fa0 2024-03-18T16:25:36.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.151+0000 7f10ef000640 1 -- 172.21.15.102:0/3102296795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 msgr2=0x7f10e80c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:36.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.151+0000 7f10ef000640 1 --2- 172.21.15.102:0/3102296795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e80c5430 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f10e0009e50 tx=0x7f10e002bf60 comp rx=0 tx=0).stop 2024-03-18T16:25:36.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.152+0000 7f10ef000640 1 -- 172.21.15.102:0/3102296795 shutdown_connections 2024-03-18T16:25:36.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.152+0000 7f10ef000640 1 --2- 172.21.15.102:0/3102296795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e80c5430 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:36.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.152+0000 7f10ef000640 1 --2- 172.21.15.102:0/3102296795 >> v2:172.21.15.102:3300/0 conn(0x7f10e8141650 0x7f10e8158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:36.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.152+0000 7f10ef000640 1 -- 172.21.15.102:0/3102296795 >> 172.21.15.102:0/3102296795 conn(0x7f10e80cb890 msgr2=0x7f10e80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:36.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.152+0000 7f10ef000640 1 -- 172.21.15.102:0/3102296795 shutdown_connections 2024-03-18T16:25:36.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.152+0000 7f10ef000640 1 -- 172.21.15.102:0/3102296795 wait complete. 2024-03-18T16:25:36.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.154+0000 7f10ef000640 1 Processor -- start 2024-03-18T16:25:36.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.155+0000 7f10ef000640 1 -- start start 2024-03-18T16:25:36.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.156+0000 7f10ef000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e81e2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:36.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.156+0000 7f10ef000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e8141650 0x7f10e8071e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:36.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.156+0000 7f10ef000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10e815c3a0 0x7f10e80723c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:36.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.156+0000 7f10ef000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80c66a0 con 0x7f10e815c3a0 2024-03-18T16:25:36.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.156+0000 7f10ef000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f10e80c63c0 con 0x7f10e8141650 2024-03-18T16:25:36.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.156+0000 7f10ef000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f10e815c7c0 con 0x7f10e80c4fa0 2024-03-18T16:25:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.157+0000 7f10edc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e81e2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.157+0000 7f10ed200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e8141650 0x7f10e8071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.157+0000 7f10ee600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10e815c3a0 0x7f10e80723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.157+0000 7f10edc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e81e2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50286/0 (socket says 172.21.15.102:50286) 2024-03-18T16:25:36.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.157+0000 7f10edc00640 1 -- 172.21.15.102:0/2236296469 learned_addr learned my addr 172.21.15.102:0/2236296469 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:36.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.161+0000 7f10ed200640 1 -- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 msgr2=0x7f10e81e2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:36.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.161+0000 7f10ed200640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e81e2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:36.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.161+0000 7f10ed200640 1 -- 172.21.15.102:0/2236296469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10e815c3a0 msgr2=0x7f10e80723c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:36.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.161+0000 7f10ed200640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10e815c3a0 0x7f10e80723c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:36.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.161+0000 7f10ed200640 1 -- 172.21.15.102:0/2236296469 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e80c2160 con 0x7f10e8141650 2024-03-18T16:25:36.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.163+0000 7f10ed200640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e8141650 0x7f10e8071e80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f10d400c370 tx=0x7f10d400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:36.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.167+0000 7f10e5600640 1 -- 172.21.15.102:0/2236296469 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d40052d0 con 0x7f10e8141650 2024-03-18T16:25:36.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.167+0000 7f10e5600640 1 -- 172.21.15.102:0/2236296469 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10d4005490 con 0x7f10e8141650 2024-03-18T16:25:36.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.167+0000 7f10e5600640 1 -- 172.21.15.102:0/2236296469 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d4007260 con 0x7f10e8141650 2024-03-18T16:25:36.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.167+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e80c2420 con 0x7f10e8141650 2024-03-18T16:25:36.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.167+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e80c3070 con 0x7f10e8141650 2024-03-18T16:25:36.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.169+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e80c74a0 con 0x7f10e8141650 2024-03-18T16:25:36.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.190+0000 7f10e5600640 1 -- 172.21.15.102:0/2236296469 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f10d4007400 con 0x7f10e8141650 2024-03-18T16:25:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.191+0000 7f10e5600640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f10c806bc20 0x7f10c806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.191+0000 7f10edc00640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f10c806bc20 0x7f10c806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.192+0000 7f10e5600640 1 -- 172.21.15.102:0/2236296469 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f10d40568a0 con 0x7f10e8141650 2024-03-18T16:25:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.192+0000 7f10edc00640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f10c806bc20 0x7f10c806e0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f10dc009b60 tx=0x7f10dc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.441+0000 7f10e5600640 1 -- 172.21.15.102:0/2236296469 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f10e80c74a0 con 0x7f10e8141650 2024-03-18T16:25:36.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.680+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10e8074fd0 con 0x7f10e8141650 2024-03-18T16:25:36.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.693+0000 7f10e5600640 1 -- 172.21.15.102:0/2236296469 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f10d4058800 con 0x7f10e8141650 2024-03-18T16:25:36.695 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:36.696 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f10c806bc20 msgr2=0x7f10c806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f10c806bc20 0x7f10c806e0e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f10dc009b60 tx=0x7f10dc009190 comp rx=0 tx=0).stop 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e8141650 msgr2=0x7f10e8071e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e8141650 0x7f10e8071e80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f10d400c370 tx=0x7f10d400c830 comp rx=0 tx=0).stop 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 shutdown_connections 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f10c806bc20 0x7f10c806e0e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10e815c3a0 0x7f10e80723c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e8141650 0x7f10e8071e80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 --2- 172.21.15.102:0/2236296469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f10e80c4fa0 0x7f10e81e2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.696+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 >> 172.21.15.102:0/2236296469 conn(0x7f10e80cb890 msgr2=0x7f10e80c8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.697+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 shutdown_connections 2024-03-18T16:25:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:36.697+0000 7f10ef000640 1 -- 172.21.15.102:0/2236296469 wait complete. 2024-03-18T16:25:37.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:37.321 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:38.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.496+0000 7f1135600640 1 Processor -- start 2024-03-18T16:25:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.496+0000 7f1135600640 1 -- start start 2024-03-18T16:25:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.497+0000 7f1135600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015a310 0x7f113006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.497+0000 7f1135600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f113015a6f0 0x7f113015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.497+0000 7f1135600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f113015baa0 0x7f113015bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.497+0000 7f1135600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f11300cd410 con 0x7f113015a6f0 2024-03-18T16:25:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.497+0000 7f1135600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f113015c470 con 0x7f113015baa0 2024-03-18T16:25:38.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.497+0000 7f1135600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f11300ca500 con 0x7f113015a310 2024-03-18T16:25:38.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.498+0000 7f1134c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f113015baa0 0x7f113015bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:38.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.498+0000 7f1134c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f113015baa0 0x7f113015bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:55082/0 (socket says 172.21.15.102:55082) 2024-03-18T16:25:38.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.498+0000 7f1134c00640 1 -- 172.21.15.102:0/4025866216 learned_addr learned my addr 172.21.15.102:0/4025866216 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:38.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.498+0000 7f112f400640 1 --2- 172.21.15.102:0/4025866216 >> v2:172.21.15.102:3300/0 conn(0x7f113015a6f0 0x7f113015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:38.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.499+0000 7f112fe00640 1 --2- 172.21.15.102:0/4025866216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015a310 0x7f113006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.501+0000 7f1134c00640 1 -- 172.21.15.102:0/4025866216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015a310 msgr2=0x7f113006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.501+0000 7f1134c00640 1 --2- 172.21.15.102:0/4025866216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015a310 0x7f113006d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.501+0000 7f1134c00640 1 -- 172.21.15.102:0/4025866216 >> v2:172.21.15.102:3300/0 conn(0x7f113015a6f0 msgr2=0x7f113015ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.501+0000 7f1134c00640 1 --2- 172.21.15.102:0/4025866216 >> v2:172.21.15.102:3300/0 conn(0x7f113015a6f0 0x7f113015ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.501+0000 7f1134c00640 1 -- 172.21.15.102:0/4025866216 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113015b050 con 0x7f113015baa0 2024-03-18T16:25:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.507+0000 7f1134c00640 1 --2- 172.21.15.102:0/4025866216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015baa0 0x7f113015bf30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f11300d0790 tx=0x7f112802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cee53ebf9d638691 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.511+0000 7f112ea00640 1 -- 172.21.15.102:0/4025866216 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112802ca80 con 0x7f113015baa0 2024-03-18T16:25:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.511+0000 7f112ea00640 1 -- 172.21.15.102:0/4025866216 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f112802cc20 con 0x7f113015baa0 2024-03-18T16:25:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.511+0000 7f112ea00640 1 -- 172.21.15.102:0/4025866216 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11280343f0 con 0x7f113015baa0 2024-03-18T16:25:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.512+0000 7f1135600640 1 -- 172.21.15.102:0/4025866216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015baa0 msgr2=0x7f113015bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.512+0000 7f1135600640 1 --2- 172.21.15.102:0/4025866216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015baa0 0x7f113015bf30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f11300d0790 tx=0x7f112802bee0 comp rx=0 tx=0).stop 2024-03-18T16:25:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.512+0000 7f1135600640 1 -- 172.21.15.102:0/4025866216 shutdown_connections 2024-03-18T16:25:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.512+0000 7f1135600640 1 --2- 172.21.15.102:0/4025866216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015baa0 0x7f113015bf30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.512+0000 7f1135600640 1 --2- 172.21.15.102:0/4025866216 >> v2:172.21.15.102:3300/0 conn(0x7f113015a6f0 0x7f113015ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.512+0000 7f1135600640 1 --2- 172.21.15.102:0/4025866216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015a310 0x7f113006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.512+0000 7f1135600640 1 -- 172.21.15.102:0/4025866216 >> 172.21.15.102:0/4025866216 conn(0x7f113008c1a0 msgr2=0x7f1130066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.512+0000 7f1135600640 1 -- 172.21.15.102:0/4025866216 shutdown_connections 2024-03-18T16:25:38.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.512+0000 7f1135600640 1 -- 172.21.15.102:0/4025866216 wait complete. 2024-03-18T16:25:38.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.515+0000 7f1135600640 1 Processor -- start 2024-03-18T16:25:38.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.515+0000 7f1135600640 1 -- start start 2024-03-18T16:25:38.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.516+0000 7f1135600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015a310 0x7f11300ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:38.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.516+0000 7f1135600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f113015a6f0 0x7f11301e6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:38.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.516+0000 7f1135600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015baa0 0x7f11301e71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:38.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.516+0000 7f1135600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f11300cce50 con 0x7f113015a6f0 2024-03-18T16:25:38.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.516+0000 7f1135600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11300caa80 con 0x7f113015a310 2024-03-18T16:25:38.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.516+0000 7f1135600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f11300cd130 con 0x7f113015baa0 2024-03-18T16:25:38.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.517+0000 7f112fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015a310 0x7f11300ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:38.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.517+0000 7f1134c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015baa0 0x7f11301e71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:38.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.517+0000 7f112f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f113015a6f0 0x7f11301e6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:38.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.517+0000 7f112fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015a310 0x7f11300ce110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:55094/0 (socket says 172.21.15.102:55094) 2024-03-18T16:25:38.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.517+0000 7f112fe00640 1 -- 172.21.15.102:0/2646468542 learned_addr learned my addr 172.21.15.102:0/2646468542 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:38.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.521+0000 7f112f400640 1 -- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015baa0 msgr2=0x7f11301e71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:38.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.521+0000 7f112f400640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015baa0 0x7f11301e71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:38.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.521+0000 7f112f400640 1 -- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015a310 msgr2=0x7f11300ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:38.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.521+0000 7f112f400640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015a310 0x7f11300ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:38.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.521+0000 7f112f400640 1 -- 172.21.15.102:0/2646468542 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11301f0120 con 0x7f113015a6f0 2024-03-18T16:25:38.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.524+0000 7f112f400640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f113015a6f0 0x7f11301e6c90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f111c00d520 tx=0x7f111c00d9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:38.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.528+0000 7f1127e00640 1 -- 172.21.15.102:0/2646468542 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c0125f0 con 0x7f113015a6f0 2024-03-18T16:25:38.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.529+0000 7f1127e00640 1 -- 172.21.15.102:0/2646468542 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f111c019070 con 0x7f113015a6f0 2024-03-18T16:25:38.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.529+0000 7f1127e00640 1 -- 172.21.15.102:0/2646468542 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c0129b0 con 0x7f113015a6f0 2024-03-18T16:25:38.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.529+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11301f4140 con 0x7f113015a6f0 2024-03-18T16:25:38.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.529+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11301f03e0 con 0x7f113015a6f0 2024-03-18T16:25:38.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.530+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1130074fd0 con 0x7f113015a6f0 2024-03-18T16:25:38.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.558+0000 7f1127e00640 1 -- 172.21.15.102:0/2646468542 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f111c012b50 con 0x7f113015a6f0 2024-03-18T16:25:38.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.559+0000 7f1127e00640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f111006bc20 0x7f111006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:38.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.560+0000 7f112fe00640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f111006bc20 0x7f111006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:38.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.561+0000 7f112fe00640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f111006bc20 0x7f111006e0e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1120009ae0 tx=0x7f1120009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:38.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.563+0000 7f1127e00640 1 -- 172.21.15.102:0/2646468542 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f111c012e40 con 0x7f113015a6f0 2024-03-18T16:25:38.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:38.836+0000 7f1127e00640 1 -- 172.21.15.102:0/2646468542 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f111c059bc0 con 0x7f113015a6f0 2024-03-18T16:25:39.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.089+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11300cbd10 con 0x7f113015a6f0 2024-03-18T16:25:39.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.112+0000 7f1127e00640 1 -- 172.21.15.102:0/2646468542 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f111c05dc60 con 0x7f113015a6f0 2024-03-18T16:25:39.115 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:39.115 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:39.116 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f111006bc20 msgr2=0x7f111006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f111006bc20 0x7f111006e0e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1120009ae0 tx=0x7f1120009190 comp rx=0 tx=0).stop 2024-03-18T16:25:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f113015a6f0 msgr2=0x7f11301e6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f113015a6f0 0x7f11301e6c90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f111c00d520 tx=0x7f111c00d9e0 comp rx=0 tx=0).stop 2024-03-18T16:25:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 shutdown_connections 2024-03-18T16:25:39.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f111006bc20 0x7f111006e0e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:39.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f113015baa0 0x7f11301e71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:39.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f113015a6f0 0x7f11301e6c90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:39.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 --2- 172.21.15.102:0/2646468542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f113015a310 0x7f11300ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:39.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.115+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 >> 172.21.15.102:0/2646468542 conn(0x7f113008c1a0 msgr2=0x7f113014eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:39.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.116+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 shutdown_connections 2024-03-18T16:25:39.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:39.116+0000 7f1135600640 1 -- 172.21.15.102:0/2646468542 wait complete. 2024-03-18T16:25:39.739 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:39.740 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:40.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:40.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.919+0000 7f1ea9e00640 1 Processor -- start 2024-03-18T16:25:40.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.920+0000 7f1ea9e00640 1 -- start start 2024-03-18T16:25:40.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.921+0000 7f1ea9e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1ea40c8590 0x7f1ea40ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:40.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.921+0000 7f1ea9e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea40ce610 0x7f1ea40c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:40.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.921+0000 7f1ea9e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea415b2f0 con 0x7f1ea4150fb0 2024-03-18T16:25:40.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.921+0000 7f1ea9e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea40c89e0 con 0x7f1ea40c8590 2024-03-18T16:25:40.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.921+0000 7f1ea9e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea415e830 con 0x7f1ea40ce610 2024-03-18T16:25:40.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.921+0000 7f1ea3400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea40ce610 0x7f1ea40c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:40.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.922+0000 7f1ea2000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1ea40c8590 0x7f1ea40ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:40.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.922+0000 7f1ea3400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea40ce610 0x7f1ea40c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:32780/0 (socket says 172.21.15.102:32780) 2024-03-18T16:25:40.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.922+0000 7f1ea3400640 1 -- 172.21.15.102:0/1336240417 learned_addr learned my addr 172.21.15.102:0/1336240417 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.925+0000 7f1ea3400640 1 -- 172.21.15.102:0/1336240417 >> v2:172.21.15.102:3300/0 conn(0x7f1ea40c8590 msgr2=0x7f1ea40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.925+0000 7f1ea3400640 1 --2- 172.21.15.102:0/1336240417 >> v2:172.21.15.102:3300/0 conn(0x7f1ea40c8590 0x7f1ea40ce0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.925+0000 7f1ea3400640 1 -- 172.21.15.102:0/1336240417 >> v1:172.21.15.154:6789/0 conn(0x7f1ea4150fb0 legacy=0x7f1ea40708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.925+0000 7f1ea3400640 1 -- 172.21.15.102:0/1336240417 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea415d9b0 con 0x7f1ea40ce610 2024-03-18T16:25:40.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.928+0000 7f1ea3400640 1 --2- 172.21.15.102:0/1336240417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea40ce610 0x7f1ea40c96b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009e50 tx=0x7f1e9c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99a7c2a3431a56eb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:40.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.931+0000 7f1ea1600640 1 -- 172.21.15.102:0/1336240417 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c02cb90 con 0x7f1ea40ce610 2024-03-18T16:25:40.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.932+0000 7f1ea1600640 1 -- 172.21.15.102:0/1336240417 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e9c02cd30 con 0x7f1ea40ce610 2024-03-18T16:25:40.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.932+0000 7f1ea1600640 1 -- 172.21.15.102:0/1336240417 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c034610 con 0x7f1ea40ce610 2024-03-18T16:25:40.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.932+0000 7f1ea9e00640 1 -- 172.21.15.102:0/1336240417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea40ce610 msgr2=0x7f1ea40c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:40.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.932+0000 7f1ea9e00640 1 --2- 172.21.15.102:0/1336240417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea40ce610 0x7f1ea40c96b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009e50 tx=0x7f1e9c02bff0 comp rx=0 tx=0).stop 2024-03-18T16:25:40.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.932+0000 7f1ea9e00640 1 -- 172.21.15.102:0/1336240417 shutdown_connections 2024-03-18T16:25:40.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.932+0000 7f1ea9e00640 1 --2- 172.21.15.102:0/1336240417 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea40ce610 0x7f1ea40c96b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:40.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.932+0000 7f1ea9e00640 1 --2- 172.21.15.102:0/1336240417 >> v2:172.21.15.102:3300/0 conn(0x7f1ea40c8590 0x7f1ea40ce0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:40.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.932+0000 7f1ea9e00640 1 -- 172.21.15.102:0/1336240417 >> 172.21.15.102:0/1336240417 conn(0x7f1ea4067370 msgr2=0x7f1ea4067780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:40.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.933+0000 7f1ea9e00640 1 -- 172.21.15.102:0/1336240417 shutdown_connections 2024-03-18T16:25:40.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.933+0000 7f1ea9e00640 1 -- 172.21.15.102:0/1336240417 wait complete. 2024-03-18T16:25:40.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.934+0000 7f1ea9e00640 1 Processor -- start 2024-03-18T16:25:40.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.935+0000 7f1ea9e00640 1 -- start start 2024-03-18T16:25:40.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.935+0000 7f1ea9e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ea40c8590 0x7f1ea414ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:40.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.935+0000 7f1ea9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea40ce610 0x7f1ea40ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:40.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.936+0000 7f1ea9e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea4150fb0 0x7f1ea414d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:40.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.936+0000 7f1ea9e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea415e1e0 con 0x7f1ea40ce610 2024-03-18T16:25:40.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.936+0000 7f1ea9e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea415df00 con 0x7f1ea40c8590 2024-03-18T16:25:40.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.936+0000 7f1ea9e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea415e4c0 con 0x7f1ea4150fb0 2024-03-18T16:25:40.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.936+0000 7f1ea3400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea4150fb0 0x7f1ea414d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:40.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.936+0000 7f1ea2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ea40c8590 0x7f1ea414ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:40.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.936+0000 7f1ea2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea40ce610 0x7f1ea40ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:40.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.937+0000 7f1ea3400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea4150fb0 0x7f1ea414d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:32790/0 (socket says 172.21.15.102:32790) 2024-03-18T16:25:40.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.937+0000 7f1ea3400640 1 -- 172.21.15.102:0/657344562 learned_addr learned my addr 172.21.15.102:0/657344562 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:40.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.942+0000 7f1ea2a00640 1 -- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea4150fb0 msgr2=0x7f1ea414d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:40.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.942+0000 7f1ea2a00640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea4150fb0 0x7f1ea414d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:40.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.942+0000 7f1ea2a00640 1 -- 172.21.15.102:0/657344562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea40ce610 msgr2=0x7f1ea40ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:40.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.942+0000 7f1ea2a00640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea40ce610 0x7f1ea40ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:40.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.942+0000 7f1ea2a00640 1 -- 172.21.15.102:0/657344562 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea40c5db0 con 0x7f1ea40c8590 2024-03-18T16:25:40.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.942+0000 7f1ea3400640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea4150fb0 0x7f1ea414d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:25:40.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.945+0000 7f1ea2a00640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ea40c8590 0x7f1ea414ed20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400c3c0 tx=0x7f1e9400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:40.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.947+0000 7f1e9b400640 1 -- 172.21.15.102:0/657344562 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e940052d0 con 0x7f1ea40c8590 2024-03-18T16:25:40.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.947+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea415b840 con 0x7f1ea40c8590 2024-03-18T16:25:40.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.947+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea40c52d0 con 0x7f1ea40c8590 2024-03-18T16:25:40.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.948+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea4074fd0 con 0x7f1ea40c8590 2024-03-18T16:25:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.949+0000 7f1e9b400640 1 -- 172.21.15.102:0/657344562 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e94005470 con 0x7f1ea40c8590 2024-03-18T16:25:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.949+0000 7f1e9b400640 1 -- 172.21.15.102:0/657344562 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94020060 con 0x7f1ea40c8590 2024-03-18T16:25:40.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.970+0000 7f1e9b400640 1 -- 172.21.15.102:0/657344562 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1e94007220 con 0x7f1ea40c8590 2024-03-18T16:25:40.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.971+0000 7f1e9b400640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e8406bbd0 0x7f1e8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:40.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.971+0000 7f1ea2000640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e8406bbd0 0x7f1e8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:40.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:40.972+0000 7f1ea2000640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e8406bbd0 0x7f1e8406e090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1ea414ecb0 tx=0x7f1e90009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:41.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.223+0000 7f1e9b400640 1 -- 172.21.15.102:0/657344562 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1e940567b0 con 0x7f1ea40c8590 2024-03-18T16:25:41.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.235+0000 7f1e9b400640 1 -- 172.21.15.102:0/657344562 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1e94058710 con 0x7f1ea40c8590 2024-03-18T16:25:41.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.492+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ea40cd7b0 con 0x7f1ea40c8590 2024-03-18T16:25:41.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.561+0000 7f1e9b400640 1 -- 172.21.15.102:0/657344562 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1e9405c7b0 con 0x7f1ea40c8590 2024-03-18T16:25:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:41.564 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:41.564 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:41.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e8406bbd0 msgr2=0x7f1e8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:41.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e8406bbd0 0x7f1e8406e090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1ea414ecb0 tx=0x7f1e90009190 comp rx=0 tx=0).stop 2024-03-18T16:25:41.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ea40c8590 msgr2=0x7f1ea414ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:41.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ea40c8590 0x7f1ea414ed20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400c3c0 tx=0x7f1e9400c880 comp rx=0 tx=0).stop 2024-03-18T16:25:41.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 shutdown_connections 2024-03-18T16:25:41.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e8406bbd0 0x7f1e8406e090 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:41.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1ea4150fb0 0x7f1ea414d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:41.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ea40ce610 0x7f1ea40ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:41.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 --2- 172.21.15.102:0/657344562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ea40c8590 0x7f1ea414ed20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:41.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 >> 172.21.15.102:0/657344562 conn(0x7f1ea4067370 msgr2=0x7f1ea40c3890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:41.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 shutdown_connections 2024-03-18T16:25:41.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:41.565+0000 7f1ea9e00640 1 -- 172.21.15.102:0/657344562 wait complete. 2024-03-18T16:25:42.188 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:42.188 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:43.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.363+0000 7f44e5e00640 1 Processor -- start 2024-03-18T16:25:43.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.363+0000 7f44e5e00640 1 -- start start 2024-03-18T16:25:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.364+0000 7f44e5e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015a310 0x7f44e006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.364+0000 7f44e5e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44e015baa0 0x7f44e015bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:43.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.364+0000 7f44e5e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44e015c470 con 0x7f44e015a6f0 2024-03-18T16:25:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.364+0000 7f44e5e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f44e00cd410 con 0x7f44e015baa0 2024-03-18T16:25:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.364+0000 7f44e5e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f44e00ca500 con 0x7f44e015a310 2024-03-18T16:25:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.365+0000 7f44e4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015a310 0x7f44e006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.365+0000 7f44e5400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44e015baa0 0x7f44e015bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.366+0000 7f44e4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015a310 0x7f44e006d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:32804/0 (socket says 172.21.15.102:32804) 2024-03-18T16:25:43.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.366+0000 7f44e4a00640 1 -- 172.21.15.102:0/3487668966 learned_addr learned my addr 172.21.15.102:0/3487668966 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.368+0000 7f44e4a00640 1 -- 172.21.15.102:0/3487668966 >> v2:172.21.15.154:3300/0 conn(0x7f44e015baa0 msgr2=0x7f44e015bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.368+0000 7f44e4a00640 1 --2- 172.21.15.102:0/3487668966 >> v2:172.21.15.154:3300/0 conn(0x7f44e015baa0 0x7f44e015bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.368+0000 7f44e4a00640 1 -- 172.21.15.102:0/3487668966 >> v1:172.21.15.154:6789/0 conn(0x7f44e015a6f0 legacy=0x7f44e015ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:43.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.368+0000 7f44e4a00640 1 -- 172.21.15.102:0/3487668966 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e00ca7a0 con 0x7f44e015a310 2024-03-18T16:25:43.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.371+0000 7f44e4a00640 1 --2- 172.21.15.102:0/3487668966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015a310 0x7f44e006d3e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009870 tx=0x7f44d402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df2f406db1a6efc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.375+0000 7f44df400640 1 -- 172.21.15.102:0/3487668966 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4034070 con 0x7f44e015a310 2024-03-18T16:25:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.375+0000 7f44df400640 1 -- 172.21.15.102:0/3487668966 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d4030840 con 0x7f44e015a310 2024-03-18T16:25:43.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.375+0000 7f44df400640 1 -- 172.21.15.102:0/3487668966 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d4030c10 con 0x7f44e015a310 2024-03-18T16:25:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.376+0000 7f44e5e00640 1 -- 172.21.15.102:0/3487668966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015a310 msgr2=0x7f44e006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.376+0000 7f44e5e00640 1 --2- 172.21.15.102:0/3487668966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015a310 0x7f44e006d3e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f44d4009870 tx=0x7f44d402b980 comp rx=0 tx=0).stop 2024-03-18T16:25:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.376+0000 7f44e5e00640 1 -- 172.21.15.102:0/3487668966 shutdown_connections 2024-03-18T16:25:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.376+0000 7f44e5e00640 1 --2- 172.21.15.102:0/3487668966 >> v2:172.21.15.154:3300/0 conn(0x7f44e015baa0 0x7f44e015bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.376+0000 7f44e5e00640 1 --2- 172.21.15.102:0/3487668966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015a310 0x7f44e006d3e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.376+0000 7f44e5e00640 1 -- 172.21.15.102:0/3487668966 >> 172.21.15.102:0/3487668966 conn(0x7f44e008c1a0 msgr2=0x7f44e0066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.376+0000 7f44e5e00640 1 -- 172.21.15.102:0/3487668966 shutdown_connections 2024-03-18T16:25:43.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.376+0000 7f44e5e00640 1 -- 172.21.15.102:0/3487668966 wait complete. 2024-03-18T16:25:43.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.378+0000 7f44e5e00640 1 Processor -- start 2024-03-18T16:25:43.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.378+0000 7f44e5e00640 1 -- start start 2024-03-18T16:25:43.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44e5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e015a310 0x7f44e00abbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:43.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44e5e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44e015a6f0 0x7f44e01ee0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44e5e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015baa0 0x7f44e00c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44e5e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f44e00cd130 con 0x7f44e015a310 2024-03-18T16:25:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44e5e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44e00cce50 con 0x7f44e015a6f0 2024-03-18T16:25:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44e5e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f44e006d920 con 0x7f44e015baa0 2024-03-18T16:25:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44e5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015baa0 0x7f44e00c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44dfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44e015a6f0 0x7f44e01ee0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44e4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e015a310 0x7f44e00abbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:43.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.379+0000 7f44e5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015baa0 0x7f44e00c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:32812/0 (socket says 172.21.15.102:32812) 2024-03-18T16:25:43.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.380+0000 7f44e5400640 1 -- 172.21.15.102:0/3070834087 learned_addr learned my addr 172.21.15.102:0/3070834087 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:43.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.383+0000 7f44dfe00640 1 -- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015baa0 msgr2=0x7f44e00c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:43.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.383+0000 7f44dfe00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015baa0 0x7f44e00c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:43.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.383+0000 7f44dfe00640 1 -- 172.21.15.102:0/3070834087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e015a310 msgr2=0x7f44e00abbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:43.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.383+0000 7f44dfe00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e015a310 0x7f44e00abbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:43.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.383+0000 7f44dfe00640 1 -- 172.21.15.102:0/3070834087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44e01f3490 con 0x7f44e015a6f0 2024-03-18T16:25:43.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.388+0000 7f44dfe00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44e015a6f0 0x7f44e01ee0b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00c3c0 tx=0x7f44cc00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:43.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.392+0000 7f44dcc00640 1 -- 172.21.15.102:0/3070834087 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc017070 con 0x7f44e015a6f0 2024-03-18T16:25:43.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.392+0000 7f44dcc00640 1 -- 172.21.15.102:0/3070834087 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44cc0052d0 con 0x7f44e015a6f0 2024-03-18T16:25:43.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.392+0000 7f44dcc00640 1 -- 172.21.15.102:0/3070834087 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc003830 con 0x7f44e015a6f0 2024-03-18T16:25:43.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.392+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44e01f2110 con 0x7f44e015a6f0 2024-03-18T16:25:43.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.392+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44e01f26d0 con 0x7f44e015a6f0 2024-03-18T16:25:43.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.393+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44e0074fd0 con 0x7f44e015a6f0 2024-03-18T16:25:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.652+0000 7f44dcc00640 1 -- 172.21.15.102:0/3070834087 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f44cc01a070 con 0x7f44e015a6f0 2024-03-18T16:25:43.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.654+0000 7f44dcc00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44c006bc20 0x7f44c006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.654+0000 7f44dcc00640 1 -- 172.21.15.102:0/3070834087 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f44cc08a160 con 0x7f44e015a6f0 2024-03-18T16:25:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.654+0000 7f44e4a00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44c006bc20 0x7f44c006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:43.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.655+0000 7f44e4a00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44c006bc20 0x7f44c006e0e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f44d4036000 tx=0x7f44d403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:43.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.663+0000 7f44dcc00640 1 -- 172.21.15.102:0/3070834087 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f44cc021080 con 0x7f44e015a6f0 2024-03-18T16:25:43.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.900+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44e0153e10 con 0x7f44e015a6f0 2024-03-18T16:25:43.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.928+0000 7f44dcc00640 1 -- 172.21.15.102:0/3070834087 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f44cc058310 con 0x7f44e015a6f0 2024-03-18T16:25:43.930 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:43.930 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:43.930 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:43.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44c006bc20 msgr2=0x7f44c006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:43.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44c006bc20 0x7f44c006e0e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f44d4036000 tx=0x7f44d403b040 comp rx=0 tx=0).stop 2024-03-18T16:25:43.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44e015a6f0 msgr2=0x7f44e01ee0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44e015a6f0 0x7f44e01ee0b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00c3c0 tx=0x7f44cc00c880 comp rx=0 tx=0).stop 2024-03-18T16:25:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 shutdown_connections 2024-03-18T16:25:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44c006bc20 0x7f44c006e0e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44e015baa0 0x7f44e00c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44e015a6f0 0x7f44e01ee0b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 --2- 172.21.15.102:0/3070834087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e015a310 0x7f44e00abbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 >> 172.21.15.102:0/3070834087 conn(0x7f44e008c1a0 msgr2=0x7f44e014fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.930+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 shutdown_connections 2024-03-18T16:25:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:43.931+0000 7f44e5e00640 1 -- 172.21.15.102:0/3070834087 wait complete. 2024-03-18T16:25:44.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:44.552 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:45.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.727+0000 7f3b9e600640 1 Processor -- start 2024-03-18T16:25:45.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.727+0000 7f3b9e600640 1 -- start start 2024-03-18T16:25:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.729+0000 7f3b9e600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b980c7f30 0x7f3b98157d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.730+0000 7f3b9e600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b98073530 con 0x7f3b98072810 2024-03-18T16:25:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.730+0000 7f3b9e600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b980c57e0 con 0x7f3b980c2d90 2024-03-18T16:25:45.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.730+0000 7f3b9e600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3b980c5d80 con 0x7f3b980c7f30 2024-03-18T16:25:45.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.730+0000 7f3b97e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3b980c2d90 0x7f3b9815b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:60484/0 (socket says 172.21.15.102:60484) 2024-03-18T16:25:45.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.730+0000 7f3b97e00640 1 -- 172.21.15.102:0/103997597 learned_addr learned my addr 172.21.15.102:0/103997597 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:25:45.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.730+0000 7f3b9d200640 1 --2- 172.21.15.102:0/103997597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b980c7f30 0x7f3b98157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:45.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.733+0000 7f3b97400640 1 -- 172.21.15.102:0/103997597 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2521554141 0 0) 0x7f3b980c57e0 con 0x7f3b980c2d90 2024-03-18T16:25:45.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.733+0000 7f3b97400640 1 -- 172.21.15.102:0/103997597 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b80003520 con 0x7f3b980c2d90 2024-03-18T16:25:45.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.734+0000 7f3b9d200640 1 -- 172.21.15.102:0/103997597 >> v1:172.21.15.154:6789/0 conn(0x7f3b980c2d90 legacy=0x7f3b9815b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:45.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.734+0000 7f3b9d200640 1 -- 172.21.15.102:0/103997597 >> v1:172.21.15.102:6789/0 conn(0x7f3b98072810 legacy=0x7f3b98161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:45.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.734+0000 7f3b9d200640 1 -- 172.21.15.102:0/103997597 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b980c31d0 con 0x7f3b980c7f30 2024-03-18T16:25:45.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.737+0000 7f3b9d200640 1 --2- 172.21.15.102:0/103997597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b980c7f30 0x7f3b98157d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0097d0 tx=0x7f3b8c02b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6521c67584532e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:45.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.742+0000 7f3b97400640 1 -- 172.21.15.102:0/103997597 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c034070 con 0x7f3b980c7f30 2024-03-18T16:25:45.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.742+0000 7f3b97400640 1 -- 172.21.15.102:0/103997597 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b8c030890 con 0x7f3b980c7f30 2024-03-18T16:25:45.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.742+0000 7f3b97400640 1 -- 172.21.15.102:0/103997597 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c030c60 con 0x7f3b980c7f30 2024-03-18T16:25:45.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.743+0000 7f3b9e600640 1 -- 172.21.15.102:0/103997597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b980c7f30 msgr2=0x7f3b98157d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:45.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.743+0000 7f3b9e600640 1 --2- 172.21.15.102:0/103997597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b980c7f30 0x7f3b98157d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0097d0 tx=0x7f3b8c02b9d0 comp rx=0 tx=0).stop 2024-03-18T16:25:45.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.743+0000 7f3b9e600640 1 -- 172.21.15.102:0/103997597 shutdown_connections 2024-03-18T16:25:45.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.743+0000 7f3b9e600640 1 --2- 172.21.15.102:0/103997597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b980c7f30 0x7f3b98157d90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:45.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.743+0000 7f3b9e600640 1 -- 172.21.15.102:0/103997597 >> 172.21.15.102:0/103997597 conn(0x7f3b980cb890 msgr2=0x7f3b980cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:45.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.743+0000 7f3b9e600640 1 -- 172.21.15.102:0/103997597 shutdown_connections 2024-03-18T16:25:45.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.743+0000 7f3b9e600640 1 -- 172.21.15.102:0/103997597 wait complete. 2024-03-18T16:25:45.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.745+0000 7f3b9e600640 1 Processor -- start 2024-03-18T16:25:45.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.745+0000 7f3b9e600640 1 -- start start 2024-03-18T16:25:45.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.746+0000 7f3b9e600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b98072810 0x7f3b9814d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:45.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.746+0000 7f3b9e600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b980c2d90 0x7f3b98156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:45.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.746+0000 7f3b9e600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b980c7f30 0x7f3b9814c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:45.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.746+0000 7f3b9e600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3b981572b0 con 0x7f3b980c7f30 2024-03-18T16:25:45.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.746+0000 7f3b9e600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3b98157900 con 0x7f3b980c2d90 2024-03-18T16:25:45.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.746+0000 7f3b9e600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3b98157590 con 0x7f3b98072810 2024-03-18T16:25:45.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.747+0000 7f3b97e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b980c2d90 0x7f3b98156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:45.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.747+0000 7f3b9d200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b98072810 0x7f3b9814d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:45.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.747+0000 7f3b97e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b980c2d90 0x7f3b98156000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36226/0 (socket says 172.21.15.102:36226) 2024-03-18T16:25:45.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.747+0000 7f3b97e00640 1 -- 172.21.15.102:0/595891445 learned_addr learned my addr 172.21.15.102:0/595891445 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:45.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.747+0000 7f3b9dc00640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b980c7f30 0x7f3b9814c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:45.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.750+0000 7f3b97e00640 1 -- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b98072810 msgr2=0x7f3b9814d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:45.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.750+0000 7f3b97e00640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b98072810 0x7f3b9814d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:45.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.750+0000 7f3b97e00640 1 -- 172.21.15.102:0/595891445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b980c7f30 msgr2=0x7f3b9814c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:45.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.750+0000 7f3b97e00640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b980c7f30 0x7f3b9814c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:45.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.750+0000 7f3b97e00640 1 -- 172.21.15.102:0/595891445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9814e600 con 0x7f3b980c2d90 2024-03-18T16:25:45.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.753+0000 7f3b97e00640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b980c2d90 0x7f3b98156000 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400c530 tx=0x7f3b8400c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:45.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.757+0000 7f3b94c00640 1 -- 172.21.15.102:0/595891445 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b84017070 con 0x7f3b980c2d90 2024-03-18T16:25:45.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.757+0000 7f3b94c00640 1 -- 172.21.15.102:0/595891445 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b84005ce0 con 0x7f3b980c2d90 2024-03-18T16:25:45.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.757+0000 7f3b94c00640 1 -- 172.21.15.102:0/595891445 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b840072b0 con 0x7f3b980c2d90 2024-03-18T16:25:45.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.757+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b98156b80 con 0x7f3b980c2d90 2024-03-18T16:25:45.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.757+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b98155ae0 con 0x7f3b980c2d90 2024-03-18T16:25:45.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.758+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b98155ce0 con 0x7f3b980c2d90 2024-03-18T16:25:45.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.777+0000 7f3b94c00640 1 -- 172.21.15.102:0/595891445 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3b8401a070 con 0x7f3b980c2d90 2024-03-18T16:25:45.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.778+0000 7f3b94c00640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b7806bbd0 0x7f3b7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:45.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.779+0000 7f3b9d200640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b7806bbd0 0x7f3b7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:45.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:45.779+0000 7f3b9d200640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b7806bbd0 0x7f3b7806e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c036000 tx=0x7f3b8c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:46.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.025+0000 7f3b94c00640 1 -- 172.21.15.102:0/595891445 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3b84056860 con 0x7f3b980c2d90 2024-03-18T16:25:46.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.050+0000 7f3b94c00640 1 -- 172.21.15.102:0/595891445 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3b98155ce0 con 0x7f3b980c2d90 2024-03-18T16:25:46.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.292+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b98074fd0 con 0x7f3b980c2d90 2024-03-18T16:25:46.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.317+0000 7f3b94c00640 1 -- 172.21.15.102:0/595891445 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3b84021020 con 0x7f3b980c2d90 2024-03-18T16:25:46.319 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:46.319 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:46.320 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b7806bbd0 msgr2=0x7f3b7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b7806bbd0 0x7f3b7806e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c036000 tx=0x7f3b8c03b040 comp rx=0 tx=0).stop 2024-03-18T16:25:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b980c2d90 msgr2=0x7f3b98156000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b980c2d90 0x7f3b98156000 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400c530 tx=0x7f3b8400c9f0 comp rx=0 tx=0).stop 2024-03-18T16:25:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 shutdown_connections 2024-03-18T16:25:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b7806bbd0 0x7f3b7806e090 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b980c7f30 0x7f3b9814c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b980c2d90 0x7f3b98156000 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400c530 tx=0x7f3b8400c9f0 comp rx=0 tx=0).stop 2024-03-18T16:25:46.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 --2- 172.21.15.102:0/595891445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3b98072810 0x7f3b9814d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:46.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 >> 172.21.15.102:0/595891445 conn(0x7f3b980cb890 msgr2=0x7f3b981476e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:46.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 shutdown_connections 2024-03-18T16:25:46.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:46.320+0000 7f3b9e600640 1 -- 172.21.15.102:0/595891445 wait complete. 2024-03-18T16:25:46.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:46.942 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:47.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.122+0000 7f7af4a00640 1 Processor -- start 2024-03-18T16:25:48.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.122+0000 7f7af4a00640 1 -- start start 2024-03-18T16:25:48.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.124+0000 7f7af4a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7af0141650 0x7f7af0158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:48.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.124+0000 7f7af4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af015c3a0 0x7f7af00c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:48.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.124+0000 7f7af4a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7af0158ad0 con 0x7f7af00c4fa0 2024-03-18T16:25:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.124+0000 7f7af4a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7af0159370 con 0x7f7af0141650 2024-03-18T16:25:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.124+0000 7f7af4a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7af0158db0 con 0x7f7af015c3a0 2024-03-18T16:25:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.125+0000 7f7aeea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af015c3a0 0x7f7af00c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:48.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.125+0000 7f7aef400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7af0141650 0x7f7af0158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:48.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.125+0000 7f7aeea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af015c3a0 0x7f7af00c6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:32828/0 (socket says 172.21.15.102:32828) 2024-03-18T16:25:48.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.125+0000 7f7aeea00640 1 -- 172.21.15.102:0/1899412543 learned_addr learned my addr 172.21.15.102:0/1899412543 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:48.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.128+0000 7f7aef400640 1 -- 172.21.15.102:0/1899412543 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af015c3a0 msgr2=0x7f7af00c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:48.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.128+0000 7f7aef400640 1 --2- 172.21.15.102:0/1899412543 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af015c3a0 0x7f7af00c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:48.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.128+0000 7f7aef400640 1 -- 172.21.15.102:0/1899412543 >> v1:172.21.15.154:6789/0 conn(0x7f7af00c4fa0 legacy=0x7f7af006f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:48.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.128+0000 7f7aef400640 1 -- 172.21.15.102:0/1899412543 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af0159090 con 0x7f7af0141650 2024-03-18T16:25:48.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.132+0000 7f7aef400640 1 --2- 172.21.15.102:0/1899412543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0141650 0x7f7af0158420 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00097a0 tx=0x7f7ae002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e230b0644c6faf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:48.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.134+0000 7f7aee000640 1 -- 172.21.15.102:0/1899412543 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0031780 con 0x7f7af0141650 2024-03-18T16:25:48.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7aee000640 1 -- 172.21.15.102:0/1899412543 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae0037070 con 0x7f7af0141650 2024-03-18T16:25:48.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7aee000640 1 -- 172.21.15.102:0/1899412543 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0031b40 con 0x7f7af0141650 2024-03-18T16:25:48.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7af4a00640 1 -- 172.21.15.102:0/1899412543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0141650 msgr2=0x7f7af0158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:48.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7af4a00640 1 --2- 172.21.15.102:0/1899412543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0141650 0x7f7af0158420 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00097a0 tx=0x7f7ae002b820 comp rx=0 tx=0).stop 2024-03-18T16:25:48.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7af4a00640 1 -- 172.21.15.102:0/1899412543 shutdown_connections 2024-03-18T16:25:48.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7af4a00640 1 --2- 172.21.15.102:0/1899412543 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af015c3a0 0x7f7af00c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:48.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7af4a00640 1 --2- 172.21.15.102:0/1899412543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af0141650 0x7f7af0158420 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:48.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7af4a00640 1 -- 172.21.15.102:0/1899412543 >> 172.21.15.102:0/1899412543 conn(0x7f7af00cb890 msgr2=0x7f7af00cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:48.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7af4a00640 1 -- 172.21.15.102:0/1899412543 shutdown_connections 2024-03-18T16:25:48.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.136+0000 7f7af4a00640 1 -- 172.21.15.102:0/1899412543 wait complete. 2024-03-18T16:25:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.138+0000 7f7af4a00640 1 Processor -- start 2024-03-18T16:25:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.138+0000 7f7af4a00640 1 -- start start 2024-03-18T16:25:48.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.139+0000 7f7af4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af00c4fa0 0x7f7af0157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:48.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.139+0000 7f7af4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af0141650 0x7f7af015a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:48.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.139+0000 7f7af4a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af015c3a0 0x7f7af01e4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:48.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.139+0000 7f7af4a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7af015c7c0 con 0x7f7af0141650 2024-03-18T16:25:48.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.139+0000 7f7af4a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7af00c66a0 con 0x7f7af015c3a0 2024-03-18T16:25:48.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.139+0000 7f7af4a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7af0141a30 con 0x7f7af00c4fa0 2024-03-18T16:25:48.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.140+0000 7f7aef400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af00c4fa0 0x7f7af0157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:48.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.140+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af015c3a0 0x7f7af01e4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:48.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.140+0000 7f7aeea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af0141650 0x7f7af015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:48.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.140+0000 7f7aef400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af00c4fa0 0x7f7af0157960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:32836/0 (socket says 172.21.15.102:32836) 2024-03-18T16:25:48.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.140+0000 7f7aef400640 1 -- 172.21.15.102:0/374947648 learned_addr learned my addr 172.21.15.102:0/374947648 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:48.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.140+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af015c3a0 0x7f7af01e4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36240/0 (socket says 172.21.15.102:36240) 2024-03-18T16:25:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.145+0000 7f7aefe00640 1 -- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af00c4fa0 msgr2=0x7f7af0157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.145+0000 7f7aefe00640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af00c4fa0 0x7f7af0157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.145+0000 7f7aefe00640 1 -- 172.21.15.102:0/374947648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af0141650 msgr2=0x7f7af015a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.145+0000 7f7aefe00640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af0141650 0x7f7af015a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:48.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.145+0000 7f7aefe00640 1 -- 172.21.15.102:0/374947648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af01565e0 con 0x7f7af015c3a0 2024-03-18T16:25:48.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.148+0000 7f7aefe00640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af015c3a0 0x7f7af01e4ae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800c780 tx=0x7f7ae800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:48.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.152+0000 7f7ae7400640 1 -- 172.21.15.102:0/374947648 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8017070 con 0x7f7af015c3a0 2024-03-18T16:25:48.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.152+0000 7f7ae7400640 1 -- 172.21.15.102:0/374947648 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae8003b90 con 0x7f7af015c3a0 2024-03-18T16:25:48.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.152+0000 7f7ae7400640 1 -- 172.21.15.102:0/374947648 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8005070 con 0x7f7af015c3a0 2024-03-18T16:25:48.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.152+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af01568c0 con 0x7f7af015c3a0 2024-03-18T16:25:48.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.152+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af0156e80 con 0x7f7af015c3a0 2024-03-18T16:25:48.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.154+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af0152590 con 0x7f7af015c3a0 2024-03-18T16:25:48.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.172+0000 7f7ae7400640 1 -- 172.21.15.102:0/374947648 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7ae801a070 con 0x7f7af015c3a0 2024-03-18T16:25:48.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.173+0000 7f7ae7400640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bc50 0x7f7ad006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:48.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.174+0000 7f7aef400640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bc50 0x7f7ad006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:48.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.175+0000 7f7aef400640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bc50 0x7f7ad006e110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000c00 tx=0x7f7ae003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:48.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.179+0000 7f7ae7400640 1 -- 172.21.15.102:0/374947648 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7ae8056940 con 0x7f7af015c3a0 2024-03-18T16:25:48.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.429+0000 7f7ae7400640 1 -- 172.21.15.102:0/374947648 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7af0152590 con 0x7f7af015c3a0 2024-03-18T16:25:48.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.668+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7af0074fd0 con 0x7f7af015c3a0 2024-03-18T16:25:48.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.682+0000 7f7ae7400640 1 -- 172.21.15.102:0/374947648 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7ae8021020 con 0x7f7af015c3a0 2024-03-18T16:25:48.684 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:48.684 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:48.684 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bc50 msgr2=0x7f7ad006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bc50 0x7f7ad006e110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000c00 tx=0x7f7ae003d040 comp rx=0 tx=0).stop 2024-03-18T16:25:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af015c3a0 msgr2=0x7f7af01e4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af015c3a0 0x7f7af01e4ae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800c780 tx=0x7f7ae800cc40 comp rx=0 tx=0).stop 2024-03-18T16:25:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 shutdown_connections 2024-03-18T16:25:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bc50 0x7f7ad006e110 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af015c3a0 0x7f7af01e4ae0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af0141650 0x7f7af015a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 --2- 172.21.15.102:0/374947648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af00c4fa0 0x7f7af0157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 >> 172.21.15.102:0/374947648 conn(0x7f7af00cb890 msgr2=0x7f7af00cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 shutdown_connections 2024-03-18T16:25:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:48.684+0000 7f7af4a00640 1 -- 172.21.15.102:0/374947648 wait complete. 2024-03-18T16:25:49.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:49.309 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:50.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:50.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.484+0000 7fd59fe00640 1 Processor -- start 2024-03-18T16:25:50.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.484+0000 7fd59fe00640 1 -- start start 2024-03-18T16:25:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.486+0000 7fd59fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd5a015ad60 0x7fd5a00c5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:50.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.486+0000 7fd59fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a006bf60 con 0x7fd5a006de50 2024-03-18T16:25:50.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.486+0000 7fd59fe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a00c94b0 con 0x7fd5a006b470 2024-03-18T16:25:50.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.486+0000 7fd59fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd5a00ca0a0 con 0x7fd5a015ad60 2024-03-18T16:25:50.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.486+0000 7fd59ea00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd5a006b470 0x7fd5a006b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:60496/0 (socket says 172.21.15.102:60496) 2024-03-18T16:25:50.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.486+0000 7fd59ea00640 1 -- 172.21.15.102:0/1479585470 learned_addr learned my addr 172.21.15.102:0/1479585470 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:25:50.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.487+0000 7fd59f400640 1 --2- 172.21.15.102:0/1479585470 >> v2:172.21.15.102:3300/0 conn(0x7fd5a015ad60 0x7fd5a00c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:50.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.489+0000 7fd59d600640 1 -- 172.21.15.102:0/1479585470 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 840093315 0 0) 0x7fd5a00c94b0 con 0x7fd5a006b470 2024-03-18T16:25:50.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.489+0000 7fd59d600640 1 -- 172.21.15.102:0/1479585470 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd584003520 con 0x7fd5a006b470 2024-03-18T16:25:50.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.492+0000 7fd59f400640 1 -- 172.21.15.102:0/1479585470 >> v1:172.21.15.154:6789/0 conn(0x7fd5a006b470 legacy=0x7fd5a006b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:50.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.492+0000 7fd59f400640 1 -- 172.21.15.102:0/1479585470 >> v1:172.21.15.102:6789/0 conn(0x7fd5a006de50 legacy=0x7fd5a0154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:50.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.492+0000 7fd59f400640 1 -- 172.21.15.102:0/1479585470 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a00c9770 con 0x7fd5a015ad60 2024-03-18T16:25:50.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.498+0000 7fd59f400640 1 --2- 172.21.15.102:0/1479585470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a015ad60 0x7fd5a00c5190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd598009d90 tx=0x7fd59802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90795223d8542fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.503+0000 7fd59d600640 1 -- 172.21.15.102:0/1479585470 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598039070 con 0x7fd5a015ad60 2024-03-18T16:25:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.503+0000 7fd59d600640 1 -- 172.21.15.102:0/1479585470 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd59802ccf0 con 0x7fd5a015ad60 2024-03-18T16:25:50.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.503+0000 7fd59d600640 1 -- 172.21.15.102:0/1479585470 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598034660 con 0x7fd5a015ad60 2024-03-18T16:25:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.503+0000 7fd59fe00640 1 -- 172.21.15.102:0/1479585470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a015ad60 msgr2=0x7fd5a00c5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.503+0000 7fd59fe00640 1 --2- 172.21.15.102:0/1479585470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a015ad60 0x7fd5a00c5190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd598009d90 tx=0x7fd59802bf40 comp rx=0 tx=0).stop 2024-03-18T16:25:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.504+0000 7fd59fe00640 1 -- 172.21.15.102:0/1479585470 shutdown_connections 2024-03-18T16:25:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.504+0000 7fd59fe00640 1 --2- 172.21.15.102:0/1479585470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a015ad60 0x7fd5a00c5190 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.504+0000 7fd59fe00640 1 -- 172.21.15.102:0/1479585470 >> 172.21.15.102:0/1479585470 conn(0x7fd5a00727e0 msgr2=0x7fd5a0072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.504+0000 7fd59fe00640 1 -- 172.21.15.102:0/1479585470 shutdown_connections 2024-03-18T16:25:50.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.504+0000 7fd59fe00640 1 -- 172.21.15.102:0/1479585470 wait complete. 2024-03-18T16:25:50.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.506+0000 7fd59fe00640 1 Processor -- start 2024-03-18T16:25:50.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.506+0000 7fd59fe00640 1 -- start start 2024-03-18T16:25:50.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.507+0000 7fd59fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a006b470 0x7fd5a00bdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:50.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.507+0000 7fd59fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5a006de50 0x7fd5a0155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:50.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.507+0000 7fd59fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5a015ad60 0x7fd5a0155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:50.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.507+0000 7fd59fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a0154860 con 0x7fd5a006b470 2024-03-18T16:25:50.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.507+0000 7fd59fe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a00c9d30 con 0x7fd5a006de50 2024-03-18T16:25:50.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.507+0000 7fd59fe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a01549e0 con 0x7fd5a015ad60 2024-03-18T16:25:50.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.507+0000 7fd59e000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5a006de50 0x7fd5a0155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:50.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.508+0000 7fd59ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a006b470 0x7fd5a00bdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:50.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.508+0000 7fd59e000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5a006de50 0x7fd5a0155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36246/0 (socket says 172.21.15.102:36246) 2024-03-18T16:25:50.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.508+0000 7fd59e000640 1 -- 172.21.15.102:0/468981326 learned_addr learned my addr 172.21.15.102:0/468981326 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:50.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.508+0000 7fd59f400640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5a015ad60 0x7fd5a0155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:50.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.511+0000 7fd59e000640 1 -- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5a015ad60 msgr2=0x7fd5a0155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:50.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.511+0000 7fd59e000640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5a015ad60 0x7fd5a0155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:50.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.511+0000 7fd59e000640 1 -- 172.21.15.102:0/468981326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a006b470 msgr2=0x7fd5a00bdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:50.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.511+0000 7fd59e000640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a006b470 0x7fd5a00bdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:50.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.511+0000 7fd59e000640 1 -- 172.21.15.102:0/468981326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a0156f30 con 0x7fd5a006de50 2024-03-18T16:25:50.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.513+0000 7fd59e000640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5a006de50 0x7fd5a0155860 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00c400 tx=0x7fd58c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:50.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.518+0000 7fd597400640 1 -- 172.21.15.102:0/468981326 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c0052d0 con 0x7fd5a006de50 2024-03-18T16:25:50.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.518+0000 7fd597400640 1 -- 172.21.15.102:0/468981326 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd58c005490 con 0x7fd5a006de50 2024-03-18T16:25:50.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.518+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a01571f0 con 0x7fd5a006de50 2024-03-18T16:25:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.518+0000 7fd597400640 1 -- 172.21.15.102:0/468981326 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58c007260 con 0x7fd5a006de50 2024-03-18T16:25:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.518+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a01577b0 con 0x7fd5a006de50 2024-03-18T16:25:50.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.519+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a00cc5f0 con 0x7fd5a006de50 2024-03-18T16:25:50.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.545+0000 7fd597400640 1 -- 172.21.15.102:0/468981326 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd58c007400 con 0x7fd5a006de50 2024-03-18T16:25:50.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.546+0000 7fd597400640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd58006bc20 0x7fd58006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:50.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.547+0000 7fd59ea00640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd58006bc20 0x7fd58006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:50.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.547+0000 7fd597400640 1 -- 172.21.15.102:0/468981326 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd58c056960 con 0x7fd5a006de50 2024-03-18T16:25:50.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.547+0000 7fd59ea00640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd58006bc20 0x7fd58006e0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd5900011e0 tx=0x7fd590007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:50.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:50.807+0000 7fd597400640 1 -- 172.21.15.102:0/468981326 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd58c0588c0 con 0x7fd5a006de50 2024-03-18T16:25:51.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.054+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5a015cd20 con 0x7fd5a006de50 2024-03-18T16:25:51.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.107+0000 7fd597400640 1 -- 172.21.15.102:0/468981326 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd58c05c960 con 0x7fd5a006de50 2024-03-18T16:25:51.109 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:51.110 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:51.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd58006bc20 msgr2=0x7fd58006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:51.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd58006bc20 0x7fd58006e0e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd5900011e0 tx=0x7fd590007d30 comp rx=0 tx=0).stop 2024-03-18T16:25:51.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5a006de50 msgr2=0x7fd5a0155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:51.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5a006de50 0x7fd5a0155860 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00c400 tx=0x7fd58c00c8c0 comp rx=0 tx=0).stop 2024-03-18T16:25:51.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 shutdown_connections 2024-03-18T16:25:51.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd58006bc20 0x7fd58006e0e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5a015ad60 0x7fd5a0155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5a006de50 0x7fd5a0155860 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 --2- 172.21.15.102:0/468981326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a006b470 0x7fd5a00bdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 >> 172.21.15.102:0/468981326 conn(0x7fd5a00727e0 msgr2=0x7fd5a00c6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 shutdown_connections 2024-03-18T16:25:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:51.111+0000 7fd59fe00640 1 -- 172.21.15.102:0/468981326 wait complete. 2024-03-18T16:25:51.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:25:51.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:25:51.741 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:51.742 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:52.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:52.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.920+0000 7f13d7e00640 1 Processor -- start 2024-03-18T16:25:52.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.920+0000 7f13d7e00640 1 -- start start 2024-03-18T16:25:52.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.921+0000 7f13d7e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13d80cfe70 0x7f13d815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:52.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.921+0000 7f13d7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80d0250 0x7f13d80c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:52.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.921+0000 7f13d7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13d80cc2f0 0x7f13d80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:52.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.921+0000 7f13d7e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f13d80ccca0 con 0x7f13d80cc2f0 2024-03-18T16:25:52.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.921+0000 7f13d7e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f13d80cabd0 con 0x7f13d80cfe70 2024-03-18T16:25:52.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.921+0000 7f13d7e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f13d80caea0 con 0x7f13d80d0250 2024-03-18T16:25:52.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.922+0000 7f13d6a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13d80cfe70 0x7f13d815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:52.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.922+0000 7f13d7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13d80cc2f0 0x7f13d80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:52.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.922+0000 7f13d6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80d0250 0x7f13d80c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:52.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.922+0000 7f13d6a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13d80cfe70 0x7f13d815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42244/0 (socket says 172.21.15.102:42244) 2024-03-18T16:25:52.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.922+0000 7f13d6a00640 1 -- 172.21.15.102:0/4026036627 learned_addr learned my addr 172.21.15.102:0/4026036627 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:52.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.925+0000 7f13d7400640 1 -- 172.21.15.102:0/4026036627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80d0250 msgr2=0x7f13d80c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:52.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.925+0000 7f13d7400640 1 --2- 172.21.15.102:0/4026036627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80d0250 0x7f13d80c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:52.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.925+0000 7f13d7400640 1 -- 172.21.15.102:0/4026036627 >> v2:172.21.15.154:3300/0 conn(0x7f13d80cfe70 msgr2=0x7f13d815ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:52.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.925+0000 7f13d7400640 1 --2- 172.21.15.102:0/4026036627 >> v2:172.21.15.154:3300/0 conn(0x7f13d80cfe70 0x7f13d815ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:52.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.925+0000 7f13d7400640 1 -- 172.21.15.102:0/4026036627 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d80cb180 con 0x7f13d80cc2f0 2024-03-18T16:25:52.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.927+0000 7f13d7400640 1 --2- 172.21.15.102:0/4026036627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13d80cc2f0 0x7f13d80cc760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f13d80cd800 tx=0x7f13d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8bfcffa0ee31f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:52.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.933+0000 7f13d5600640 1 -- 172.21.15.102:0/4026036627 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0039070 con 0x7f13d80cc2f0 2024-03-18T16:25:52.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.934+0000 7f13d5600640 1 -- 172.21.15.102:0/4026036627 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d002cc60 con 0x7f13d80cc2f0 2024-03-18T16:25:52.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.934+0000 7f13d5600640 1 -- 172.21.15.102:0/4026036627 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0034660 con 0x7f13d80cc2f0 2024-03-18T16:25:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.935+0000 7f13d7e00640 1 -- 172.21.15.102:0/4026036627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13d80cc2f0 msgr2=0x7f13d80cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.935+0000 7f13d7e00640 1 --2- 172.21.15.102:0/4026036627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13d80cc2f0 0x7f13d80cc760 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f13d80cd800 tx=0x7f13d002beb0 comp rx=0 tx=0).stop 2024-03-18T16:25:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.935+0000 7f13d7e00640 1 -- 172.21.15.102:0/4026036627 shutdown_connections 2024-03-18T16:25:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.935+0000 7f13d7e00640 1 --2- 172.21.15.102:0/4026036627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13d80cc2f0 0x7f13d80cc760 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.935+0000 7f13d7e00640 1 --2- 172.21.15.102:0/4026036627 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80d0250 0x7f13d80c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.935+0000 7f13d7e00640 1 --2- 172.21.15.102:0/4026036627 >> v2:172.21.15.154:3300/0 conn(0x7f13d80cfe70 0x7f13d815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.935+0000 7f13d7e00640 1 -- 172.21.15.102:0/4026036627 >> 172.21.15.102:0/4026036627 conn(0x7f13d808c3e0 msgr2=0x7f13d806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.935+0000 7f13d7e00640 1 -- 172.21.15.102:0/4026036627 shutdown_connections 2024-03-18T16:25:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.935+0000 7f13d7e00640 1 -- 172.21.15.102:0/4026036627 wait complete. 2024-03-18T16:25:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.937+0000 7f13d7e00640 1 Processor -- start 2024-03-18T16:25:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.938+0000 7f13d7e00640 1 -- start start 2024-03-18T16:25:52.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.939+0000 7f13d7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13d80cc2f0 0x7f13d81522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:52.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.939+0000 7f13d7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80cfe70 0x7f13d81e1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:52.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.939+0000 7f13d7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d80d0250 0x7f13d81e1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:52.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.939+0000 7f13d7e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f13d80cba20 con 0x7f13d80cc2f0 2024-03-18T16:25:52.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.939+0000 7f13d7e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13d80cb740 con 0x7f13d80d0250 2024-03-18T16:25:52.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.939+0000 7f13d7e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f13d80cbd00 con 0x7f13d80cfe70 2024-03-18T16:25:52.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.940+0000 7f13d7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d80d0250 0x7f13d81e1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:52.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.940+0000 7f13d6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80cfe70 0x7f13d81e1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:52.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.940+0000 7f13d6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13d80cc2f0 0x7f13d81522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:52.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.940+0000 7f13d7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d80d0250 0x7f13d81e1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42254/0 (socket says 172.21.15.102:42254) 2024-03-18T16:25:52.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.940+0000 7f13d7400640 1 -- 172.21.15.102:0/3211539082 learned_addr learned my addr 172.21.15.102:0/3211539082 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:52.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.944+0000 7f13d7400640 1 -- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80cfe70 msgr2=0x7f13d81e1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:52.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.944+0000 7f13d7400640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80cfe70 0x7f13d81e1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:52.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.944+0000 7f13d7400640 1 -- 172.21.15.102:0/3211539082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13d80cc2f0 msgr2=0x7f13d81522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:52.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.944+0000 7f13d7400640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13d80cc2f0 0x7f13d81522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:52.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.944+0000 7f13d7400640 1 -- 172.21.15.102:0/3211539082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d8152e00 con 0x7f13d80d0250 2024-03-18T16:25:52.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.946+0000 7f13d7400640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d80d0250 0x7f13d81e1550 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009e50 tx=0x7f13d0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.951+0000 7f13cea00640 1 -- 172.21.15.102:0/3211539082 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0042020 con 0x7f13d80d0250 2024-03-18T16:25:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.951+0000 7f13cea00640 1 -- 172.21.15.102:0/3211539082 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d003a040 con 0x7f13d80d0250 2024-03-18T16:25:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.951+0000 7f13cea00640 1 -- 172.21.15.102:0/3211539082 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0039070 con 0x7f13d80d0250 2024-03-18T16:25:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.951+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d81573b0 con 0x7f13d80d0250 2024-03-18T16:25:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.951+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d81f09e0 con 0x7f13d80d0250 2024-03-18T16:25:52.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.953+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d80d0c00 con 0x7f13d80d0250 2024-03-18T16:25:52.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.980+0000 7f13cea00640 1 -- 172.21.15.102:0/3211539082 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f13d0005d10 con 0x7f13d80d0250 2024-03-18T16:25:52.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.981+0000 7f13cea00640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13b806bc20 0x7f13b806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:52.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.982+0000 7f13d6a00640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13b806bc20 0x7f13b806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:52.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.983+0000 7f13d6a00640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13b806bc20 0x7f13b806e0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f13c8006cc0 tx=0x7f13c8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:52.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:52.983+0000 7f13cea00640 1 -- 172.21.15.102:0/3211539082 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f13d0079900 con 0x7f13d80d0250 2024-03-18T16:25:53.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.236+0000 7f13cea00640 1 -- 172.21.15.102:0/3211539082 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f13d80d0c00 con 0x7f13d80d0250 2024-03-18T16:25:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.498+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13d80c87c0 con 0x7f13d80d0250 2024-03-18T16:25:53.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.516+0000 7f13cea00640 1 -- 172.21.15.102:0/3211539082 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f13d80c87c0 con 0x7f13d80d0250 2024-03-18T16:25:53.518 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:53.518 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:53.518 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.519+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13b806bc20 msgr2=0x7f13b806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.519+0000 7f13d7e00640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13b806bc20 0x7f13b806e0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f13c8006cc0 tx=0x7f13c8009240 comp rx=0 tx=0).stop 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.519+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d80d0250 msgr2=0x7f13d81e1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.519+0000 7f13d7e00640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d80d0250 0x7f13d81e1550 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009e50 tx=0x7f13d0004a40 comp rx=0 tx=0).stop 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.520+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 shutdown_connections 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.520+0000 7f13d7e00640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13b806bc20 0x7f13b806e0e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.520+0000 7f13d7e00640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13d80d0250 0x7f13d81e1550 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.520+0000 7f13d7e00640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13d80cfe70 0x7f13d81e1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.520+0000 7f13d7e00640 1 --2- 172.21.15.102:0/3211539082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13d80cc2f0 0x7f13d81522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.520+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 >> 172.21.15.102:0/3211539082 conn(0x7f13d808c3e0 msgr2=0x7f13d80cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.520+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 shutdown_connections 2024-03-18T16:25:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:53.520+0000 7f13d7e00640 1 -- 172.21.15.102:0/3211539082 wait complete. 2024-03-18T16:25:54.142 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:54.142 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:55.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:55.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.382+0000 7f6d8b200640 1 Processor -- start 2024-03-18T16:25:55.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.382+0000 7f6d8b200640 1 -- start start 2024-03-18T16:25:55.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.384+0000 7f6d8b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d840c8b20 0x7f6d84072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:55.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.384+0000 7f6d8b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6d84158760 0x7f6d8415b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:55.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.384+0000 7f6d8b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d84157ac0 con 0x7f6d840c8740 2024-03-18T16:25:55.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.384+0000 7f6d8b200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6d8415a860 con 0x7f6d84158760 2024-03-18T16:25:55.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.384+0000 7f6d8b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d84072840 con 0x7f6d840c8b20 2024-03-18T16:25:55.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.385+0000 7f6d83400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d840c8b20 0x7f6d84072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:55.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.385+0000 7f6d83400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d840c8b20 0x7f6d84072300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:39244/0 (socket says 172.21.15.102:39244) 2024-03-18T16:25:55.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.385+0000 7f6d83400640 1 -- 172.21.15.102:0/1800933144 learned_addr learned my addr 172.21.15.102:0/1800933144 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:55.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.387+0000 7f6d83400640 1 -- 172.21.15.102:0/1800933144 >> v2:172.21.15.102:3300/0 conn(0x7f6d84158760 msgr2=0x7f6d8415b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:55.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.387+0000 7f6d83400640 1 --2- 172.21.15.102:0/1800933144 >> v2:172.21.15.102:3300/0 conn(0x7f6d84158760 0x7f6d8415b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:55.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.387+0000 7f6d83400640 1 -- 172.21.15.102:0/1800933144 >> v1:172.21.15.102:6789/0 conn(0x7f6d840c8740 legacy=0x7f6d841572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:55.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.387+0000 7f6d83400640 1 -- 172.21.15.102:0/1800933144 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d84157da0 con 0x7f6d840c8b20 2024-03-18T16:25:55.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.390+0000 7f6d83400640 1 --2- 172.21.15.102:0/1800933144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d840c8b20 0x7f6d84072300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009870 tx=0x7f6d7002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8011c5331d0e68a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:55.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.392+0000 7f6d82a00640 1 -- 172.21.15.102:0/1800933144 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70035020 con 0x7f6d840c8b20 2024-03-18T16:25:55.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.394+0000 7f6d82a00640 1 -- 172.21.15.102:0/1800933144 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d70031640 con 0x7f6d840c8b20 2024-03-18T16:25:55.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.394+0000 7f6d82a00640 1 -- 172.21.15.102:0/1800933144 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7002d040 con 0x7f6d840c8b20 2024-03-18T16:25:55.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.395+0000 7f6d8b200640 1 -- 172.21.15.102:0/1800933144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d840c8b20 msgr2=0x7f6d84072300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:55.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.395+0000 7f6d8b200640 1 --2- 172.21.15.102:0/1800933144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d840c8b20 0x7f6d84072300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009870 tx=0x7f6d7002b8f0 comp rx=0 tx=0).stop 2024-03-18T16:25:55.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.395+0000 7f6d8b200640 1 -- 172.21.15.102:0/1800933144 shutdown_connections 2024-03-18T16:25:55.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.395+0000 7f6d8b200640 1 --2- 172.21.15.102:0/1800933144 >> v2:172.21.15.102:3300/0 conn(0x7f6d84158760 0x7f6d8415b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:55.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.395+0000 7f6d8b200640 1 --2- 172.21.15.102:0/1800933144 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d840c8b20 0x7f6d84072300 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:55.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.395+0000 7f6d8b200640 1 -- 172.21.15.102:0/1800933144 >> 172.21.15.102:0/1800933144 conn(0x7f6d840700d0 msgr2=0x7f6d8406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:55.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.395+0000 7f6d8b200640 1 -- 172.21.15.102:0/1800933144 shutdown_connections 2024-03-18T16:25:55.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.395+0000 7f6d8b200640 1 -- 172.21.15.102:0/1800933144 wait complete. 2024-03-18T16:25:55.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.398+0000 7f6d8b200640 1 Processor -- start 2024-03-18T16:25:55.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.398+0000 7f6d8b200640 1 -- start start 2024-03-18T16:25:55.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.399+0000 7f6d8b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d840c8740 0x7f6d840cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:55.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.399+0000 7f6d8b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d840c8b20 0x7f6d84145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:55.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.399+0000 7f6d8b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d84158760 0x7f6d84147940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:55.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.399+0000 7f6d8b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8415aab0 con 0x7f6d840c8740 2024-03-18T16:25:55.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.399+0000 7f6d8b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8415b7f0 con 0x7f6d840c8b20 2024-03-18T16:25:55.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.399+0000 7f6d8b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6d8415ad90 con 0x7f6d84158760 2024-03-18T16:25:55.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.400+0000 7f6d83400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d840c8b20 0x7f6d84145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:55.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.400+0000 7f6d89200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d84158760 0x7f6d84147940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:55.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.400+0000 7f6d83400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d840c8b20 0x7f6d84145430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42256/0 (socket says 172.21.15.102:42256) 2024-03-18T16:25:55.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.400+0000 7f6d83400640 1 -- 172.21.15.102:0/4225777412 learned_addr learned my addr 172.21.15.102:0/4225777412 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:55.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.403+0000 7f6d83e00640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d840c8740 0x7f6d840cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:55.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.404+0000 7f6d83400640 1 -- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d84158760 msgr2=0x7f6d84147940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.404+0000 7f6d83400640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d84158760 0x7f6d84147940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.404+0000 7f6d83400640 1 -- 172.21.15.102:0/4225777412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d840c8740 msgr2=0x7f6d840cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.404+0000 7f6d83400640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d840c8740 0x7f6d840cc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:55.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.404+0000 7f6d83400640 1 -- 172.21.15.102:0/4225777412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d841edff0 con 0x7f6d840c8b20 2024-03-18T16:25:55.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.407+0000 7f6d83400640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d840c8b20 0x7f6d84145430 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6d7002b930 tx=0x7f6d700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:55.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.415+0000 7f6d7be00640 1 -- 172.21.15.102:0/4225777412 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7003ac60 con 0x7f6d840c8b20 2024-03-18T16:25:55.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.415+0000 7f6d7be00640 1 -- 172.21.15.102:0/4225777412 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d7003ae00 con 0x7f6d840c8b20 2024-03-18T16:25:55.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.415+0000 7f6d7be00640 1 -- 172.21.15.102:0/4225777412 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70044610 con 0x7f6d840c8b20 2024-03-18T16:25:55.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.415+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d841ee2d0 con 0x7f6d840c8b20 2024-03-18T16:25:55.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.415+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d841ee890 con 0x7f6d840c8b20 2024-03-18T16:25:55.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.416+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d84074fd0 con 0x7f6d840c8b20 2024-03-18T16:25:55.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.723+0000 7f6d7be00640 1 -- 172.21.15.102:0/4225777412 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6d7004b060 con 0x7f6d840c8b20 2024-03-18T16:25:55.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.724+0000 7f6d7be00640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d6406bc50 0x7f6d6406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:55.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.725+0000 7f6d83e00640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d6406bc50 0x7f6d6406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:55.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.725+0000 7f6d83e00640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d6406bc50 0x7f6d6406e110 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400a730 tx=0x7f6d74008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:55.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.729+0000 7f6d7be00640 1 -- 172.21.15.102:0/4225777412 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6d700315f0 con 0x7f6d840c8b20 2024-03-18T16:25:55.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.741+0000 7f6d7be00640 1 -- 172.21.15.102:0/4225777412 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6d7003f070 con 0x7f6d840c8b20 2024-03-18T16:25:55.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:55.987+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d841476a0 con 0x7f6d840c8b20 2024-03-18T16:25:56.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.013+0000 7f6d7be00640 1 -- 172.21.15.102:0/4225777412 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f6d7007b7d0 con 0x7f6d840c8b20 2024-03-18T16:25:56.015 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:56.015 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:56.016 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d6406bc50 msgr2=0x7f6d6406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d6406bc50 0x7f6d6406e110 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400a730 tx=0x7f6d74008040 comp rx=0 tx=0).stop 2024-03-18T16:25:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d840c8b20 msgr2=0x7f6d84145430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d840c8b20 0x7f6d84145430 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6d7002b930 tx=0x7f6d700046c0 comp rx=0 tx=0).stop 2024-03-18T16:25:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 shutdown_connections 2024-03-18T16:25:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d6406bc50 0x7f6d6406e110 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6d84158760 0x7f6d84147940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d840c8b20 0x7f6d84145430 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 --2- 172.21.15.102:0/4225777412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d840c8740 0x7f6d840cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 >> 172.21.15.102:0/4225777412 conn(0x7f6d840700d0 msgr2=0x7f6d8406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 shutdown_connections 2024-03-18T16:25:56.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:56.017+0000 7f6d8b200640 1 -- 172.21.15.102:0/4225777412 wait complete. 2024-03-18T16:25:56.649 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:56.649 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:25:57.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:25:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.821+0000 7f0efa800640 1 Processor -- start 2024-03-18T16:25:57.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.821+0000 7f0efa800640 1 -- start start 2024-03-18T16:25:57.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.823+0000 7f0efa800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0ef415c3a0 0x7f0ef4163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.823+0000 7f0efa800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef40c72d0 con 0x7f0ef4141650 2024-03-18T16:25:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.823+0000 7f0efa800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef4158840 con 0x7f0ef4158420 2024-03-18T16:25:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.823+0000 7f0efa800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0ef4159370 con 0x7f0ef415c3a0 2024-03-18T16:25:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.823+0000 7f0ef8a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0ef4158420 0x7f0ef40c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:43864/0 (socket says 172.21.15.102:43864) 2024-03-18T16:25:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.823+0000 7f0ef8a00640 1 -- 172.21.15.102:0/1836320801 learned_addr learned my addr 172.21.15.102:0/1836320801 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:25:57.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.826+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086454113 0 0) 0x7f0ef4158840 con 0x7f0ef4158420 2024-03-18T16:25:57.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.826+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0edc003520 con 0x7f0ef4158420 2024-03-18T16:25:57.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.828+0000 7f0ef9e00640 1 --2- 172.21.15.102:0/1836320801 >> v2:172.21.15.102:3300/0 conn(0x7f0ef415c3a0 0x7f0ef4163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4028205032 0 0) 0x7f0edc003520 con 0x7f0ef4158420 2024-03-18T16:25:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 >> v2:172.21.15.102:3300/0 conn(0x7f0ef415c3a0 msgr2=0x7f0ef4163cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 --2- 172.21.15.102:0/1836320801 >> v2:172.21.15.102:3300/0 conn(0x7f0ef415c3a0 0x7f0ef4163cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 >> v1:172.21.15.102:6789/0 conn(0x7f0ef4141650 legacy=0x7f0ef40c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef4158ad0 con 0x7f0ef4158420 2024-03-18T16:25:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f0ee0001670 con 0x7f0ef4158420 2024-03-18T16:25:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 >> v1:172.21.15.154:6789/0 conn(0x7f0ef4158420 legacy=0x7f0ef40c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:57.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0edc003a00 0x7f0edc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:57.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0edc004800 0x7f0edc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:57.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0edc005210 0x7f0edc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:57.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4159370 con 0x7f0edc004800 2024-03-18T16:25:57.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0edc00c160 con 0x7f0edc003a00 2024-03-18T16:25:57.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.832+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0edc00c360 con 0x7f0edc005210 2024-03-18T16:25:57.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.833+0000 7f0ef9e00640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0edc005210 0x7f0edc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:57.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.836+0000 7f0ef9400640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0edc003a00 0x7f0edc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.837+0000 7f0ef9e00640 1 -- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0edc003a00 msgr2=0x7f0edc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.837+0000 7f0ef9e00640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0edc003a00 0x7f0edc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.837+0000 7f0ef9e00640 1 -- 172.21.15.102:0/1836320801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0edc004800 msgr2=0x7f0edc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:25:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.837+0000 7f0ef9e00640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0edc004800 0x7f0edc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:57.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.837+0000 7f0ef9e00640 1 -- 172.21.15.102:0/1836320801 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0edc003520 con 0x7f0edc005210 2024-03-18T16:25:57.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.839+0000 7f0ef9e00640 1 -- 172.21.15.102:0/1836320801 reap_dead start 2024-03-18T16:25:57.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.842+0000 7f0ef9e00640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0edc005210 0x7f0edc00b9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0eec007360 tx=0x7f0edc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2d2a6c7fc41a15 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:57.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.847+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec009be0 con 0x7f0edc005210 2024-03-18T16:25:57.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.847+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec02ccf0 con 0x7f0edc005210 2024-03-18T16:25:57.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.847+0000 7f0ef3e00640 1 -- 172.21.15.102:0/1836320801 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec003bd0 con 0x7f0edc005210 2024-03-18T16:25:57.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.848+0000 7f0efa800640 1 -- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0edc005210 msgr2=0x7f0edc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:57.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.848+0000 7f0efa800640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0edc005210 0x7f0edc00b9c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0eec007360 tx=0x7f0edc005830 comp rx=0 tx=0).stop 2024-03-18T16:25:57.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.848+0000 7f0efa800640 1 -- 172.21.15.102:0/1836320801 shutdown_connections 2024-03-18T16:25:57.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.848+0000 7f0efa800640 1 --2- 172.21.15.102:0/1836320801 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0edc005210 0x7f0edc00b9c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:57.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.848+0000 7f0efa800640 1 -- 172.21.15.102:0/1836320801 >> 172.21.15.102:0/1836320801 conn(0x7f0ef40cb890 msgr2=0x7f0ef40cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:57.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.848+0000 7f0efa800640 1 -- 172.21.15.102:0/1836320801 shutdown_connections 2024-03-18T16:25:57.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.848+0000 7f0efa800640 1 -- 172.21.15.102:0/1836320801 wait complete. 2024-03-18T16:25:57.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.851+0000 7f0efa800640 1 Processor -- start 2024-03-18T16:25:57.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.851+0000 7f0efa800640 1 -- start start 2024-03-18T16:25:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.852+0000 7f0efa800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0edc005210 0x7f0ef415b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.852+0000 7f0efa800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ef41547b0 0x7f0ef4154bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.852+0000 7f0efa800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef41538d0 0x7f0ef4153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.852+0000 7f0efa800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef40c63c0 con 0x7f0ef41547b0 2024-03-18T16:25:57.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.852+0000 7f0efa800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef4159090 con 0x7f0edc005210 2024-03-18T16:25:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.852+0000 7f0efa800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef40c66a0 con 0x7f0ef41538d0 2024-03-18T16:25:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.853+0000 7f0ef9e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef41538d0 0x7f0ef4153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.853+0000 7f0ef9e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef41538d0 0x7f0ef4153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:39250/0 (socket says 172.21.15.102:39250) 2024-03-18T16:25:57.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.853+0000 7f0ef9e00640 1 -- 172.21.15.102:0/485826364 learned_addr learned my addr 172.21.15.102:0/485826364 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:25:57.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.855+0000 7f0ef9400640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0edc005210 0x7f0ef415b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:57.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.856+0000 7f0ef8a00640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ef41547b0 0x7f0ef4154bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:57.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.860+0000 7f0ef9e00640 1 -- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0edc005210 msgr2=0x7f0ef415b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:57.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.860+0000 7f0ef9e00640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0edc005210 0x7f0ef415b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:57.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.860+0000 7f0ef9e00640 1 -- 172.21.15.102:0/485826364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ef41547b0 msgr2=0x7f0ef4154bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:57.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.860+0000 7f0ef9e00640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ef41547b0 0x7f0ef4154bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:57.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.860+0000 7f0ef9e00640 1 -- 172.21.15.102:0/485826364 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef4159580 con 0x7f0ef41538d0 2024-03-18T16:25:57.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.866+0000 7f0ef9e00640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef41538d0 0x7f0ef4153d50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0eec02c760 tx=0x7f0eec004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:57.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.915+0000 7f0ef1600640 1 -- 172.21.15.102:0/485826364 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec042070 con 0x7f0ef41538d0 2024-03-18T16:25:57.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.916+0000 7f0ef1600640 1 -- 172.21.15.102:0/485826364 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec004500 con 0x7f0ef41538d0 2024-03-18T16:25:57.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.916+0000 7f0ef1600640 1 -- 172.21.15.102:0/485826364 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec034440 con 0x7f0ef41538d0 2024-03-18T16:25:57.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.916+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef41f1010 con 0x7f0ef41538d0 2024-03-18T16:25:57.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.916+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef4152df0 con 0x7f0ef41538d0 2024-03-18T16:25:57.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.917+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ef4074fd0 con 0x7f0ef41538d0 2024-03-18T16:25:57.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.950+0000 7f0ef1600640 1 -- 172.21.15.102:0/485826364 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0eec005d10 con 0x7f0ef41538d0 2024-03-18T16:25:57.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.951+0000 7f0ef1600640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ed406bbd0 0x7f0ed406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:25:57.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.952+0000 7f0ef9400640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ed406bbd0 0x7f0ed406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:25:57.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:57.953+0000 7f0ef9400640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ed406bbd0 0x7f0ed406e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8007fd0 tx=0x7f0ee8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:25:58.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.203+0000 7f0ef1600640 1 -- 172.21.15.102:0/485826364 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f0eec0786b0 con 0x7f0ef41538d0 2024-03-18T16:25:58.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.214+0000 7f0ef1600640 1 -- 172.21.15.102:0/485826364 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0eec03a070 con 0x7f0ef41538d0 2024-03-18T16:25:58.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.457+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ef4072260 con 0x7f0ef41538d0 2024-03-18T16:25:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.472+0000 7f0ef1600640 1 -- 172.21.15.102:0/485826364 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f0eec07a610 con 0x7f0ef41538d0 2024-03-18T16:25:58.474 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:25:58.474 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:25:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:25:58.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ed406bbd0 msgr2=0x7f0ed406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ed406bbd0 0x7f0ed406e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8007fd0 tx=0x7f0ee8009040 comp rx=0 tx=0).stop 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef41538d0 msgr2=0x7f0ef4153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef41538d0 0x7f0ef4153d50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0eec02c760 tx=0x7f0eec004a40 comp rx=0 tx=0).stop 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 shutdown_connections 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ed406bbd0 0x7f0ed406e090 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ef41538d0 0x7f0ef4153d50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ef41547b0 0x7f0ef4154bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 --2- 172.21.15.102:0/485826364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0edc005210 0x7f0ef415b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.474+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 >> 172.21.15.102:0/485826364 conn(0x7f0ef40cb890 msgr2=0x7f0ef40cc130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.475+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 shutdown_connections 2024-03-18T16:25:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:25:58.475+0000 7f0efa800640 1 -- 172.21.15.102:0/485826364 wait complete. 2024-03-18T16:25:59.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:25:59.099 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:00.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:00.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.273+0000 7f9b2a000640 1 Processor -- start 2024-03-18T16:26:00.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.273+0000 7f9b2a000640 1 -- start start 2024-03-18T16:26:00.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.274+0000 7f9b2a000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9b2415ad60 0x7f9b240c5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:00.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.274+0000 7f9b2a000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b2406bf60 con 0x7f9b2406b470 2024-03-18T16:26:00.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.274+0000 7f9b2a000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b240c94b0 con 0x7f9b2406de50 2024-03-18T16:26:00.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.274+0000 7f9b2a000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9b240ca0a0 con 0x7f9b2415ad60 2024-03-18T16:26:00.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.275+0000 7f9b23e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f9b2406de50 0x7f9b24154150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:43874/0 (socket says 172.21.15.102:43874) 2024-03-18T16:26:00.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.275+0000 7f9b23e00640 1 -- 172.21.15.102:0/4128939134 learned_addr learned my addr 172.21.15.102:0/4128939134 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:00.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.275+0000 7f9b29600640 1 --2- 172.21.15.102:0/4128939134 >> v2:172.21.15.102:3300/0 conn(0x7f9b2415ad60 0x7f9b240c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:00.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.277+0000 7f9b23400640 1 -- 172.21.15.102:0/4128939134 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1279083122 0 0) 0x7f9b240c94b0 con 0x7f9b2406de50 2024-03-18T16:26:00.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.277+0000 7f9b23400640 1 -- 172.21.15.102:0/4128939134 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b0c003520 con 0x7f9b2406de50 2024-03-18T16:26:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.280+0000 7f9b29600640 1 -- 172.21.15.102:0/4128939134 >> v1:172.21.15.154:6789/0 conn(0x7f9b2406de50 legacy=0x7f9b24154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.280+0000 7f9b29600640 1 -- 172.21.15.102:0/4128939134 >> v1:172.21.15.102:6789/0 conn(0x7f9b2406b470 legacy=0x7f9b2406b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.280+0000 7f9b29600640 1 -- 172.21.15.102:0/4128939134 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b240c9770 con 0x7f9b2415ad60 2024-03-18T16:26:00.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.287+0000 7f9b29600640 1 --2- 172.21.15.102:0/4128939134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b2415ad60 0x7f9b240c5190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009d90 tx=0x7f9b1c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d89ef945f7900d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.293+0000 7f9b23400640 1 -- 172.21.15.102:0/4128939134 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c039070 con 0x7f9b2415ad60 2024-03-18T16:26:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.293+0000 7f9b23400640 1 -- 172.21.15.102:0/4128939134 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b1c02ccf0 con 0x7f9b2415ad60 2024-03-18T16:26:00.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.293+0000 7f9b23400640 1 -- 172.21.15.102:0/4128939134 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c034660 con 0x7f9b2415ad60 2024-03-18T16:26:00.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.294+0000 7f9b2a000640 1 -- 172.21.15.102:0/4128939134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b2415ad60 msgr2=0x7f9b240c5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:00.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.294+0000 7f9b2a000640 1 --2- 172.21.15.102:0/4128939134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b2415ad60 0x7f9b240c5190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c009d90 tx=0x7f9b1c02bf40 comp rx=0 tx=0).stop 2024-03-18T16:26:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.294+0000 7f9b2a000640 1 -- 172.21.15.102:0/4128939134 shutdown_connections 2024-03-18T16:26:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.294+0000 7f9b2a000640 1 --2- 172.21.15.102:0/4128939134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b2415ad60 0x7f9b240c5190 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.294+0000 7f9b2a000640 1 -- 172.21.15.102:0/4128939134 >> 172.21.15.102:0/4128939134 conn(0x7f9b240727e0 msgr2=0x7f9b24072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.294+0000 7f9b2a000640 1 -- 172.21.15.102:0/4128939134 shutdown_connections 2024-03-18T16:26:00.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.294+0000 7f9b2a000640 1 -- 172.21.15.102:0/4128939134 wait complete. 2024-03-18T16:26:00.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.296+0000 7f9b2a000640 1 Processor -- start 2024-03-18T16:26:00.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.296+0000 7f9b2a000640 1 -- start start 2024-03-18T16:26:00.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.297+0000 7f9b2a000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b2406b470 0x7f9b2415e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:00.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.297+0000 7f9b2a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b2406de50 0x7f9b240c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:00.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.297+0000 7f9b2a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b2415ad60 0x7f9b241e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:00.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.297+0000 7f9b2a000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9b24154860 con 0x7f9b2406de50 2024-03-18T16:26:00.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.297+0000 7f9b2a000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9b240c9d30 con 0x7f9b2406b470 2024-03-18T16:26:00.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.297+0000 7f9b2a000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9b241549e0 con 0x7f9b2415ad60 2024-03-18T16:26:00.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.298+0000 7f9b28c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b2406b470 0x7f9b2415e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:00.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.298+0000 7f9b29600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b2415ad60 0x7f9b241e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:00.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.298+0000 7f9b23e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b2406de50 0x7f9b240c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:00.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.298+0000 7f9b28c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b2406b470 0x7f9b2415e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42296/0 (socket says 172.21.15.102:42296) 2024-03-18T16:26:00.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.298+0000 7f9b28c00640 1 -- 172.21.15.102:0/3441315934 learned_addr learned my addr 172.21.15.102:0/3441315934 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:00.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.303+0000 7f9b28c00640 1 -- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b2415ad60 msgr2=0x7f9b241e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:00.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.303+0000 7f9b28c00640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b2415ad60 0x7f9b241e3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:00.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.303+0000 7f9b28c00640 1 -- 172.21.15.102:0/3441315934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b2406de50 msgr2=0x7f9b240c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:00.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.303+0000 7f9b28c00640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b2406de50 0x7f9b240c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:00.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.303+0000 7f9b28c00640 1 -- 172.21.15.102:0/3441315934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b241f0600 con 0x7f9b2406b470 2024-03-18T16:26:00.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.303+0000 7f9b29600640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b2415ad60 0x7f9b241e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:26:00.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.306+0000 7f9b28c00640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b2406b470 0x7f9b2415e1b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800c400 tx=0x7f9b1800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:00.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.310+0000 7f9b20c00640 1 -- 172.21.15.102:0/3441315934 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b180052d0 con 0x7f9b2406b470 2024-03-18T16:26:00.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.310+0000 7f9b20c00640 1 -- 172.21.15.102:0/3441315934 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b18005490 con 0x7f9b2406b470 2024-03-18T16:26:00.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.310+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b241f08e0 con 0x7f9b2406b470 2024-03-18T16:26:00.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.311+0000 7f9b20c00640 1 -- 172.21.15.102:0/3441315934 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b18007260 con 0x7f9b2406b470 2024-03-18T16:26:00.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.311+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b241f0ea0 con 0x7f9b2406b470 2024-03-18T16:26:00.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.312+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b241f10a0 con 0x7f9b2406b470 2024-03-18T16:26:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.572+0000 7f9b20c00640 1 -- 172.21.15.102:0/3441315934 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9b18007400 con 0x7f9b2406b470 2024-03-18T16:26:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.574+0000 7f9b20c00640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b0406bc20 0x7f9b0406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.575+0000 7f9b20c00640 1 -- 172.21.15.102:0/3441315934 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9b1808aa60 con 0x7f9b2406b470 2024-03-18T16:26:00.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.575+0000 7f9b23e00640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b0406bc20 0x7f9b0406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:00.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.576+0000 7f9b23e00640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b0406bc20 0x7f9b0406e0e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9b240ca750 tx=0x7f9b10007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:00.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.585+0000 7f9b20c00640 1 -- 172.21.15.102:0/3441315934 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9b241f10a0 con 0x7f9b2406b470 2024-03-18T16:26:00.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.840+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b241e38f0 con 0x7f9b2406b470 2024-03-18T16:26:00.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.854+0000 7f9b20c00640 1 -- 172.21.15.102:0/3441315934 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f9b18058c10 con 0x7f9b2406b470 2024-03-18T16:26:00.856 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:00.856 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:00.857 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:00.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b0406bc20 msgr2=0x7f9b0406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:00.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b0406bc20 0x7f9b0406e0e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9b240ca750 tx=0x7f9b10007d30 comp rx=0 tx=0).stop 2024-03-18T16:26:00.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b2406b470 msgr2=0x7f9b2415e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:00.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b2406b470 0x7f9b2415e1b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9b1800c400 tx=0x7f9b1800c8c0 comp rx=0 tx=0).stop 2024-03-18T16:26:00.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 shutdown_connections 2024-03-18T16:26:00.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b0406bc20 0x7f9b0406e0e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:00.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b2415ad60 0x7f9b241e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:00.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b2406de50 0x7f9b240c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:00.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 --2- 172.21.15.102:0/3441315934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b2406b470 0x7f9b2415e1b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:00.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 >> 172.21.15.102:0/3441315934 conn(0x7f9b240727e0 msgr2=0x7f9b240c6160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:00.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 shutdown_connections 2024-03-18T16:26:00.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:00.857+0000 7f9b2a000640 1 -- 172.21.15.102:0/3441315934 wait complete. 2024-03-18T16:26:01.480 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:01.480 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:02.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:02.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.658+0000 7f6ed2000640 1 Processor -- start 2024-03-18T16:26:02.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.658+0000 7f6ed2000640 1 -- start start 2024-03-18T16:26:02.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.659+0000 7f6ed2000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15baa0 0x7f6ecc15bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:02.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.659+0000 7f6ed2000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ecc0ca500 con 0x7f6ecc15a6f0 2024-03-18T16:26:02.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.659+0000 7f6ed2000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ecc0ca7a0 con 0x7f6ecc15a310 2024-03-18T16:26:02.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.659+0000 7f6ed2000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6ecc0cd410 con 0x7f6ecc15baa0 2024-03-18T16:26:02.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.660+0000 7f6ecbe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6ecc15a6f0 0x7f6ecc15ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56210/0 (socket says 172.21.15.102:56210) 2024-03-18T16:26:02.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.660+0000 7f6ecbe00640 1 -- 172.21.15.102:0/437287171 learned_addr learned my addr 172.21.15.102:0/437287171 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:02.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.660+0000 7f6ed1600640 1 --2- 172.21.15.102:0/437287171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15baa0 0x7f6ecc15bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:02.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.662+0000 7f6ecb400640 1 -- 172.21.15.102:0/437287171 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3136215163 0 0) 0x7f6ecc0ca7a0 con 0x7f6ecc15a310 2024-03-18T16:26:02.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.662+0000 7f6ecb400640 1 -- 172.21.15.102:0/437287171 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6eb4003520 con 0x7f6ecc15a310 2024-03-18T16:26:02.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.662+0000 7f6ed1600640 1 -- 172.21.15.102:0/437287171 >> v1:172.21.15.154:6789/0 conn(0x7f6ecc15a310 legacy=0x7f6ecc06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:02.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.662+0000 7f6ed1600640 1 -- 172.21.15.102:0/437287171 >> v1:172.21.15.102:6789/0 conn(0x7f6ecc15a6f0 legacy=0x7f6ecc15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:02.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.662+0000 7f6ed1600640 1 -- 172.21.15.102:0/437287171 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ecc0caa80 con 0x7f6ecc15baa0 2024-03-18T16:26:02.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.665+0000 7f6ed1600640 1 --2- 172.21.15.102:0/437287171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15baa0 0x7f6ecc15bf50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009dc0 tx=0x7f6ec402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bea1c80764227966 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:02.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.669+0000 7f6ecb400640 1 -- 172.21.15.102:0/437287171 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4039070 con 0x7f6ecc15baa0 2024-03-18T16:26:02.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.669+0000 7f6ecb400640 1 -- 172.21.15.102:0/437287171 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec402ccd0 con 0x7f6ecc15baa0 2024-03-18T16:26:02.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.669+0000 7f6ecb400640 1 -- 172.21.15.102:0/437287171 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4034660 con 0x7f6ecc15baa0 2024-03-18T16:26:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.670+0000 7f6ed2000640 1 -- 172.21.15.102:0/437287171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15baa0 msgr2=0x7f6ecc15bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.670+0000 7f6ed2000640 1 --2- 172.21.15.102:0/437287171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15baa0 0x7f6ecc15bf50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4009dc0 tx=0x7f6ec402bf20 comp rx=0 tx=0).stop 2024-03-18T16:26:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.670+0000 7f6ed2000640 1 -- 172.21.15.102:0/437287171 shutdown_connections 2024-03-18T16:26:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.670+0000 7f6ed2000640 1 --2- 172.21.15.102:0/437287171 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15baa0 0x7f6ecc15bf50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.670+0000 7f6ed2000640 1 -- 172.21.15.102:0/437287171 >> 172.21.15.102:0/437287171 conn(0x7f6ecc08c1a0 msgr2=0x7f6ecc066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.670+0000 7f6ed2000640 1 -- 172.21.15.102:0/437287171 shutdown_connections 2024-03-18T16:26:02.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.670+0000 7f6ed2000640 1 -- 172.21.15.102:0/437287171 wait complete. 2024-03-18T16:26:02.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.672+0000 7f6ed2000640 1 Processor -- start 2024-03-18T16:26:02.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.673+0000 7f6ed2000640 1 -- start start 2024-03-18T16:26:02.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.674+0000 7f6ed2000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15a310 0x7f6ecc0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:02.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.674+0000 7f6ed2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ecc15a6f0 0x7f6ecc0c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:02.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.674+0000 7f6ed2000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ecc15baa0 0x7f6ecc0ae630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:02.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.674+0000 7f6ed2000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6ecc0cd130 con 0x7f6ecc15a6f0 2024-03-18T16:26:02.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.674+0000 7f6ed2000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ecc15c490 con 0x7f6ecc15baa0 2024-03-18T16:26:02.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.674+0000 7f6ed2000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6ecc1548c0 con 0x7f6ecc15a310 2024-03-18T16:26:02.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.675+0000 7f6ed0c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15a310 0x7f6ecc0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:02.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.675+0000 7f6ed1600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ecc15baa0 0x7f6ecc0ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:02.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.675+0000 7f6ed0c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15a310 0x7f6ecc0c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:54508/0 (socket says 172.21.15.102:54508) 2024-03-18T16:26:02.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.675+0000 7f6ed0c00640 1 -- 172.21.15.102:0/2694120687 learned_addr learned my addr 172.21.15.102:0/2694120687 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:02.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.675+0000 7f6ecbe00640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ecc15a6f0 0x7f6ecc0c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.678+0000 7f6ed0c00640 1 -- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ecc15baa0 msgr2=0x7f6ecc0ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.678+0000 7f6ed0c00640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ecc15baa0 0x7f6ecc0ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:02.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.678+0000 7f6ed0c00640 1 -- 172.21.15.102:0/2694120687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ecc15a6f0 msgr2=0x7f6ecc0c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:02.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.678+0000 7f6ed0c00640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ecc15a6f0 0x7f6ecc0c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:02.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.678+0000 7f6ed0c00640 1 -- 172.21.15.102:0/2694120687 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ecc0abec0 con 0x7f6ecc15a310 2024-03-18T16:26:02.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.681+0000 7f6ed0c00640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15a310 0x7f6ecc0c44b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ec000c430 tx=0x7f6ec000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:02.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.684+0000 7f6ec8c00640 1 -- 172.21.15.102:0/2694120687 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0017070 con 0x7f6ecc15a310 2024-03-18T16:26:02.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.684+0000 7f6ec8c00640 1 -- 172.21.15.102:0/2694120687 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec00052f0 con 0x7f6ecc15a310 2024-03-18T16:26:02.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.684+0000 7f6ec8c00640 1 -- 172.21.15.102:0/2694120687 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0003830 con 0x7f6ecc15a310 2024-03-18T16:26:02.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.684+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ecc153510 con 0x7f6ecc15a310 2024-03-18T16:26:02.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.685+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ecc0c64c0 con 0x7f6ecc15a310 2024-03-18T16:26:02.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.686+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ecc074fd0 con 0x7f6ecc15a310 2024-03-18T16:26:02.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.705+0000 7f6ec8c00640 1 -- 172.21.15.102:0/2694120687 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6ec001a070 con 0x7f6ecc15a310 2024-03-18T16:26:02.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.707+0000 7f6ec8c00640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6eac06bc50 0x7f6eac06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.707+0000 7f6ecbe00640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6eac06bc50 0x7f6eac06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:02.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.708+0000 7f6ec8c00640 1 -- 172.21.15.102:0/2694120687 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6ec0056710 con 0x7f6ecc15a310 2024-03-18T16:26:02.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.708+0000 7f6ecbe00640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6eac06bc50 0x7f6eac06e110 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0c9bb0 tx=0x7f6eb8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:02.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:02.960+0000 7f6ec8c00640 1 -- 172.21.15.102:0/2694120687 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6ec0021020 con 0x7f6ecc15a310 2024-03-18T16:26:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.268+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ecc0caed0 con 0x7f6ecc15a310 2024-03-18T16:26:03.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.282+0000 7f6ec8c00640 1 -- 172.21.15.102:0/2694120687 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f6ecc0abec0 con 0x7f6ecc15a310 2024-03-18T16:26:03.284 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:03.284 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:03.284 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:03.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6eac06bc50 msgr2=0x7f6eac06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:03.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6eac06bc50 0x7f6eac06e110 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0c9bb0 tx=0x7f6eb8009040 comp rx=0 tx=0).stop 2024-03-18T16:26:03.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15a310 msgr2=0x7f6ecc0c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:03.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15a310 0x7f6ecc0c44b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ec000c430 tx=0x7f6ec000c8f0 comp rx=0 tx=0).stop 2024-03-18T16:26:03.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 shutdown_connections 2024-03-18T16:26:03.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6eac06bc50 0x7f6eac06e110 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:03.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ecc15baa0 0x7f6ecc0ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:03.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ecc15a6f0 0x7f6ecc0c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:03.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 --2- 172.21.15.102:0/2694120687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6ecc15a310 0x7f6ecc0c44b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:03.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 >> 172.21.15.102:0/2694120687 conn(0x7f6ecc08c1a0 msgr2=0x7f6ecc14eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:03.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 shutdown_connections 2024-03-18T16:26:03.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:03.285+0000 7f6ed2000640 1 -- 172.21.15.102:0/2694120687 wait complete. 2024-03-18T16:26:03.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:03.910 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:04.910 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:05.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.080+0000 7f7bca200640 1 Processor -- start 2024-03-18T16:26:05.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.081+0000 7f7bca200640 1 -- start start 2024-03-18T16:26:05.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.082+0000 7f7bca200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7bc40d0250 0x7f7bc415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:05.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.082+0000 7f7bca200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cc2f0 0x7f7bc40cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:05.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.082+0000 7f7bca200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bc40cabd0 con 0x7f7bc40cfe70 2024-03-18T16:26:05.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.082+0000 7f7bca200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7bc40caea0 con 0x7f7bc40d0250 2024-03-18T16:26:05.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.082+0000 7f7bca200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7bc40cb180 con 0x7f7bc40cc2f0 2024-03-18T16:26:05.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.083+0000 7f7bc3e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7bc40d0250 0x7f7bc415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:05.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.083+0000 7f7bc9800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cc2f0 0x7f7bc40cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:05.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.083+0000 7f7bc3e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7bc40d0250 0x7f7bc415ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34456/0 (socket says 172.21.15.102:34456) 2024-03-18T16:26:05.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.083+0000 7f7bc3e00640 1 -- 172.21.15.102:0/38912341 learned_addr learned my addr 172.21.15.102:0/38912341 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:05.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.085+0000 7f7bc3e00640 1 -- 172.21.15.102:0/38912341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cc2f0 msgr2=0x7f7bc40cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:05.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.085+0000 7f7bc3e00640 1 --2- 172.21.15.102:0/38912341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cc2f0 0x7f7bc40cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:05.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.085+0000 7f7bc3e00640 1 -- 172.21.15.102:0/38912341 >> v1:172.21.15.154:6789/0 conn(0x7f7bc40cfe70 legacy=0x7f7bc40c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:05.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.085+0000 7f7bc3e00640 1 -- 172.21.15.102:0/38912341 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc40cb460 con 0x7f7bc40d0250 2024-03-18T16:26:05.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.088+0000 7f7bc3e00640 1 --2- 172.21.15.102:0/38912341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 0x7f7bc415ae40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0009870 tx=0x7f7bb002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ebee7e8f3dc9bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:05.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bc3400640 1 -- 172.21.15.102:0/38912341 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0034070 con 0x7f7bc40d0250 2024-03-18T16:26:05.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bc3400640 1 -- 172.21.15.102:0/38912341 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bb0030840 con 0x7f7bc40d0250 2024-03-18T16:26:05.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bc3400640 1 -- 172.21.15.102:0/38912341 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0030c10 con 0x7f7bc40d0250 2024-03-18T16:26:05.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bca200640 1 -- 172.21.15.102:0/38912341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 msgr2=0x7f7bc415ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:05.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bca200640 1 --2- 172.21.15.102:0/38912341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 0x7f7bc415ae40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0009870 tx=0x7f7bb002b980 comp rx=0 tx=0).stop 2024-03-18T16:26:05.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bca200640 1 -- 172.21.15.102:0/38912341 shutdown_connections 2024-03-18T16:26:05.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bca200640 1 --2- 172.21.15.102:0/38912341 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cc2f0 0x7f7bc40cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:05.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bca200640 1 --2- 172.21.15.102:0/38912341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 0x7f7bc415ae40 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:05.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bca200640 1 -- 172.21.15.102:0/38912341 >> 172.21.15.102:0/38912341 conn(0x7f7bc408c3e0 msgr2=0x7f7bc406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:05.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bca200640 1 -- 172.21.15.102:0/38912341 shutdown_connections 2024-03-18T16:26:05.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.092+0000 7f7bca200640 1 -- 172.21.15.102:0/38912341 wait complete. 2024-03-18T16:26:05.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.094+0000 7f7bca200640 1 Processor -- start 2024-03-18T16:26:05.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.094+0000 7f7bca200640 1 -- start start 2024-03-18T16:26:05.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.095+0000 7f7bca200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc40cc2f0 0x7f7bc41e0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:05.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.095+0000 7f7bca200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cfe70 0x7f7bc41e1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:05.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.095+0000 7f7bca200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 0x7f7bc4153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:05.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.095+0000 7f7bca200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc40cbd00 con 0x7f7bc40cc2f0 2024-03-18T16:26:05.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.095+0000 7f7bca200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc40cba20 con 0x7f7bc40d0250 2024-03-18T16:26:05.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.095+0000 7f7bca200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7bc40ccc50 con 0x7f7bc40cfe70 2024-03-18T16:26:05.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.096+0000 7f7bc9800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 0x7f7bc4153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:05.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.096+0000 7f7bc9800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 0x7f7bc4153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34462/0 (socket says 172.21.15.102:34462) 2024-03-18T16:26:05.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.096+0000 7f7bc3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cfe70 0x7f7bc41e1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:05.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.096+0000 7f7bc9800640 1 -- 172.21.15.102:0/2487258730 learned_addr learned my addr 172.21.15.102:0/2487258730 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:05.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.096+0000 7f7bc8e00640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc40cc2f0 0x7f7bc41e0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:05.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.099+0000 7f7bc9800640 1 -- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cfe70 msgr2=0x7f7bc41e1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:05.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.099+0000 7f7bc9800640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cfe70 0x7f7bc41e1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:05.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.099+0000 7f7bc9800640 1 -- 172.21.15.102:0/2487258730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc40cc2f0 msgr2=0x7f7bc41e0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:05.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.099+0000 7f7bc9800640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc40cc2f0 0x7f7bc41e0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:05.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.099+0000 7f7bc9800640 1 -- 172.21.15.102:0/2487258730 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc40c8100 con 0x7f7bc40d0250 2024-03-18T16:26:05.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.102+0000 7f7bc9800640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 0x7f7bc4153b90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00efc0 tx=0x7f7bbc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:05.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.105+0000 7f7bc0c00640 1 -- 172.21.15.102:0/2487258730 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc018020 con 0x7f7bc40d0250 2024-03-18T16:26:05.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.105+0000 7f7bc0c00640 1 -- 172.21.15.102:0/2487258730 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bbc014480 con 0x7f7bc40d0250 2024-03-18T16:26:05.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.105+0000 7f7bc0c00640 1 -- 172.21.15.102:0/2487258730 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc014870 con 0x7f7bc40d0250 2024-03-18T16:26:05.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.105+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc4159ed0 con 0x7f7bc40d0250 2024-03-18T16:26:05.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.105+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bc41f09f0 con 0x7f7bc40d0250 2024-03-18T16:26:05.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.107+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bc4074fd0 con 0x7f7bc40d0250 2024-03-18T16:26:05.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.378+0000 7f7bc0c00640 1 -- 172.21.15.102:0/2487258730 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7bbc005d10 con 0x7f7bc40d0250 2024-03-18T16:26:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.381+0000 7f7bc0c00640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ba406bbd0 0x7f7ba406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:05.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.381+0000 7f7bc8e00640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ba406bbd0 0x7f7ba406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:05.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.381+0000 7f7bc0c00640 1 -- 172.21.15.102:0/2487258730 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7bbc058220 con 0x7f7bc40d0250 2024-03-18T16:26:05.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.382+0000 7f7bc8e00640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ba406bbd0 0x7f7ba406e090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80017f0 tx=0x7f7bb8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:05.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.393+0000 7f7bc0c00640 1 -- 172.21.15.102:0/2487258730 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7bbc022030 con 0x7f7bc40d0250 2024-03-18T16:26:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.642+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bc40c8ba0 con 0x7f7bc40d0250 2024-03-18T16:26:05.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.679+0000 7f7bc0c00640 1 -- 172.21.15.102:0/2487258730 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7bbc010040 con 0x7f7bc40d0250 2024-03-18T16:26:05.682 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:05.682 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:05.682 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:05.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.681+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ba406bbd0 msgr2=0x7f7ba406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:05.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.681+0000 7f7bca200640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ba406bbd0 0x7f7ba406e090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80017f0 tx=0x7f7bb8009040 comp rx=0 tx=0).stop 2024-03-18T16:26:05.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.681+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 msgr2=0x7f7bc4153b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:05.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.681+0000 7f7bca200640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 0x7f7bc4153b90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00efc0 tx=0x7f7bbc00c490 comp rx=0 tx=0).stop 2024-03-18T16:26:05.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.681+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 shutdown_connections 2024-03-18T16:26:05.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.682+0000 7f7bca200640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ba406bbd0 0x7f7ba406e090 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:05.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.682+0000 7f7bca200640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bc40d0250 0x7f7bc4153b90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:05.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.682+0000 7f7bca200640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bc40cfe70 0x7f7bc41e1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:05.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.682+0000 7f7bca200640 1 --2- 172.21.15.102:0/2487258730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bc40cc2f0 0x7f7bc41e0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:05.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.682+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 >> 172.21.15.102:0/2487258730 conn(0x7f7bc408c3e0 msgr2=0x7f7bc40cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:05.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.682+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 shutdown_connections 2024-03-18T16:26:05.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:05.682+0000 7f7bca200640 1 -- 172.21.15.102:0/2487258730 wait complete. 2024-03-18T16:26:06.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:06.310 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:07.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:07.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.484+0000 7fc17d600640 1 Processor -- start 2024-03-18T16:26:07.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.485+0000 7fc17d600640 1 -- start start 2024-03-18T16:26:07.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.486+0000 7fc17d600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc178141650 0x7fc178158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:07.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.486+0000 7fc17d600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc17815c3a0 0x7fc1780c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:07.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.486+0000 7fc17d600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc178158ad0 con 0x7fc1780c4fa0 2024-03-18T16:26:07.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.486+0000 7fc17d600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc178159370 con 0x7fc17815c3a0 2024-03-18T16:26:07.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.486+0000 7fc17d600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc178158db0 con 0x7fc178141650 2024-03-18T16:26:07.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.487+0000 7fc177400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc17815c3a0 0x7fc1780c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:07.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.487+0000 7fc177e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc178141650 0x7fc178158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:07.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.487+0000 7fc17cc00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc1780c4fa0 0x7fc17806f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56220/0 (socket says 172.21.15.102:56220) 2024-03-18T16:26:07.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.487+0000 7fc17cc00640 1 -- 172.21.15.102:0/3866295381 learned_addr learned my addr 172.21.15.102:0/3866295381 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:07.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.489+0000 7fc177400640 1 -- 172.21.15.102:0/3866295381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc178141650 msgr2=0x7fc178158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:07.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.489+0000 7fc177400640 1 --2- 172.21.15.102:0/3866295381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc178141650 0x7fc178158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:07.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.489+0000 7fc177400640 1 -- 172.21.15.102:0/3866295381 >> v1:172.21.15.102:6789/0 conn(0x7fc1780c4fa0 legacy=0x7fc17806f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:07.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.489+0000 7fc177400640 1 -- 172.21.15.102:0/3866295381 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc178159090 con 0x7fc17815c3a0 2024-03-18T16:26:07.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.492+0000 7fc177400640 1 --2- 172.21.15.102:0/3866295381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc17815c3a0 0x7fc1780c6bc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc1640097a0 tx=0x7fc16402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e49ec1e95327eaf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:07.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.495+0000 7fc176a00640 1 -- 172.21.15.102:0/3866295381 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164035020 con 0x7fc17815c3a0 2024-03-18T16:26:07.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.495+0000 7fc176a00640 1 -- 172.21.15.102:0/3866295381 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc164031810 con 0x7fc17815c3a0 2024-03-18T16:26:07.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.495+0000 7fc176a00640 1 -- 172.21.15.102:0/3866295381 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164031be0 con 0x7fc17815c3a0 2024-03-18T16:26:07.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.496+0000 7fc17d600640 1 -- 172.21.15.102:0/3866295381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc17815c3a0 msgr2=0x7fc1780c6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:07.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.496+0000 7fc17d600640 1 --2- 172.21.15.102:0/3866295381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc17815c3a0 0x7fc1780c6bc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc1640097a0 tx=0x7fc16402b8b0 comp rx=0 tx=0).stop 2024-03-18T16:26:07.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.496+0000 7fc17d600640 1 -- 172.21.15.102:0/3866295381 shutdown_connections 2024-03-18T16:26:07.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.496+0000 7fc17d600640 1 --2- 172.21.15.102:0/3866295381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc17815c3a0 0x7fc1780c6bc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:07.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.496+0000 7fc17d600640 1 --2- 172.21.15.102:0/3866295381 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc178141650 0x7fc178158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:07.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.496+0000 7fc17d600640 1 -- 172.21.15.102:0/3866295381 >> 172.21.15.102:0/3866295381 conn(0x7fc1780cb890 msgr2=0x7fc1780cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:07.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.496+0000 7fc17d600640 1 -- 172.21.15.102:0/3866295381 shutdown_connections 2024-03-18T16:26:07.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.496+0000 7fc17d600640 1 -- 172.21.15.102:0/3866295381 wait complete. 2024-03-18T16:26:07.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.499+0000 7fc17d600640 1 Processor -- start 2024-03-18T16:26:07.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.499+0000 7fc17d600640 1 -- start start 2024-03-18T16:26:07.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc17d600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1780c4fa0 0x7fc178159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:07.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc17d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178141650 0x7fc1781573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:07.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc17d600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17815c3a0 0x7fc178157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:07.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc17d600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc17815c7c0 con 0x7fc178141650 2024-03-18T16:26:07.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc17d600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc1780c66a0 con 0x7fc1780c4fa0 2024-03-18T16:26:07.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc17d600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc178141a30 con 0x7fc17815c3a0 2024-03-18T16:26:07.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc177e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1780c4fa0 0x7fc178159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:07.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc17cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17815c3a0 0x7fc178157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:07.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc177400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178141650 0x7fc1781573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:07.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.501+0000 7fc177e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1780c4fa0 0x7fc178159d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34488/0 (socket says 172.21.15.102:34488) 2024-03-18T16:26:07.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.502+0000 7fc177e00640 1 -- 172.21.15.102:0/2779618312 learned_addr learned my addr 172.21.15.102:0/2779618312 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:07.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.505+0000 7fc177e00640 1 -- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17815c3a0 msgr2=0x7fc178157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:07.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.505+0000 7fc177e00640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17815c3a0 0x7fc178157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:07.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.505+0000 7fc177e00640 1 -- 172.21.15.102:0/2779618312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178141650 msgr2=0x7fc1781573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:07.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.505+0000 7fc177e00640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178141650 0x7fc1781573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:07.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.505+0000 7fc177e00640 1 -- 172.21.15.102:0/2779618312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1781530c0 con 0x7fc1780c4fa0 2024-03-18T16:26:07.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.507+0000 7fc177e00640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1780c4fa0 0x7fc178159d10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc16800e9b0 tx=0x7fc16800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:07.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.511+0000 7fc16fe00640 1 -- 172.21.15.102:0/2779618312 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc168017070 con 0x7fc1780c4fa0 2024-03-18T16:26:07.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.511+0000 7fc16fe00640 1 -- 172.21.15.102:0/2779618312 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16800c9a0 con 0x7fc1780c4fa0 2024-03-18T16:26:07.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.511+0000 7fc16fe00640 1 -- 172.21.15.102:0/2779618312 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16800cd90 con 0x7fc1780c4fa0 2024-03-18T16:26:07.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.511+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1780c43c0 con 0x7fc1780c4fa0 2024-03-18T16:26:07.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.511+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1781e5870 con 0x7fc1780c4fa0 2024-03-18T16:26:07.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.513+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1780b7f00 con 0x7fc1780c4fa0 2024-03-18T16:26:07.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.770+0000 7fc16fe00640 1 -- 172.21.15.102:0/2779618312 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc16801a070 con 0x7fc1780c4fa0 2024-03-18T16:26:07.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.772+0000 7fc16fe00640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15806bc50 0x7fc15806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:07.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.773+0000 7fc16fe00640 1 -- 172.21.15.102:0/2779618312 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fc16808b510 con 0x7fc1780c4fa0 2024-03-18T16:26:07.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.773+0000 7fc177400640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15806bc50 0x7fc15806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:07.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.774+0000 7fc177400640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15806bc50 0x7fc15806e110 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc164036000 tx=0x7fc16403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:07.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:07.782+0000 7fc16fe00640 1 -- 172.21.15.102:0/2779618312 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc168059630 con 0x7fc1780c4fa0 2024-03-18T16:26:08.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.076+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1780734f0 con 0x7fc1780c4fa0 2024-03-18T16:26:08.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.091+0000 7fc16fe00640 1 -- 172.21.15.102:0/2779618312 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fc16805d6d0 con 0x7fc1780c4fa0 2024-03-18T16:26:08.094 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:08.094 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:08.098 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15806bc50 msgr2=0x7fc15806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15806bc50 0x7fc15806e110 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc164036000 tx=0x7fc16403c040 comp rx=0 tx=0).stop 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1780c4fa0 msgr2=0x7fc178159d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1780c4fa0 0x7fc178159d10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc16800e9b0 tx=0x7fc16800ee70 comp rx=0 tx=0).stop 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 shutdown_connections 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15806bc50 0x7fc15806e110 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17815c3a0 0x7fc178157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc178141650 0x7fc1781573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 --2- 172.21.15.102:0/2779618312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1780c4fa0 0x7fc178159d10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 >> 172.21.15.102:0/2779618312 conn(0x7fc1780cb890 msgr2=0x7fc178140820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 shutdown_connections 2024-03-18T16:26:08.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:08.094+0000 7fc17d600640 1 -- 172.21.15.102:0/2779618312 wait complete. 2024-03-18T16:26:08.718 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:08.718 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:09.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:09.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.890+0000 7efd91800640 1 Processor -- start 2024-03-18T16:26:09.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.890+0000 7efd91800640 1 -- start start 2024-03-18T16:26:09.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.891+0000 7efd91800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:09.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.891+0000 7efd91800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd8c15a6f0 0x7efd8c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:09.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.891+0000 7efd91800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd8c0ca500 con 0x7efd8c15baa0 2024-03-18T16:26:09.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.891+0000 7efd91800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efd8c0cd410 con 0x7efd8c15a6f0 2024-03-18T16:26:09.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.891+0000 7efd91800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd8c15b050 con 0x7efd8c15a310 2024-03-18T16:26:09.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.892+0000 7efd8b400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd8c15a6f0 0x7efd8c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:09.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.892+0000 7efd8be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:09.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.892+0000 7efd8b400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd8c15a6f0 0x7efd8c15ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34492/0 (socket says 172.21.15.102:34492) 2024-03-18T16:26:09.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.892+0000 7efd8b400640 1 -- 172.21.15.102:0/2348972605 learned_addr learned my addr 172.21.15.102:0/2348972605 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:09.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.896+0000 7efd8be00640 1 -- 172.21.15.102:0/2348972605 >> v2:172.21.15.154:3300/0 conn(0x7efd8c15a6f0 msgr2=0x7efd8c15ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:09.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.896+0000 7efd8be00640 1 --2- 172.21.15.102:0/2348972605 >> v2:172.21.15.154:3300/0 conn(0x7efd8c15a6f0 0x7efd8c15ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:09.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.896+0000 7efd8be00640 1 -- 172.21.15.102:0/2348972605 >> v1:172.21.15.154:6789/0 conn(0x7efd8c15baa0 legacy=0x7efd8c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:09.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.896+0000 7efd8be00640 1 -- 172.21.15.102:0/2348972605 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c0ca7a0 con 0x7efd8c15a310 2024-03-18T16:26:09.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.899+0000 7efd8be00640 1 --2- 172.21.15.102:0/2348972605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c06d3e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0097a0 tx=0x7efd7c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d884a09274957ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:09.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.903+0000 7efd8aa00640 1 -- 172.21.15.102:0/2348972605 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c035020 con 0x7efd8c15a310 2024-03-18T16:26:09.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.903+0000 7efd8aa00640 1 -- 172.21.15.102:0/2348972605 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd7c031830 con 0x7efd8c15a310 2024-03-18T16:26:09.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.903+0000 7efd8aa00640 1 -- 172.21.15.102:0/2348972605 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c031c00 con 0x7efd8c15a310 2024-03-18T16:26:09.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.904+0000 7efd91800640 1 -- 172.21.15.102:0/2348972605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 msgr2=0x7efd8c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:09.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.904+0000 7efd91800640 1 --2- 172.21.15.102:0/2348972605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c06d3e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0097a0 tx=0x7efd7c02b8d0 comp rx=0 tx=0).stop 2024-03-18T16:26:09.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.904+0000 7efd91800640 1 -- 172.21.15.102:0/2348972605 shutdown_connections 2024-03-18T16:26:09.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.904+0000 7efd91800640 1 --2- 172.21.15.102:0/2348972605 >> v2:172.21.15.154:3300/0 conn(0x7efd8c15a6f0 0x7efd8c15ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:09.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.904+0000 7efd91800640 1 --2- 172.21.15.102:0/2348972605 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c06d3e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:09.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.904+0000 7efd91800640 1 -- 172.21.15.102:0/2348972605 >> 172.21.15.102:0/2348972605 conn(0x7efd8c08c1a0 msgr2=0x7efd8c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:09.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.904+0000 7efd91800640 1 -- 172.21.15.102:0/2348972605 shutdown_connections 2024-03-18T16:26:09.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.904+0000 7efd91800640 1 -- 172.21.15.102:0/2348972605 wait complete. 2024-03-18T16:26:09.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.906+0000 7efd91800640 1 Processor -- start 2024-03-18T16:26:09.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.906+0000 7efd91800640 1 -- start start 2024-03-18T16:26:09.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.907+0000 7efd91800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c1f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:09.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.907+0000 7efd91800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd8c15a6f0 0x7efd8c0c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:09.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.907+0000 7efd91800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd8c15baa0 0x7efd8c1594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:09.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.907+0000 7efd91800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c0cd130 con 0x7efd8c15a6f0 2024-03-18T16:26:09.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.907+0000 7efd91800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c0cce50 con 0x7efd8c15baa0 2024-03-18T16:26:09.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.907+0000 7efd91800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd8c06d920 con 0x7efd8c15a310 2024-03-18T16:26:09.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.907+0000 7efd8be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c1f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:09.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.908+0000 7efd8b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd8c15a6f0 0x7efd8c0c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:09.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.908+0000 7efd8be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c1f4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:54576/0 (socket says 172.21.15.102:54576) 2024-03-18T16:26:09.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.908+0000 7efd8be00640 1 -- 172.21.15.102:0/3734676504 learned_addr learned my addr 172.21.15.102:0/3734676504 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:09.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.908+0000 7efd90e00640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd8c15baa0 0x7efd8c1594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:09.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.911+0000 7efd8be00640 1 -- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd8c15baa0 msgr2=0x7efd8c1594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:09.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.911+0000 7efd8be00640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd8c15baa0 0x7efd8c1594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:09.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.911+0000 7efd8be00640 1 -- 172.21.15.102:0/3734676504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd8c15a6f0 msgr2=0x7efd8c0c46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:09.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.911+0000 7efd8be00640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd8c15a6f0 0x7efd8c0c46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:09.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.911+0000 7efd8be00640 1 -- 172.21.15.102:0/3734676504 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c0c4c00 con 0x7efd8c15a310 2024-03-18T16:26:09.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.914+0000 7efd8be00640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c1f4470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efd7c02bee0 tx=0x7efd7c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:09.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.918+0000 7efd83e00640 1 -- 172.21.15.102:0/3734676504 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c041070 con 0x7efd8c15a310 2024-03-18T16:26:09.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.918+0000 7efd83e00640 1 -- 172.21.15.102:0/3734676504 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd7c03b920 con 0x7efd8c15a310 2024-03-18T16:26:09.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.918+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8c1578d0 con 0x7efd8c15a310 2024-03-18T16:26:09.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.918+0000 7efd83e00640 1 -- 172.21.15.102:0/3734676504 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c035040 con 0x7efd8c15a310 2024-03-18T16:26:09.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.919+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd8c1f06e0 con 0x7efd8c15a310 2024-03-18T16:26:09.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.920+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8c074fd0 con 0x7efd8c15a310 2024-03-18T16:26:09.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.953+0000 7efd83e00640 1 -- 172.21.15.102:0/3734676504 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efd7c04c060 con 0x7efd8c15a310 2024-03-18T16:26:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.955+0000 7efd83e00640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd6c06bc20 0x7efd6c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:09.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.956+0000 7efd8b400640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd6c06bc20 0x7efd6c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:09.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.957+0000 7efd8b400640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd6c06bc20 0x7efd6c06e0e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efd8c153d90 tx=0x7efd78009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:09.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:09.958+0000 7efd83e00640 1 -- 172.21.15.102:0/3734676504 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7efd7c078b50 con 0x7efd8c15a310 2024-03-18T16:26:10.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.218+0000 7efd83e00640 1 -- 172.21.15.102:0/3734676504 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efd7c07aab0 con 0x7efd8c15a310 2024-03-18T16:26:10.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.527+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd8c1f0ea0 con 0x7efd8c15a310 2024-03-18T16:26:10.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.541+0000 7efd83e00640 1 -- 172.21.15.102:0/3734676504 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7efd8c1f0ea0 con 0x7efd8c15a310 2024-03-18T16:26:10.544 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:10.544 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:10.544 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.544+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd6c06bc20 msgr2=0x7efd6c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.544+0000 7efd91800640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd6c06bc20 0x7efd6c06e0e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efd8c153d90 tx=0x7efd78009240 comp rx=0 tx=0).stop 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.544+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 msgr2=0x7efd8c1f4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.544+0000 7efd91800640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c1f4470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7efd7c02bee0 tx=0x7efd7c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.545+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 shutdown_connections 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.545+0000 7efd91800640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd6c06bc20 0x7efd6c06e0e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.545+0000 7efd91800640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd8c15baa0 0x7efd8c1594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.545+0000 7efd91800640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd8c15a6f0 0x7efd8c0c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.545+0000 7efd91800640 1 --2- 172.21.15.102:0/3734676504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd8c15a310 0x7efd8c1f4470 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.545+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 >> 172.21.15.102:0/3734676504 conn(0x7efd8c08c1a0 msgr2=0x7efd8c0ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.545+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 shutdown_connections 2024-03-18T16:26:10.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:10.545+0000 7efd91800640 1 -- 172.21.15.102:0/3734676504 wait complete. 2024-03-18T16:26:11.177 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:11.177 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:12.179 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:12.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.352+0000 7f860f000640 1 Processor -- start 2024-03-18T16:26:12.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.352+0000 7f860f000640 1 -- start start 2024-03-18T16:26:12.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.353+0000 7f860f000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8608141650 0x7f8608158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:12.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.353+0000 7f860f000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 0x7f86080c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:12.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.353+0000 7f860f000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86080c72d0 con 0x7f860815c3a0 2024-03-18T16:26:12.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.353+0000 7f860f000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8608159370 con 0x7f8608141650 2024-03-18T16:26:12.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.353+0000 7f860f000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8608158ad0 con 0x7f86080c4fa0 2024-03-18T16:26:12.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.354+0000 7f860e600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 0x7f86080c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:12.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.354+0000 7f860d200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f860815c3a0 0x7f86080c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48652/0 (socket says 172.21.15.102:48652) 2024-03-18T16:26:12.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.354+0000 7f860d200640 1 -- 172.21.15.102:0/3037285522 learned_addr learned my addr 172.21.15.102:0/3037285522 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:12.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.355+0000 7f860dc00640 1 --2- 172.21.15.102:0/3037285522 >> v2:172.21.15.102:3300/0 conn(0x7f8608141650 0x7f8608158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:12.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.358+0000 7f860e600640 1 -- 172.21.15.102:0/3037285522 >> v2:172.21.15.102:3300/0 conn(0x7f8608141650 msgr2=0x7f8608158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:12.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.358+0000 7f860e600640 1 --2- 172.21.15.102:0/3037285522 >> v2:172.21.15.102:3300/0 conn(0x7f8608141650 0x7f8608158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.358+0000 7f860e600640 1 -- 172.21.15.102:0/3037285522 >> v1:172.21.15.102:6789/0 conn(0x7f860815c3a0 legacy=0x7f86080c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:12.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.358+0000 7f860e600640 1 -- 172.21.15.102:0/3037285522 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8608158db0 con 0x7f86080c4fa0 2024-03-18T16:26:12.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.361+0000 7f860e600640 1 --2- 172.21.15.102:0/3037285522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 0x7f86080c5430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8600009d90 tx=0x7f860002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c058dac611a5e506 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:12.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.364+0000 7f8607e00640 1 -- 172.21.15.102:0/3037285522 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8600039070 con 0x7f86080c4fa0 2024-03-18T16:26:12.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.366+0000 7f8607e00640 1 -- 172.21.15.102:0/3037285522 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f860002cc60 con 0x7f86080c4fa0 2024-03-18T16:26:12.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.367+0000 7f860f000640 1 -- 172.21.15.102:0/3037285522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 msgr2=0x7f86080c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:12.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.367+0000 7f8607e00640 1 -- 172.21.15.102:0/3037285522 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8600034660 con 0x7f86080c4fa0 2024-03-18T16:26:12.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.367+0000 7f860f000640 1 --2- 172.21.15.102:0/3037285522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 0x7f86080c5430 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8600009d90 tx=0x7f860002beb0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.367+0000 7f860f000640 1 -- 172.21.15.102:0/3037285522 shutdown_connections 2024-03-18T16:26:12.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.367+0000 7f860f000640 1 --2- 172.21.15.102:0/3037285522 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 0x7f86080c5430 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.367+0000 7f860f000640 1 --2- 172.21.15.102:0/3037285522 >> v2:172.21.15.102:3300/0 conn(0x7f8608141650 0x7f8608158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.367+0000 7f860f000640 1 -- 172.21.15.102:0/3037285522 >> 172.21.15.102:0/3037285522 conn(0x7f86080cb890 msgr2=0x7f86080cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:12.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.367+0000 7f860f000640 1 -- 172.21.15.102:0/3037285522 shutdown_connections 2024-03-18T16:26:12.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.367+0000 7f860f000640 1 -- 172.21.15.102:0/3037285522 wait complete. 2024-03-18T16:26:12.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.369+0000 7f860f000640 1 Processor -- start 2024-03-18T16:26:12.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.370+0000 7f860f000640 1 -- start start 2024-03-18T16:26:12.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.371+0000 7f860f000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 0x7f8608158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:12.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.371+0000 7f860f000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8608141650 0x7f8608157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:12.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.371+0000 7f860f000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860815c3a0 0x7f86081577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:12.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.371+0000 7f860f000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f86080c66a0 con 0x7f8608141650 2024-03-18T16:26:12.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.371+0000 7f860f000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f86080c63c0 con 0x7f860815c3a0 2024-03-18T16:26:12.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.371+0000 7f860f000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f860815c7c0 con 0x7f86080c4fa0 2024-03-18T16:26:12.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.372+0000 7f860dc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 0x7f8608158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:12.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.372+0000 7f860e600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860815c3a0 0x7f86081577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:12.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.372+0000 7f860d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8608141650 0x7f8608157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:12.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.372+0000 7f860e600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860815c3a0 0x7f86081577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:58286/0 (socket says 172.21.15.102:58286) 2024-03-18T16:26:12.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.372+0000 7f860e600640 1 -- 172.21.15.102:0/1991745375 learned_addr learned my addr 172.21.15.102:0/1991745375 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:12.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.375+0000 7f860e600640 1 -- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 msgr2=0x7f8608158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:12.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.375+0000 7f860e600640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 0x7f8608158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.375+0000 7f860e600640 1 -- 172.21.15.102:0/1991745375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8608141650 msgr2=0x7f8608157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:12.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.375+0000 7f860e600640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8608141650 0x7f8608157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.375+0000 7f860e600640 1 -- 172.21.15.102:0/1991745375 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86081f1340 con 0x7f860815c3a0 2024-03-18T16:26:12.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.378+0000 7f860e600640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860815c3a0 0x7f86081577a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8600009d90 tx=0x7f8600004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:12.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.381+0000 7f8605600640 1 -- 172.21.15.102:0/1991745375 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8600039050 con 0x7f860815c3a0 2024-03-18T16:26:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.381+0000 7f8605600640 1 -- 172.21.15.102:0/1991745375 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8600045070 con 0x7f860815c3a0 2024-03-18T16:26:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.381+0000 7f8605600640 1 -- 172.21.15.102:0/1991745375 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860003e830 con 0x7f860815c3a0 2024-03-18T16:26:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.382+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8608155880 con 0x7f860815c3a0 2024-03-18T16:26:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.382+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8608156a60 con 0x7f860815c3a0 2024-03-18T16:26:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.383+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8608155d80 con 0x7f860815c3a0 2024-03-18T16:26:12.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.402+0000 7f8605600640 1 -- 172.21.15.102:0/1991745375 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f860003e9d0 con 0x7f860815c3a0 2024-03-18T16:26:12.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.403+0000 7f8605600640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f85e806bbd0 0x7f85e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:12.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.404+0000 7f860dc00640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f85e806bbd0 0x7f85e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:12.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.404+0000 7f860dc00640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f85e806bbd0 0x7f85e806e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009b40 tx=0x7f85fc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:12.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.405+0000 7f8605600640 1 -- 172.21.15.102:0/1991745375 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f860007a810 con 0x7f860815c3a0 2024-03-18T16:26:12.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.653+0000 7f8605600640 1 -- 172.21.15.102:0/1991745375 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8608155d80 con 0x7f860815c3a0 2024-03-18T16:26:12.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.888+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8608067c20 con 0x7f860815c3a0 2024-03-18T16:26:12.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.901+0000 7f8605600640 1 -- 172.21.15.102:0/1991745375 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8608067c20 con 0x7f860815c3a0 2024-03-18T16:26:12.903 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:12.903 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:12.904 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:12.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.904+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f85e806bbd0 msgr2=0x7f85e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:12.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.904+0000 7f860f000640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f85e806bbd0 0x7f85e806e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f85fc009b40 tx=0x7f85fc009190 comp rx=0 tx=0).stop 2024-03-18T16:26:12.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.904+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860815c3a0 msgr2=0x7f86081577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:12.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.904+0000 7f860f000640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860815c3a0 0x7f86081577a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8600009d90 tx=0x7f8600004a40 comp rx=0 tx=0).stop 2024-03-18T16:26:12.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.905+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 shutdown_connections 2024-03-18T16:26:12.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.905+0000 7f860f000640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f85e806bbd0 0x7f85e806e090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.905+0000 7f860f000640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f860815c3a0 0x7f86081577a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.905+0000 7f860f000640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8608141650 0x7f8608157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.905+0000 7f860f000640 1 --2- 172.21.15.102:0/1991745375 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f86080c4fa0 0x7f8608158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:12.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.905+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 >> 172.21.15.102:0/1991745375 conn(0x7f86080cb890 msgr2=0x7f860813fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:12.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.905+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 shutdown_connections 2024-03-18T16:26:12.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:12.905+0000 7f860f000640 1 -- 172.21.15.102:0/1991745375 wait complete. 2024-03-18T16:26:13.534 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:13.535 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:14.536 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:14.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.708+0000 7fd76f200640 1 Processor -- start 2024-03-18T16:26:14.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.709+0000 7fd76f200640 1 -- start start 2024-03-18T16:26:14.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.710+0000 7fd76f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd770158420 0x7fd77015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:14.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.710+0000 7fd76f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7700c72d0 con 0x7fd770141650 2024-03-18T16:26:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.710+0000 7fd76f200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd770158840 con 0x7fd7700c4fa0 2024-03-18T16:26:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.710+0000 7fd76f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd77015c8e0 con 0x7fd770158420 2024-03-18T16:26:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.711+0000 7fd76e800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd7700c4fa0 0x7fd770162cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:53952/0 (socket says 172.21.15.102:53952) 2024-03-18T16:26:14.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.711+0000 7fd76e800640 1 -- 172.21.15.102:0/1614672920 learned_addr learned my addr 172.21.15.102:0/1614672920 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:14.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.711+0000 7fd76d400640 1 --2- 172.21.15.102:0/1614672920 >> v2:172.21.15.102:3300/0 conn(0x7fd770158420 0x7fd77015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:14.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.713+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1671114267 0 0) 0x7fd770158840 con 0x7fd7700c4fa0 2024-03-18T16:26:14.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.714+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd754003520 con 0x7fd7700c4fa0 2024-03-18T16:26:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3029489373 0 0) 0x7fd754003520 con 0x7fd7700c4fa0 2024-03-18T16:26:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 >> v2:172.21.15.102:3300/0 conn(0x7fd770158420 msgr2=0x7fd77015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 --2- 172.21.15.102:0/1614672920 >> v2:172.21.15.102:3300/0 conn(0x7fd770158420 0x7fd77015c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:14.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 >> v1:172.21.15.102:6789/0 conn(0x7fd770141650 legacy=0x7fd7700c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd770158ad0 con 0x7fd7700c4fa0 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7fd7680019f0 con 0x7fd7700c4fa0 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 >> v1:172.21.15.154:6789/0 conn(0x7fd7700c4fa0 legacy=0x7fd770162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540039b0 0x7fd754003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754004800 0x7fd754004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7540051c0 0x7fd75400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.717+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd77015c8e0 con 0x7fd7540051c0 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.718+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd754010470 con 0x7fd754004800 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.718+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd754010670 con 0x7fd7540039b0 2024-03-18T16:26:14.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.718+0000 7fd76de00640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540039b0 0x7fd754003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:14.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.719+0000 7fd76d400640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754004800 0x7fd754004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.721+0000 7fd76de00640 1 -- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754004800 msgr2=0x7fd754004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.721+0000 7fd76de00640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd754004800 0x7fd754004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.721+0000 7fd76de00640 1 -- 172.21.15.102:0/1614672920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7540051c0 msgr2=0x7fd75400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.721+0000 7fd76de00640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7540051c0 0x7fd75400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.721+0000 7fd76de00640 1 -- 172.21.15.102:0/1614672920 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd770158ad0 con 0x7fd7540039b0 2024-03-18T16:26:14.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.721+0000 7fd76e800640 1 -- 172.21.15.102:0/1614672920 reap_dead start 2024-03-18T16:26:14.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.724+0000 7fd76de00640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540039b0 0x7fd754003dd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd760001750 tx=0x7fd754005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47c275c9407d374e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:14.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.726+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760001370 con 0x7fd7540039b0 2024-03-18T16:26:14.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.728+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7600052d0 con 0x7fd7540039b0 2024-03-18T16:26:14.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.728+0000 7fd76ca00640 1 -- 172.21.15.102:0/1614672920 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760003830 con 0x7fd7540039b0 2024-03-18T16:26:14.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.729+0000 7fd76f200640 1 -- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540039b0 msgr2=0x7fd754003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.729+0000 7fd76f200640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540039b0 0x7fd754003dd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd760001750 tx=0x7fd754005830 comp rx=0 tx=0).stop 2024-03-18T16:26:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.729+0000 7fd76f200640 1 -- 172.21.15.102:0/1614672920 shutdown_connections 2024-03-18T16:26:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.729+0000 7fd76f200640 1 --2- 172.21.15.102:0/1614672920 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7540039b0 0x7fd754003dd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.729+0000 7fd76f200640 1 -- 172.21.15.102:0/1614672920 >> 172.21.15.102:0/1614672920 conn(0x7fd7700cb890 msgr2=0x7fd7700cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.729+0000 7fd76f200640 1 -- 172.21.15.102:0/1614672920 shutdown_connections 2024-03-18T16:26:14.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.729+0000 7fd76f200640 1 -- 172.21.15.102:0/1614672920 wait complete. 2024-03-18T16:26:14.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.732+0000 7fd76f200640 1 Processor -- start 2024-03-18T16:26:14.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.732+0000 7fd76f200640 1 -- start start 2024-03-18T16:26:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.733+0000 7fd76f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7540039b0 0x7fd77015b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.733+0000 7fd76f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd770152d90 0x7fd7700ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.733+0000 7fd76f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7701531f0 0x7fd7700cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.733+0000 7fd76f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd770159090 con 0x7fd7540039b0 2024-03-18T16:26:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.733+0000 7fd76f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd770159370 con 0x7fd770152d90 2024-03-18T16:26:14.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.733+0000 7fd76f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd7700c63c0 con 0x7fd7701531f0 2024-03-18T16:26:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.734+0000 7fd76d400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd770152d90 0x7fd7700ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.734+0000 7fd76e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7701531f0 0x7fd7700cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.734+0000 7fd76de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7540039b0 0x7fd77015b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.734+0000 7fd76d400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd770152d90 0x7fd7700ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:58298/0 (socket says 172.21.15.102:58298) 2024-03-18T16:26:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.734+0000 7fd76d400640 1 -- 172.21.15.102:0/1256209539 learned_addr learned my addr 172.21.15.102:0/1256209539 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:14.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.734+0000 7fd76e800640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7701531f0 0x7fd7700cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53914/0 (socket says 172.21.15.102:53914) 2024-03-18T16:26:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.737+0000 7fd76e800640 1 -- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd770152d90 msgr2=0x7fd7700ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.737+0000 7fd76e800640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd770152d90 0x7fd7700ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.737+0000 7fd76e800640 1 -- 172.21.15.102:0/1256209539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7540039b0 msgr2=0x7fd77015b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:14.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.738+0000 7fd76e800640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7540039b0 0x7fd77015b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:14.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.738+0000 7fd76e800640 1 -- 172.21.15.102:0/1256209539 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77015bf00 con 0x7fd7701531f0 2024-03-18T16:26:14.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.740+0000 7fd76e800640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7701531f0 0x7fd7700cae30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd770070160 tx=0x7fd76800cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:14.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.744+0000 7fd766000640 1 -- 172.21.15.102:0/1256209539 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768004d60 con 0x7fd7701531f0 2024-03-18T16:26:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.745+0000 7fd766000640 1 -- 172.21.15.102:0/1256209539 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76800a5b0 con 0x7fd7701531f0 2024-03-18T16:26:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.745+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77015c1c0 con 0x7fd7701531f0 2024-03-18T16:26:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.745+0000 7fd766000640 1 -- 172.21.15.102:0/1256209539 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76800a9a0 con 0x7fd7701531f0 2024-03-18T16:26:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.745+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd770156f40 con 0x7fd7701531f0 2024-03-18T16:26:14.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.746+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd770074fd0 con 0x7fd7701531f0 2024-03-18T16:26:14.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.770+0000 7fd766000640 1 -- 172.21.15.102:0/1256209539 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd76800ab40 con 0x7fd7701531f0 2024-03-18T16:26:14.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.771+0000 7fd766000640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd75006bc50 0x7fd75006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:14.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.772+0000 7fd76de00640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd75006bc50 0x7fd75006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:14.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.772+0000 7fd766000640 1 -- 172.21.15.102:0/1256209539 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd76800ae30 con 0x7fd7701531f0 2024-03-18T16:26:14.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:14.772+0000 7fd76de00640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd75006bc50 0x7fd75006e110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd760033000 tx=0x7fd760009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:15.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.036+0000 7fd766000640 1 -- 172.21.15.102:0/1256209539 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd768058960 con 0x7fd7701531f0 2024-03-18T16:26:15.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.281+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7700b7f00 con 0x7fd7701531f0 2024-03-18T16:26:15.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.325+0000 7fd766000640 1 -- 172.21.15.102:0/1256209539 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd76805ca00 con 0x7fd7701531f0 2024-03-18T16:26:15.327 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:15.327 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:15.328 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:15.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd75006bc50 msgr2=0x7fd75006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:15.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd75006bc50 0x7fd75006e110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd760033000 tx=0x7fd760009030 comp rx=0 tx=0).stop 2024-03-18T16:26:15.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7701531f0 msgr2=0x7fd7700cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:15.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7701531f0 0x7fd7700cae30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd770070160 tx=0x7fd76800cbe0 comp rx=0 tx=0).stop 2024-03-18T16:26:15.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 shutdown_connections 2024-03-18T16:26:15.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd75006bc50 0x7fd75006e110 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:15.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7701531f0 0x7fd7700cae30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:15.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd770152d90 0x7fd7700ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:15.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 --2- 172.21.15.102:0/1256209539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7540039b0 0x7fd77015b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:15.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 >> 172.21.15.102:0/1256209539 conn(0x7fd7700cb890 msgr2=0x7fd77013e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:15.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 shutdown_connections 2024-03-18T16:26:15.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:15.328+0000 7fd76f200640 1 -- 172.21.15.102:0/1256209539 wait complete. 2024-03-18T16:26:15.958 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:15.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:16.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:17.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.136+0000 7f2453e00640 1 Processor -- start 2024-03-18T16:26:17.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.136+0000 7f2453e00640 1 -- start start 2024-03-18T16:26:17.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.137+0000 7f2453e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24540c9a20 0x7f24540cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:17.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.137+0000 7f2453e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f245415aff0 0x7f24540d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:17.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.137+0000 7f2453e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2454154a90 con 0x7f245406d450 2024-03-18T16:26:17.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.137+0000 7f2453e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f245415e270 con 0x7f245415aff0 2024-03-18T16:26:17.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.137+0000 7f2453e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f245415e530 con 0x7f24540c9a20 2024-03-18T16:26:17.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.138+0000 7f2453400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f245415aff0 0x7f24540d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:17.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.138+0000 7f2452a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24540c9a20 0x7f24540cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:17.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.138+0000 7f2453400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f245415aff0 0x7f24540d0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:58312/0 (socket says 172.21.15.102:58312) 2024-03-18T16:26:17.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.138+0000 7f2453400640 1 -- 172.21.15.102:0/1372578457 learned_addr learned my addr 172.21.15.102:0/1372578457 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:17.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.140+0000 7f2453400640 1 -- 172.21.15.102:0/1372578457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24540c9a20 msgr2=0x7f24540cfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:17.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.140+0000 7f2453400640 1 --2- 172.21.15.102:0/1372578457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24540c9a20 0x7f24540cfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:17.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.140+0000 7f2453400640 1 -- 172.21.15.102:0/1372578457 >> v1:172.21.15.154:6789/0 conn(0x7f245406d450 legacy=0x7f245415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:17.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.140+0000 7f2453400640 1 -- 172.21.15.102:0/1372578457 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24540ca610 con 0x7f245415aff0 2024-03-18T16:26:17.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.144+0000 7f2453400640 1 --2- 172.21.15.102:0/1372578457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 0x7f24540d0340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f244c009e50 tx=0x7f244c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a63727e6455ef22b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.148+0000 7f2451600640 1 -- 172.21.15.102:0/1372578457 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c02cb00 con 0x7f245415aff0 2024-03-18T16:26:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.148+0000 7f2451600640 1 -- 172.21.15.102:0/1372578457 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f244c02cca0 con 0x7f245415aff0 2024-03-18T16:26:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.148+0000 7f2451600640 1 -- 172.21.15.102:0/1372578457 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c034610 con 0x7f245415aff0 2024-03-18T16:26:17.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.149+0000 7f2453e00640 1 -- 172.21.15.102:0/1372578457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 msgr2=0x7f24540d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:17.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.149+0000 7f2453e00640 1 --2- 172.21.15.102:0/1372578457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 0x7f24540d0340 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f244c009e50 tx=0x7f244c02bf60 comp rx=0 tx=0).stop 2024-03-18T16:26:17.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.150+0000 7f2453e00640 1 -- 172.21.15.102:0/1372578457 shutdown_connections 2024-03-18T16:26:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.150+0000 7f2453e00640 1 --2- 172.21.15.102:0/1372578457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 0x7f24540d0340 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.150+0000 7f2453e00640 1 --2- 172.21.15.102:0/1372578457 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24540c9a20 0x7f24540cfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.150+0000 7f2453e00640 1 -- 172.21.15.102:0/1372578457 >> 172.21.15.102:0/1372578457 conn(0x7f2454067160 msgr2=0x7f2454067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.150+0000 7f2453e00640 1 -- 172.21.15.102:0/1372578457 shutdown_connections 2024-03-18T16:26:17.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.150+0000 7f2453e00640 1 -- 172.21.15.102:0/1372578457 wait complete. 2024-03-18T16:26:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.152+0000 7f2453e00640 1 Processor -- start 2024-03-18T16:26:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.153+0000 7f2453e00640 1 -- start start 2024-03-18T16:26:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.154+0000 7f2453e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245406d450 0x7f24540af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.154+0000 7f2453e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24540c9a20 0x7f2454153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.154+0000 7f2453e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 0x7f2454154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.154+0000 7f2453e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24540cad50 con 0x7f24540c9a20 2024-03-18T16:26:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.154+0000 7f2453e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24540cabd0 con 0x7f245415aff0 2024-03-18T16:26:17.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.154+0000 7f2453e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f245406e1d0 con 0x7f245406d450 2024-03-18T16:26:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.155+0000 7f2453400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 0x7f2454154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.155+0000 7f2453400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 0x7f2454154310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:58316/0 (socket says 172.21.15.102:58316) 2024-03-18T16:26:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.155+0000 7f2453400640 1 -- 172.21.15.102:0/3138375445 learned_addr learned my addr 172.21.15.102:0/3138375445 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.155+0000 7f2452000640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24540c9a20 0x7f2454153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:17.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.155+0000 7f2452a00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245406d450 0x7f24540af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.158+0000 7f2453400640 1 -- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245406d450 msgr2=0x7f24540af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.158+0000 7f2453400640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245406d450 0x7f24540af570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:17.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.158+0000 7f2453400640 1 -- 172.21.15.102:0/3138375445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24540c9a20 msgr2=0x7f2454153dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.158+0000 7f2453400640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24540c9a20 0x7f2454153dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:17.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.158+0000 7f2453400640 1 -- 172.21.15.102:0/3138375445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24541e52b0 con 0x7f245415aff0 2024-03-18T16:26:17.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.161+0000 7f2453400640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 0x7f2454154310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f244c009e50 tx=0x7f244c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.163+0000 7f244aa00640 1 -- 172.21.15.102:0/3138375445 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c044020 con 0x7f245415aff0 2024-03-18T16:26:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.163+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24541e3e90 con 0x7f245415aff0 2024-03-18T16:26:17.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.163+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24540b0c30 con 0x7f245415aff0 2024-03-18T16:26:17.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.165+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2454067540 con 0x7f245415aff0 2024-03-18T16:26:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.166+0000 7f244aa00640 1 -- 172.21.15.102:0/3138375445 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f244c02ce50 con 0x7f245415aff0 2024-03-18T16:26:17.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.166+0000 7f244aa00640 1 -- 172.21.15.102:0/3138375445 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244c009510 con 0x7f245415aff0 2024-03-18T16:26:17.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.450+0000 7f244aa00640 1 -- 172.21.15.102:0/3138375445 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f244c047020 con 0x7f245415aff0 2024-03-18T16:26:17.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.452+0000 7f244aa00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f243406bbd0 0x7f243406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:17.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.452+0000 7f2452a00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f243406bbd0 0x7f243406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:17.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.453+0000 7f2452a00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f243406bbd0 0x7f243406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2444009b40 tx=0x7f2444009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:17.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.456+0000 7f244aa00640 1 -- 172.21.15.102:0/3138375445 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f244c079be0 con 0x7f245415aff0 2024-03-18T16:26:17.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.470+0000 7f244aa00640 1 -- 172.21.15.102:0/3138375445 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2454067540 con 0x7f245415aff0 2024-03-18T16:26:17.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.718+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2454074fd0 con 0x7f245415aff0 2024-03-18T16:26:17.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.763+0000 7f244aa00640 1 -- 172.21.15.102:0/3138375445 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f244c04e030 con 0x7f245415aff0 2024-03-18T16:26:17.765 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:17.765 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:17.765 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:17.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f243406bbd0 msgr2=0x7f243406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:17.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f243406bbd0 0x7f243406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2444009b40 tx=0x7f2444009190 comp rx=0 tx=0).stop 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 msgr2=0x7f2454154310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 0x7f2454154310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f244c009e50 tx=0x7f244c004a40 comp rx=0 tx=0).stop 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 shutdown_connections 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f243406bbd0 0x7f243406e090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f245415aff0 0x7f2454154310 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24540c9a20 0x7f2454153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 --2- 172.21.15.102:0/3138375445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f245406d450 0x7f24540af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 >> 172.21.15.102:0/3138375445 conn(0x7f2454067160 msgr2=0x7f24541eaf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.767+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 shutdown_connections 2024-03-18T16:26:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:17.768+0000 7f2453e00640 1 -- 172.21.15.102:0/3138375445 wait complete. 2024-03-18T16:26:18.396 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:18.396 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:19.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:19.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.569+0000 7f9432400640 1 Processor -- start 2024-03-18T16:26:19.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.569+0000 7f9432400640 1 -- start start 2024-03-18T16:26:19.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.570+0000 7f9432400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15a6f0 0x7f942c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:19.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.570+0000 7f9432400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f942c15baa0 0x7f942c15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:19.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.570+0000 7f9432400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f942c15c470 con 0x7f942c15a310 2024-03-18T16:26:19.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.570+0000 7f9432400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f942c0cd410 con 0x7f942c15baa0 2024-03-18T16:26:19.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.570+0000 7f9432400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f942c0ca500 con 0x7f942c15a6f0 2024-03-18T16:26:19.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.571+0000 7f9431a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f942c15baa0 0x7f942c15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:19.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.571+0000 7f942be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15a6f0 0x7f942c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:19.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.571+0000 7f9431a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f942c15baa0 0x7f942c15bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:58318/0 (socket says 172.21.15.102:58318) 2024-03-18T16:26:19.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.571+0000 7f9431a00640 1 -- 172.21.15.102:0/1576458064 learned_addr learned my addr 172.21.15.102:0/1576458064 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:19.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.575+0000 7f942be00640 1 -- 172.21.15.102:0/1576458064 >> v2:172.21.15.154:3300/0 conn(0x7f942c15baa0 msgr2=0x7f942c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:19.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.575+0000 7f942be00640 1 --2- 172.21.15.102:0/1576458064 >> v2:172.21.15.154:3300/0 conn(0x7f942c15baa0 0x7f942c15bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:19.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.575+0000 7f942be00640 1 -- 172.21.15.102:0/1576458064 >> v1:172.21.15.154:6789/0 conn(0x7f942c15a310 legacy=0x7f942c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:19.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.575+0000 7f942be00640 1 -- 172.21.15.102:0/1576458064 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c15b050 con 0x7f942c15a6f0 2024-03-18T16:26:19.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.578+0000 7f942be00640 1 --2- 172.21.15.102:0/1576458064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15a6f0 0x7f942c15ab10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f94180097a0 tx=0x7f941802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2f049b1af2de68b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:19.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.584+0000 7f942b400640 1 -- 172.21.15.102:0/1576458064 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418035020 con 0x7f942c15a6f0 2024-03-18T16:26:19.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.584+0000 7f942b400640 1 -- 172.21.15.102:0/1576458064 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9418031830 con 0x7f942c15a6f0 2024-03-18T16:26:19.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.584+0000 7f942b400640 1 -- 172.21.15.102:0/1576458064 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418031c00 con 0x7f942c15a6f0 2024-03-18T16:26:19.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.585+0000 7f9432400640 1 -- 172.21.15.102:0/1576458064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15a6f0 msgr2=0x7f942c15ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:19.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.585+0000 7f9432400640 1 --2- 172.21.15.102:0/1576458064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15a6f0 0x7f942c15ab10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f94180097a0 tx=0x7f941802b8d0 comp rx=0 tx=0).stop 2024-03-18T16:26:19.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.585+0000 7f9432400640 1 -- 172.21.15.102:0/1576458064 shutdown_connections 2024-03-18T16:26:19.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.585+0000 7f9432400640 1 --2- 172.21.15.102:0/1576458064 >> v2:172.21.15.154:3300/0 conn(0x7f942c15baa0 0x7f942c15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:19.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.585+0000 7f9432400640 1 --2- 172.21.15.102:0/1576458064 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15a6f0 0x7f942c15ab10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:19.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.585+0000 7f9432400640 1 -- 172.21.15.102:0/1576458064 >> 172.21.15.102:0/1576458064 conn(0x7f942c08c1a0 msgr2=0x7f942c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:19.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.585+0000 7f9432400640 1 -- 172.21.15.102:0/1576458064 shutdown_connections 2024-03-18T16:26:19.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.586+0000 7f9432400640 1 -- 172.21.15.102:0/1576458064 wait complete. 2024-03-18T16:26:19.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.587+0000 7f9432400640 1 Processor -- start 2024-03-18T16:26:19.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.588+0000 7f9432400640 1 -- start start 2024-03-18T16:26:19.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.588+0000 7f9432400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f942c15a310 0x7f942c0c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:19.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.588+0000 7f9432400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f942c15a6f0 0x7f942c0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:19.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.589+0000 7f9432400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15baa0 0x7f942c0c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:19.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.589+0000 7f9432400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f942c0cce50 con 0x7f942c15a310 2024-03-18T16:26:19.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.589+0000 7f9432400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f942c0caa80 con 0x7f942c15a6f0 2024-03-18T16:26:19.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.589+0000 7f9432400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f942c0cd130 con 0x7f942c15baa0 2024-03-18T16:26:19.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.589+0000 7f942be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f942c15a6f0 0x7f942c0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:19.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.589+0000 7f9431a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15baa0 0x7f942c0c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:19.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.589+0000 7f9431000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f942c15a310 0x7f942c0c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:19.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.589+0000 7f942be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f942c15a6f0 0x7f942c0c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:58326/0 (socket says 172.21.15.102:58326) 2024-03-18T16:26:19.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.589+0000 7f942be00640 1 -- 172.21.15.102:0/1808280830 learned_addr learned my addr 172.21.15.102:0/1808280830 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:19.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.593+0000 7f942be00640 1 -- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15baa0 msgr2=0x7f942c0c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:19.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.593+0000 7f942be00640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15baa0 0x7f942c0c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:19.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.593+0000 7f942be00640 1 -- 172.21.15.102:0/1808280830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f942c15a310 msgr2=0x7f942c0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:19.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.593+0000 7f942be00640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f942c15a310 0x7f942c0c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:19.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.593+0000 7f942be00640 1 -- 172.21.15.102:0/1808280830 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c0c9a20 con 0x7f942c15a6f0 2024-03-18T16:26:19.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.596+0000 7f942be00640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f942c15a6f0 0x7f942c0c7160 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f941802bee0 tx=0x7f94180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:19.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.600+0000 7f9428c00640 1 -- 172.21.15.102:0/1808280830 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418035040 con 0x7f942c15a6f0 2024-03-18T16:26:19.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.600+0000 7f9428c00640 1 -- 172.21.15.102:0/1808280830 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f941803b920 con 0x7f942c15a6f0 2024-03-18T16:26:19.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.600+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942c0c9d00 con 0x7f942c15a6f0 2024-03-18T16:26:19.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.600+0000 7f9428c00640 1 -- 172.21.15.102:0/1808280830 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941803bd10 con 0x7f942c15a6f0 2024-03-18T16:26:19.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.600+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942c159450 con 0x7f942c15a6f0 2024-03-18T16:26:19.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.601+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942c074fd0 con 0x7f942c15a6f0 2024-03-18T16:26:19.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.861+0000 7f9428c00640 1 -- 172.21.15.102:0/1808280830 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9418038020 con 0x7f942c15a6f0 2024-03-18T16:26:19.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.862+0000 7f9428c00640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f940c06bc50 0x7f940c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:19.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.862+0000 7f9428c00640 1 -- 172.21.15.102:0/1808280830 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f94180ad060 con 0x7f942c15a6f0 2024-03-18T16:26:19.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.862+0000 7f9431000640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f940c06bc50 0x7f940c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:19.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.863+0000 7f9431000640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f940c06bc50 0x7f940c06e110 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f94200016e0 tx=0x7f9420009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:19.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:19.873+0000 7f9428c00640 1 -- 172.21.15.102:0/1808280830 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9418031800 con 0x7f942c15a6f0 2024-03-18T16:26:20.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.114+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f942c0caed0 con 0x7f942c15a6f0 2024-03-18T16:26:20.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.128+0000 7f9428c00640 1 -- 172.21.15.102:0/1808280830 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f941807b210 con 0x7f942c15a6f0 2024-03-18T16:26:20.130 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:20.131 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:20.131 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:20.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f940c06bc50 msgr2=0x7f940c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:20.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f940c06bc50 0x7f940c06e110 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f94200016e0 tx=0x7f9420009040 comp rx=0 tx=0).stop 2024-03-18T16:26:20.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f942c15a6f0 msgr2=0x7f942c0c7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:20.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f942c15a6f0 0x7f942c0c7160 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f941802bee0 tx=0x7f94180046c0 comp rx=0 tx=0).stop 2024-03-18T16:26:20.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 shutdown_connections 2024-03-18T16:26:20.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f940c06bc50 0x7f940c06e110 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:20.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f942c15baa0 0x7f942c0c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:20.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f942c15a6f0 0x7f942c0c7160 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:20.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 --2- 172.21.15.102:0/1808280830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f942c15a310 0x7f942c0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:20.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 >> 172.21.15.102:0/1808280830 conn(0x7f942c08c1a0 msgr2=0x7f942c15d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:20.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 shutdown_connections 2024-03-18T16:26:20.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:20.132+0000 7f9432400640 1 -- 172.21.15.102:0/1808280830 wait complete. 2024-03-18T16:26:20.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:20.755 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:21.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:26:21.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:26:21.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:21.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.931+0000 7f1bd3e00640 1 Processor -- start 2024-03-18T16:26:21.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.931+0000 7f1bd3e00640 1 -- start start 2024-03-18T16:26:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.932+0000 7f1bd3e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1bd40733b0 0x7f1bd4157a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.932+0000 7f1bd3e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bd40c75b0 con 0x7f1bd414dad0 2024-03-18T16:26:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.932+0000 7f1bd3e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bd40c7850 con 0x7f1bd415b1c0 2024-03-18T16:26:21.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.932+0000 7f1bd3e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1bd414deb0 con 0x7f1bd40733b0 2024-03-18T16:26:21.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.933+0000 7f1bd2000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1bd40733b0 0x7f1bd4157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:21.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.933+0000 7f1bd2a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1bd414dad0 0x7f1bd40c6230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52020/0 (socket says 172.21.15.102:52020) 2024-03-18T16:26:21.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.933+0000 7f1bd2a00640 1 -- 172.21.15.102:0/3037743803 learned_addr learned my addr 172.21.15.102:0/3037743803 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.935+0000 7f1bd2000640 1 -- 172.21.15.102:0/3037743803 >> v1:172.21.15.154:6789/0 conn(0x7f1bd415b1c0 legacy=0x7f1bd4161940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.935+0000 7f1bd2000640 1 -- 172.21.15.102:0/3037743803 >> v1:172.21.15.102:6789/0 conn(0x7f1bd414dad0 legacy=0x7f1bd40c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:21.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.935+0000 7f1bd2000640 1 -- 172.21.15.102:0/3037743803 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd40c7b30 con 0x7f1bd40733b0 2024-03-18T16:26:21.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.938+0000 7f1bd2000640 1 --2- 172.21.15.102:0/3037743803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 0x7f1bd4157a40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40702e0 tx=0x7f1bc002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6becb37cd2469015 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:21.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.941+0000 7f1bd1600640 1 -- 172.21.15.102:0/3037743803 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0035020 con 0x7f1bd40733b0 2024-03-18T16:26:21.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.943+0000 7f1bd1600640 1 -- 172.21.15.102:0/3037743803 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc0031810 con 0x7f1bd40733b0 2024-03-18T16:26:21.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.943+0000 7f1bd1600640 1 -- 172.21.15.102:0/3037743803 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0031be0 con 0x7f1bd40733b0 2024-03-18T16:26:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.943+0000 7f1bd3e00640 1 -- 172.21.15.102:0/3037743803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 msgr2=0x7f1bd4157a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.943+0000 7f1bd3e00640 1 --2- 172.21.15.102:0/3037743803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 0x7f1bd4157a40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40702e0 tx=0x7f1bc002b8b0 comp rx=0 tx=0).stop 2024-03-18T16:26:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.944+0000 7f1bd3e00640 1 -- 172.21.15.102:0/3037743803 shutdown_connections 2024-03-18T16:26:21.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.944+0000 7f1bd3e00640 1 --2- 172.21.15.102:0/3037743803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 0x7f1bd4157a40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:21.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.944+0000 7f1bd3e00640 1 -- 172.21.15.102:0/3037743803 >> 172.21.15.102:0/3037743803 conn(0x7f1bd40727e0 msgr2=0x7f1bd4072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:21.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.944+0000 7f1bd3e00640 1 -- 172.21.15.102:0/3037743803 shutdown_connections 2024-03-18T16:26:21.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.944+0000 7f1bd3e00640 1 -- 172.21.15.102:0/3037743803 wait complete. 2024-03-18T16:26:21.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.946+0000 7f1bd3e00640 1 Processor -- start 2024-03-18T16:26:21.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.946+0000 7f1bd3e00640 1 -- start start 2024-03-18T16:26:21.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.948+0000 7f1bd3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 0x7f1bd4143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:21.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.948+0000 7f1bd3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bd414dad0 0x7f1bd4142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:21.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.948+0000 7f1bd3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bd415b1c0 0x7f1bd41434f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:21.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.948+0000 7f1bd3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd415a9c0 con 0x7f1bd414dad0 2024-03-18T16:26:21.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.948+0000 7f1bd3e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd415ad30 con 0x7f1bd40733b0 2024-03-18T16:26:21.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.948+0000 7f1bd3e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd40c7e10 con 0x7f1bd415b1c0 2024-03-18T16:26:21.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.948+0000 7f1bd2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 0x7f1bd4143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:21.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.949+0000 7f1bd2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 0x7f1bd4143e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:51348/0 (socket says 172.21.15.102:51348) 2024-03-18T16:26:21.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.949+0000 7f1bd2a00640 1 -- 172.21.15.102:0/4272116483 learned_addr learned my addr 172.21.15.102:0/4272116483 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:21.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.949+0000 7f1bd3400640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bd415b1c0 0x7f1bd41434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:21.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.949+0000 7f1bd2000640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bd414dad0 0x7f1bd4142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:21.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.952+0000 7f1bd2a00640 1 -- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bd415b1c0 msgr2=0x7f1bd41434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:21.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.952+0000 7f1bd2a00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bd415b1c0 0x7f1bd41434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:21.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.952+0000 7f1bd2a00640 1 -- 172.21.15.102:0/4272116483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bd414dad0 msgr2=0x7f1bd4142fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:21.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.952+0000 7f1bd2a00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bd414dad0 0x7f1bd4142fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:21.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.952+0000 7f1bd2a00640 1 -- 172.21.15.102:0/4272116483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd4143a30 con 0x7f1bd40733b0 2024-03-18T16:26:21.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.955+0000 7f1bd2a00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 0x7f1bd4143e10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400c3c0 tx=0x7f1bc400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:21.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.958+0000 7f1bcaa00640 1 -- 172.21.15.102:0/4272116483 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4017070 con 0x7f1bd40733b0 2024-03-18T16:26:21.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.958+0000 7f1bcaa00640 1 -- 172.21.15.102:0/4272116483 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc4005ce0 con 0x7f1bd40733b0 2024-03-18T16:26:21.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.958+0000 7f1bcaa00640 1 -- 172.21.15.102:0/4272116483 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc40072f0 con 0x7f1bd40733b0 2024-03-18T16:26:21.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.958+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd414a370 con 0x7f1bd40733b0 2024-03-18T16:26:21.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.959+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd4145ec0 con 0x7f1bd40733b0 2024-03-18T16:26:21.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:21.960+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd4074fd0 con 0x7f1bd40733b0 2024-03-18T16:26:22.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.223+0000 7f1bcaa00640 1 -- 172.21.15.102:0/4272116483 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1bc401a070 con 0x7f1bd40733b0 2024-03-18T16:26:22.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.224+0000 7f1bcaa00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1bb406bc20 0x7f1bb406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:22.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.225+0000 7f1bd2000640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1bb406bc20 0x7f1bb406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:22.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.225+0000 7f1bcaa00640 1 -- 172.21.15.102:0/4272116483 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1bc4056770 con 0x7f1bd40733b0 2024-03-18T16:26:22.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.226+0000 7f1bd2000640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1bb406bc20 0x7f1bb406e0e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0036000 tx=0x7f1bc003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:22.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.236+0000 7f1bcaa00640 1 -- 172.21.15.102:0/4272116483 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1bc4021020 con 0x7f1bd40733b0 2024-03-18T16:26:22.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.547+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bd40c7390 con 0x7f1bd40733b0 2024-03-18T16:26:22.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.569+0000 7f1bcaa00640 1 -- 172.21.15.102:0/4272116483 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1bc40586d0 con 0x7f1bd40733b0 2024-03-18T16:26:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:22.571 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:22.571 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:22.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1bb406bc20 msgr2=0x7f1bb406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1bb406bc20 0x7f1bb406e0e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0036000 tx=0x7f1bc003c040 comp rx=0 tx=0).stop 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 msgr2=0x7f1bd4143e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 0x7f1bd4143e10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400c3c0 tx=0x7f1bc400c880 comp rx=0 tx=0).stop 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 shutdown_connections 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1bb406bc20 0x7f1bb406e0e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1bd415b1c0 0x7f1bd41434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1bd414dad0 0x7f1bd4142fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 --2- 172.21.15.102:0/4272116483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1bd40733b0 0x7f1bd4143e10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.571+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 >> 172.21.15.102:0/4272116483 conn(0x7f1bd40727e0 msgr2=0x7f1bd40c6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.572+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 shutdown_connections 2024-03-18T16:26:22.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:22.572+0000 7f1bd3e00640 1 -- 172.21.15.102:0/4272116483 wait complete. 2024-03-18T16:26:23.195 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:23.195 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:24.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:24.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.371+0000 7f13b6400640 1 Processor -- start 2024-03-18T16:26:24.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.371+0000 7f13b6400640 1 -- start start 2024-03-18T16:26:24.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.373+0000 7f13b6400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13b015a6f0 0x7f13b015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:24.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.373+0000 7f13b6400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015baa0 0x7f13b015bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:24.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.373+0000 7f13b6400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13b015c470 con 0x7f13b015a310 2024-03-18T16:26:24.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.373+0000 7f13b6400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f13b00cd410 con 0x7f13b015a6f0 2024-03-18T16:26:24.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.373+0000 7f13b6400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f13b00ca500 con 0x7f13b015baa0 2024-03-18T16:26:24.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.374+0000 7f13b5a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015baa0 0x7f13b015bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:24.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.374+0000 7f13afe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13b015a6f0 0x7f13b015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:24.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.374+0000 7f13b5a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015baa0 0x7f13b015bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34270/0 (socket says 172.21.15.102:34270) 2024-03-18T16:26:24.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.374+0000 7f13b5a00640 1 -- 172.21.15.102:0/1646812634 learned_addr learned my addr 172.21.15.102:0/1646812634 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:24.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.377+0000 7f13afe00640 1 -- 172.21.15.102:0/1646812634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015baa0 msgr2=0x7f13b015bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:24.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.377+0000 7f13afe00640 1 --2- 172.21.15.102:0/1646812634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015baa0 0x7f13b015bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:24.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.377+0000 7f13afe00640 1 -- 172.21.15.102:0/1646812634 >> v1:172.21.15.102:6789/0 conn(0x7f13b015a310 legacy=0x7f13b006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:24.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.377+0000 7f13afe00640 1 -- 172.21.15.102:0/1646812634 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b015b080 con 0x7f13b015a6f0 2024-03-18T16:26:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.380+0000 7f13afe00640 1 --2- 172.21.15.102:0/1646812634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b015a6f0 0x7f13b015ab40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f139c0097a0 tx=0x7f139c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f346ce058dfc2a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.386+0000 7f13af400640 1 -- 172.21.15.102:0/1646812634 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0315a0 con 0x7f13b015a6f0 2024-03-18T16:26:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.386+0000 7f13af400640 1 -- 172.21.15.102:0/1646812634 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f139c037070 con 0x7f13b015a6f0 2024-03-18T16:26:24.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.386+0000 7f13af400640 1 -- 172.21.15.102:0/1646812634 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c031960 con 0x7f13b015a6f0 2024-03-18T16:26:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.387+0000 7f13b6400640 1 -- 172.21.15.102:0/1646812634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b015a6f0 msgr2=0x7f13b015ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.387+0000 7f13b6400640 1 --2- 172.21.15.102:0/1646812634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b015a6f0 0x7f13b015ab40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f139c0097a0 tx=0x7f139c02b840 comp rx=0 tx=0).stop 2024-03-18T16:26:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.387+0000 7f13b6400640 1 -- 172.21.15.102:0/1646812634 shutdown_connections 2024-03-18T16:26:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.387+0000 7f13b6400640 1 --2- 172.21.15.102:0/1646812634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015baa0 0x7f13b015bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.387+0000 7f13b6400640 1 --2- 172.21.15.102:0/1646812634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b015a6f0 0x7f13b015ab40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.387+0000 7f13b6400640 1 -- 172.21.15.102:0/1646812634 >> 172.21.15.102:0/1646812634 conn(0x7f13b008c1a0 msgr2=0x7f13b0066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:24.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.387+0000 7f13b6400640 1 -- 172.21.15.102:0/1646812634 shutdown_connections 2024-03-18T16:26:24.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.388+0000 7f13b6400640 1 -- 172.21.15.102:0/1646812634 wait complete. 2024-03-18T16:26:24.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.390+0000 7f13b6400640 1 Processor -- start 2024-03-18T16:26:24.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.390+0000 7f13b6400640 1 -- start start 2024-03-18T16:26:24.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.392+0000 7f13b6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b015a310 0x7f13b0157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.392+0000 7f13b6400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015a6f0 0x7f13b00c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.392+0000 7f13b6400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b015baa0 0x7f13b00c5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.392+0000 7f13b6400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f13b00cce50 con 0x7f13b015a310 2024-03-18T16:26:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.392+0000 7f13b6400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13b00caa80 con 0x7f13b015baa0 2024-03-18T16:26:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.392+0000 7f13b6400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f13b00cd130 con 0x7f13b015a6f0 2024-03-18T16:26:24.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.392+0000 7f13afe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015a6f0 0x7f13b00c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.392+0000 7f13b5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b015a310 0x7f13b0157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.393+0000 7f13b5a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b015baa0 0x7f13b00c5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.393+0000 7f13afe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015a6f0 0x7f13b00c4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34282/0 (socket says 172.21.15.102:34282) 2024-03-18T16:26:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.393+0000 7f13afe00640 1 -- 172.21.15.102:0/708555410 learned_addr learned my addr 172.21.15.102:0/708555410 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:24.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.397+0000 7f13b5a00640 1 -- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015a6f0 msgr2=0x7f13b00c4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:24.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.397+0000 7f13b5a00640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015a6f0 0x7f13b00c4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:24.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.397+0000 7f13b5a00640 1 -- 172.21.15.102:0/708555410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b015a310 msgr2=0x7f13b0157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:24.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.397+0000 7f13b5a00640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b015a310 0x7f13b0157740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:24.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.397+0000 7f13b5a00640 1 -- 172.21.15.102:0/708555410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b0150c40 con 0x7f13b015baa0 2024-03-18T16:26:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.400+0000 7f13b5a00640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b015baa0 0x7f13b00c5180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f13a800efc0 tx=0x7f13a800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.403+0000 7f13acc00640 1 -- 172.21.15.102:0/708555410 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a800cd20 con 0x7f13b015baa0 2024-03-18T16:26:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.403+0000 7f13acc00640 1 -- 172.21.15.102:0/708555410 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a8003990 con 0x7f13b015baa0 2024-03-18T16:26:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.403+0000 7f13acc00640 1 -- 172.21.15.102:0/708555410 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8003d80 con 0x7f13b015baa0 2024-03-18T16:26:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.404+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b01f08e0 con 0x7f13b015baa0 2024-03-18T16:26:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.404+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b01f0ea0 con 0x7f13b015baa0 2024-03-18T16:26:24.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.405+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b00cac90 con 0x7f13b015baa0 2024-03-18T16:26:24.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.425+0000 7f13acc00640 1 -- 172.21.15.102:0/708555410 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f13a8005d10 con 0x7f13b015baa0 2024-03-18T16:26:24.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.426+0000 7f13acc00640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f139006bc50 0x7f139006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:24.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.427+0000 7f13acc00640 1 -- 172.21.15.102:0/708555410 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f13a8058100 con 0x7f13b015baa0 2024-03-18T16:26:24.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.427+0000 7f13b5000640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f139006bc50 0x7f139006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:24.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.428+0000 7f13b5000640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f139006bc50 0x7f139006e110 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f13a4006fd0 tx=0x7f13a4009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:24.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.679+0000 7f13acc00640 1 -- 172.21.15.102:0/708555410 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f13b00cac90 con 0x7f13b015baa0 2024-03-18T16:26:24.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.960+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13b0074fd0 con 0x7f13b015baa0 2024-03-18T16:26:24.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.975+0000 7f13acc00640 1 -- 172.21.15.102:0/708555410 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f13a8010040 con 0x7f13b015baa0 2024-03-18T16:26:24.977 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:24.977 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:24.977 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f139006bc50 msgr2=0x7f139006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f139006bc50 0x7f139006e110 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f13a4006fd0 tx=0x7f13a4009060 comp rx=0 tx=0).stop 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b015baa0 msgr2=0x7f13b00c5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b015baa0 0x7f13b00c5180 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f13a800efc0 tx=0x7f13a800c490 comp rx=0 tx=0).stop 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 shutdown_connections 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f139006bc50 0x7f139006e110 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f13a4006fd0 tx=0x7f13a4009060 comp rx=0 tx=0).stop 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13b015baa0 0x7f13b00c5180 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13b015a6f0 0x7f13b00c4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 --2- 172.21.15.102:0/708555410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b015a310 0x7f13b0157740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:24.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 >> 172.21.15.102:0/708555410 conn(0x7f13b008c1a0 msgr2=0x7f13b014ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:24.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 shutdown_connections 2024-03-18T16:26:24.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:24.978+0000 7f13b6400640 1 -- 172.21.15.102:0/708555410 wait complete. 2024-03-18T16:26:25.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:25.607 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:26.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:26.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.780+0000 7f9b55a00640 1 Processor -- start 2024-03-18T16:26:26.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.781+0000 7f9b55a00640 1 -- start start 2024-03-18T16:26:26.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.782+0000 7f9b55a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 0x7f9b5015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:26.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.782+0000 7f9b55a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b500cabd0 con 0x7f9b500cc2f0 2024-03-18T16:26:26.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.782+0000 7f9b55a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b500caea0 con 0x7f9b500cfe70 2024-03-18T16:26:26.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.782+0000 7f9b55a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9b500cb180 con 0x7f9b500d0250 2024-03-18T16:26:26.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.783+0000 7f9b4f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 0x7f9b5015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:26.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.783+0000 7f9b55000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9b500cc2f0 0x7f9b500cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52040/0 (socket says 172.21.15.102:52040) 2024-03-18T16:26:26.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.783+0000 7f9b55000640 1 -- 172.21.15.102:0/4123053675 learned_addr learned my addr 172.21.15.102:0/4123053675 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:26.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.786+0000 7f9b4ea00640 1 -- 172.21.15.102:0/4123053675 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2667259437 0 0) 0x7f9b500caea0 con 0x7f9b500cfe70 2024-03-18T16:26:26.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.786+0000 7f9b4ea00640 1 -- 172.21.15.102:0/4123053675 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b34003520 con 0x7f9b500cfe70 2024-03-18T16:26:26.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.786+0000 7f9b4ea00640 1 -- 172.21.15.102:0/4123053675 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329878421 0 0) 0x7f9b500cabd0 con 0x7f9b500cc2f0 2024-03-18T16:26:26.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.786+0000 7f9b4ea00640 1 -- 172.21.15.102:0/4123053675 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b500caea0 con 0x7f9b500cc2f0 2024-03-18T16:26:26.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.786+0000 7f9b4f400640 1 -- 172.21.15.102:0/4123053675 >> v1:172.21.15.154:6789/0 conn(0x7f9b500cfe70 legacy=0x7f9b500c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:26.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.786+0000 7f9b4f400640 1 -- 172.21.15.102:0/4123053675 >> v1:172.21.15.102:6789/0 conn(0x7f9b500cc2f0 legacy=0x7f9b500cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:26.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.786+0000 7f9b4f400640 1 -- 172.21.15.102:0/4123053675 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b500cb460 con 0x7f9b500d0250 2024-03-18T16:26:26.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.789+0000 7f9b4f400640 1 --2- 172.21.15.102:0/4123053675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 0x7f9b5015ae40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0097d0 tx=0x7f9b3c02b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5219fdee08758b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:26.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.794+0000 7f9b4ea00640 1 -- 172.21.15.102:0/4123053675 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c034070 con 0x7f9b500d0250 2024-03-18T16:26:26.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.794+0000 7f9b4ea00640 1 -- 172.21.15.102:0/4123053675 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b3c030890 con 0x7f9b500d0250 2024-03-18T16:26:26.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.794+0000 7f9b4ea00640 1 -- 172.21.15.102:0/4123053675 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c030c60 con 0x7f9b500d0250 2024-03-18T16:26:26.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.794+0000 7f9b55a00640 1 -- 172.21.15.102:0/4123053675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 msgr2=0x7f9b5015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:26.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.794+0000 7f9b55a00640 1 --2- 172.21.15.102:0/4123053675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 0x7f9b5015ae40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0097d0 tx=0x7f9b3c02b9d0 comp rx=0 tx=0).stop 2024-03-18T16:26:26.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.795+0000 7f9b55a00640 1 -- 172.21.15.102:0/4123053675 shutdown_connections 2024-03-18T16:26:26.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.795+0000 7f9b55a00640 1 --2- 172.21.15.102:0/4123053675 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 0x7f9b5015ae40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:26.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.795+0000 7f9b55a00640 1 -- 172.21.15.102:0/4123053675 >> 172.21.15.102:0/4123053675 conn(0x7f9b5008c3e0 msgr2=0x7f9b5006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:26.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.795+0000 7f9b55a00640 1 -- 172.21.15.102:0/4123053675 shutdown_connections 2024-03-18T16:26:26.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.795+0000 7f9b55a00640 1 -- 172.21.15.102:0/4123053675 wait complete. 2024-03-18T16:26:26.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.797+0000 7f9b55a00640 1 Processor -- start 2024-03-18T16:26:26.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.798+0000 7f9b55a00640 1 -- start start 2024-03-18T16:26:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.799+0000 7f9b55a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b500cc2f0 0x7f9b501ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.799+0000 7f9b55a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b500cfe70 0x7f9b500c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.799+0000 7f9b55a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 0x7f9b501f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.799+0000 7f9b55a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9b500cbd00 con 0x7f9b500cc2f0 2024-03-18T16:26:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.799+0000 7f9b55a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9b500cba20 con 0x7f9b500cfe70 2024-03-18T16:26:26.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.799+0000 7f9b55a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9b500c8330 con 0x7f9b500d0250 2024-03-18T16:26:26.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.800+0000 7f9b4f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b500cfe70 0x7f9b500c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:26.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.800+0000 7f9b4fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b500cc2f0 0x7f9b501ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:26.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.800+0000 7f9b55000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 0x7f9b501f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:26.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.800+0000 7f9b4f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b500cfe70 0x7f9b500c7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:51362/0 (socket says 172.21.15.102:51362) 2024-03-18T16:26:26.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.800+0000 7f9b4f400640 1 -- 172.21.15.102:0/1145768604 learned_addr learned my addr 172.21.15.102:0/1145768604 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:26.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.804+0000 7f9b4f400640 1 -- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 msgr2=0x7f9b501f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:26.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.804+0000 7f9b4f400640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 0x7f9b501f3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:26.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.804+0000 7f9b4f400640 1 -- 172.21.15.102:0/1145768604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b500cc2f0 msgr2=0x7f9b501ef8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:26.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.804+0000 7f9b4f400640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b500cc2f0 0x7f9b501ef8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:26.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.804+0000 7f9b4f400640 1 -- 172.21.15.102:0/1145768604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b501efef0 con 0x7f9b500cfe70 2024-03-18T16:26:26.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.807+0000 7f9b4f400640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b500cfe70 0x7f9b500c7800 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0098f0 tx=0x7f9b3c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:26.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.810+0000 7f9b47e00640 1 -- 172.21.15.102:0/1145768604 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c034040 con 0x7f9b500cfe70 2024-03-18T16:26:26.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.810+0000 7f9b47e00640 1 -- 172.21.15.102:0/1145768604 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b3c03ada0 con 0x7f9b500cfe70 2024-03-18T16:26:26.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.810+0000 7f9b47e00640 1 -- 172.21.15.102:0/1145768604 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c006e10 con 0x7f9b500cfe70 2024-03-18T16:26:26.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.810+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b501f01d0 con 0x7f9b500cfe70 2024-03-18T16:26:26.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.811+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b501f2f40 con 0x7f9b500cfe70 2024-03-18T16:26:26.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.812+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b50074fd0 con 0x7f9b500cfe70 2024-03-18T16:26:26.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.832+0000 7f9b47e00640 1 -- 172.21.15.102:0/1145768604 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9b3c04b060 con 0x7f9b500cfe70 2024-03-18T16:26:26.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.834+0000 7f9b47e00640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b3006bc20 0x7f9b3006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:26.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.834+0000 7f9b47e00640 1 -- 172.21.15.102:0/1145768604 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9b3c0ad250 con 0x7f9b500cfe70 2024-03-18T16:26:26.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.834+0000 7f9b4fe00640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b3006bc20 0x7f9b3006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:26.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:26.835+0000 7f9b4fe00640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b3006bc20 0x7f9b3006e0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9b400011e0 tx=0x7f9b40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:27.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.087+0000 7f9b47e00640 1 -- 172.21.15.102:0/1145768604 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9b3c03f070 con 0x7f9b500cfe70 2024-03-18T16:26:27.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.351+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b50074870 con 0x7f9b500cfe70 2024-03-18T16:26:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.456+0000 7f9b47e00640 1 -- 172.21.15.102:0/1145768604 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f9b3c07b370 con 0x7f9b500cfe70 2024-03-18T16:26:27.458 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:27.458 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:27.458 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.458+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b3006bc20 msgr2=0x7f9b3006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.458+0000 7f9b55a00640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b3006bc20 0x7f9b3006e0e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9b400011e0 tx=0x7f9b40009040 comp rx=0 tx=0).stop 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.458+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b500cfe70 msgr2=0x7f9b500c7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.458+0000 7f9b55a00640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b500cfe70 0x7f9b500c7800 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0098f0 tx=0x7f9b3c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.459+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 shutdown_connections 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.459+0000 7f9b55a00640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9b3006bc20 0x7f9b3006e0e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.459+0000 7f9b55a00640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9b500d0250 0x7f9b501f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.459+0000 7f9b55a00640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b500cfe70 0x7f9b500c7800 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.459+0000 7f9b55a00640 1 --2- 172.21.15.102:0/1145768604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9b500cc2f0 0x7f9b501ef8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.459+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 >> 172.21.15.102:0/1145768604 conn(0x7f9b5008c3e0 msgr2=0x7f9b5015a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.459+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 shutdown_connections 2024-03-18T16:26:27.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:27.459+0000 7f9b55a00640 1 -- 172.21.15.102:0/1145768604 wait complete. 2024-03-18T16:26:28.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:28.087 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:29.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.260+0000 7f3ebda00640 1 Processor -- start 2024-03-18T16:26:29.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.260+0000 7f3ebda00640 1 -- start start 2024-03-18T16:26:29.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.261+0000 7f3ebda00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 0x7f3eb80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:29.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.261+0000 7f3ebda00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3eb80ca5c0 con 0x7f3eb8154770 2024-03-18T16:26:29.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.261+0000 7f3ebda00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3eb80ca860 con 0x7f3eb80ce2f0 2024-03-18T16:26:29.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.261+0000 7f3ebda00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3eb815e7f0 con 0x7f3eb815e290 2024-03-18T16:26:29.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.262+0000 7f3eb7400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 0x7f3eb80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:29.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.262+0000 7f3ebd000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3eb80ce2f0 0x7f3eb8164bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44008/0 (socket says 172.21.15.102:44008) 2024-03-18T16:26:29.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.262+0000 7f3ebd000640 1 -- 172.21.15.102:0/2243904140 learned_addr learned my addr 172.21.15.102:0/2243904140 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:29.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.265+0000 7f3eb6a00640 1 -- 172.21.15.102:0/2243904140 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1745880737 0 0) 0x7f3eb80ca860 con 0x7f3eb80ce2f0 2024-03-18T16:26:29.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.265+0000 7f3eb6a00640 1 -- 172.21.15.102:0/2243904140 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e9c003520 con 0x7f3eb80ce2f0 2024-03-18T16:26:29.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.265+0000 7f3eb7400640 1 -- 172.21.15.102:0/2243904140 >> v1:172.21.15.154:6789/0 conn(0x7f3eb80ce2f0 legacy=0x7f3eb8164bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:29.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.265+0000 7f3eb7400640 1 -- 172.21.15.102:0/2243904140 >> v1:172.21.15.102:6789/0 conn(0x7f3eb8154770 legacy=0x7f3eb815ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:29.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.265+0000 7f3eb7400640 1 -- 172.21.15.102:0/2243904140 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb815a420 con 0x7f3eb815e290 2024-03-18T16:26:29.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.268+0000 7f3eb7400640 1 --2- 172.21.15.102:0/2243904140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 0x7f3eb80cc560 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40097d0 tx=0x7f3ea402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abcd94a85c81efa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:29.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.270+0000 7f3eb6a00640 1 -- 172.21.15.102:0/2243904140 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4031770 con 0x7f3eb815e290 2024-03-18T16:26:29.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.272+0000 7f3eb6a00640 1 -- 172.21.15.102:0/2243904140 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ea4037070 con 0x7f3eb815e290 2024-03-18T16:26:29.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.272+0000 7f3eb6a00640 1 -- 172.21.15.102:0/2243904140 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4031b10 con 0x7f3eb815e290 2024-03-18T16:26:29.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.273+0000 7f3ebda00640 1 -- 172.21.15.102:0/2243904140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 msgr2=0x7f3eb80cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:29.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.273+0000 7f3ebda00640 1 --2- 172.21.15.102:0/2243904140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 0x7f3eb80cc560 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40097d0 tx=0x7f3ea402b810 comp rx=0 tx=0).stop 2024-03-18T16:26:29.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.273+0000 7f3ebda00640 1 -- 172.21.15.102:0/2243904140 shutdown_connections 2024-03-18T16:26:29.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.273+0000 7f3ebda00640 1 --2- 172.21.15.102:0/2243904140 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 0x7f3eb80cc560 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:29.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.273+0000 7f3ebda00640 1 -- 172.21.15.102:0/2243904140 >> 172.21.15.102:0/2243904140 conn(0x7f3eb8066f80 msgr2=0x7f3eb8067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:29.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.273+0000 7f3ebda00640 1 -- 172.21.15.102:0/2243904140 shutdown_connections 2024-03-18T16:26:29.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.273+0000 7f3ebda00640 1 -- 172.21.15.102:0/2243904140 wait complete. 2024-03-18T16:26:29.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.276+0000 7f3ebda00640 1 Processor -- start 2024-03-18T16:26:29.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.276+0000 7f3ebda00640 1 -- start start 2024-03-18T16:26:29.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.277+0000 7f3ebda00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb80ce2f0 0x7f3eb81f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:29.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.277+0000 7f3ebda00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb8154770 0x7f3eb8157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:29.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.277+0000 7f3ebda00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 0x7f3eb8154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:29.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.277+0000 7f3ebda00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb80cdf60 con 0x7f3eb80ce2f0 2024-03-18T16:26:29.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.277+0000 7f3ebda00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb815a790 con 0x7f3eb8154770 2024-03-18T16:26:29.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.277+0000 7f3ebda00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3eb80cab40 con 0x7f3eb815e290 2024-03-18T16:26:29.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.278+0000 7f3eb7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb8154770 0x7f3eb8157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:29.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.278+0000 7f3eb7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb80ce2f0 0x7f3eb81f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:29.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.278+0000 7f3eb7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb8154770 0x7f3eb8157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:51366/0 (socket says 172.21.15.102:51366) 2024-03-18T16:26:29.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.278+0000 7f3eb7400640 1 -- 172.21.15.102:0/2508815608 learned_addr learned my addr 172.21.15.102:0/2508815608 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:29.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.280+0000 7f3ebd000640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 0x7f3eb8154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:29.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.282+0000 7f3eb7400640 1 -- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 msgr2=0x7f3eb8154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:29.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.282+0000 7f3eb7400640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 0x7f3eb8154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:29.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.282+0000 7f3eb7400640 1 -- 172.21.15.102:0/2508815608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb80ce2f0 msgr2=0x7f3eb81f0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:29.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.282+0000 7f3eb7400640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb80ce2f0 0x7f3eb81f0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:29.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.282+0000 7f3eb7400640 1 -- 172.21.15.102:0/2508815608 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb8155310 con 0x7f3eb8154770 2024-03-18T16:26:29.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.284+0000 7f3eb7400640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb8154770 0x7f3eb8157920 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4034040 tx=0x7f3ea40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:29.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.289+0000 7f3eafe00640 1 -- 172.21.15.102:0/2508815608 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4006e10 con 0x7f3eb8154770 2024-03-18T16:26:29.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.289+0000 7f3eafe00640 1 -- 172.21.15.102:0/2508815608 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ea4031570 con 0x7f3eb8154770 2024-03-18T16:26:29.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.289+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb81555f0 con 0x7f3eb8154770 2024-03-18T16:26:29.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.289+0000 7f3eafe00640 1 -- 172.21.15.102:0/2508815608 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4031920 con 0x7f3eb8154770 2024-03-18T16:26:29.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.289+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb81f0540 con 0x7f3eb8154770 2024-03-18T16:26:29.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.290+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3eb815de20 con 0x7f3eb8154770 2024-03-18T16:26:29.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.314+0000 7f3eafe00640 1 -- 172.21.15.102:0/2508815608 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3ea4031070 con 0x7f3eb8154770 2024-03-18T16:26:29.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.316+0000 7f3eafe00640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e9806bc20 0x7f3e9806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:29.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.316+0000 7f3eb7e00640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e9806bc20 0x7f3e9806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:29.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.317+0000 7f3eb7e00640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e9806bc20 0x7f3e9806e0e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8000f60 tx=0x7f3ea8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:29.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.584+0000 7f3eafe00640 1 -- 172.21.15.102:0/2508815608 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3ea40795d0 con 0x7f3eb8154770 2024-03-18T16:26:29.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.597+0000 7f3eafe00640 1 -- 172.21.15.102:0/2508815608 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3eb815de20 con 0x7f3eb8154770 2024-03-18T16:26:29.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.884+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3eb8074fd0 con 0x7f3eb8154770 2024-03-18T16:26:29.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.897+0000 7f3eafe00640 1 -- 172.21.15.102:0/2508815608 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3ea407b530 con 0x7f3eb8154770 2024-03-18T16:26:29.899 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:29.899 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:29.899 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:29.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e9806bc20 msgr2=0x7f3e9806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:29.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e9806bc20 0x7f3e9806e0e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8000f60 tx=0x7f3ea8009040 comp rx=0 tx=0).stop 2024-03-18T16:26:29.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb8154770 msgr2=0x7f3eb8157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:29.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb8154770 0x7f3eb8157920 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4034040 tx=0x7f3ea40046c0 comp rx=0 tx=0).stop 2024-03-18T16:26:29.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 shutdown_connections 2024-03-18T16:26:29.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e9806bc20 0x7f3e9806e0e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:29.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3eb815e290 0x7f3eb8154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:29.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb8154770 0x7f3eb8157920 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:29.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 --2- 172.21.15.102:0/2508815608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3eb80ce2f0 0x7f3eb81f0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:29.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 >> 172.21.15.102:0/2508815608 conn(0x7f3eb8066f80 msgr2=0x7f3eb80caea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:29.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 shutdown_connections 2024-03-18T16:26:29.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:29.899+0000 7f3ebda00640 1 -- 172.21.15.102:0/2508815608 wait complete. 2024-03-18T16:26:30.522 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:30.522 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:31.523 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:31.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.696+0000 7f7157e00640 1 Processor -- start 2024-03-18T16:26:31.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.696+0000 7f7157e00640 1 -- start start 2024-03-18T16:26:31.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.698+0000 7f7157e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7158158420 0x7f715815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:31.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.698+0000 7f7157e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f71580c4fa0 0x7f71580c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:31.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.698+0000 7f7157e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71580c72d0 con 0x7f7158141650 2024-03-18T16:26:31.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.698+0000 7f7157e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f715815c8e0 con 0x7f7158158420 2024-03-18T16:26:31.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.698+0000 7f7157e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7158159370 con 0x7f71580c4fa0 2024-03-18T16:26:31.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.699+0000 7f7157400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f71580c4fa0 0x7f71580c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:31.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.699+0000 7f7156000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7158158420 0x7f715815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:31.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.699+0000 7f7157400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f71580c4fa0 0x7f71580c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35788/0 (socket says 172.21.15.102:35788) 2024-03-18T16:26:31.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.699+0000 7f7157400640 1 -- 172.21.15.102:0/3690663222 learned_addr learned my addr 172.21.15.102:0/3690663222 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:31.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.702+0000 7f7157400640 1 -- 172.21.15.102:0/3690663222 >> v2:172.21.15.102:3300/0 conn(0x7f7158158420 msgr2=0x7f715815c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:31.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.702+0000 7f7157400640 1 --2- 172.21.15.102:0/3690663222 >> v2:172.21.15.102:3300/0 conn(0x7f7158158420 0x7f715815c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:31.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.702+0000 7f7157400640 1 -- 172.21.15.102:0/3690663222 >> v1:172.21.15.102:6789/0 conn(0x7f7158141650 legacy=0x7f71580c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:31.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.702+0000 7f7157400640 1 -- 172.21.15.102:0/3690663222 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7158158840 con 0x7f71580c4fa0 2024-03-18T16:26:31.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.705+0000 7f7157400640 1 --2- 172.21.15.102:0/3690663222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 0x7f71580c5410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f71580c9720 tx=0x7f715002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=623a6aaec97bcef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:31.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.710+0000 7f7155600640 1 -- 172.21.15.102:0/3690663222 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715002ca80 con 0x7f71580c4fa0 2024-03-18T16:26:31.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.710+0000 7f7155600640 1 -- 172.21.15.102:0/3690663222 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715002cc20 con 0x7f71580c4fa0 2024-03-18T16:26:31.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.710+0000 7f7155600640 1 -- 172.21.15.102:0/3690663222 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71500343f0 con 0x7f71580c4fa0 2024-03-18T16:26:31.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.711+0000 7f7157e00640 1 -- 172.21.15.102:0/3690663222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 msgr2=0x7f71580c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:31.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.711+0000 7f7157e00640 1 --2- 172.21.15.102:0/3690663222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 0x7f71580c5410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f71580c9720 tx=0x7f715002bee0 comp rx=0 tx=0).stop 2024-03-18T16:26:31.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.712+0000 7f7157e00640 1 -- 172.21.15.102:0/3690663222 shutdown_connections 2024-03-18T16:26:31.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.712+0000 7f7157e00640 1 --2- 172.21.15.102:0/3690663222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 0x7f71580c5410 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:31.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.712+0000 7f7157e00640 1 --2- 172.21.15.102:0/3690663222 >> v2:172.21.15.102:3300/0 conn(0x7f7158158420 0x7f715815c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:31.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.712+0000 7f7157e00640 1 -- 172.21.15.102:0/3690663222 >> 172.21.15.102:0/3690663222 conn(0x7f71580cb890 msgr2=0x7f71580cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:31.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.712+0000 7f7157e00640 1 -- 172.21.15.102:0/3690663222 shutdown_connections 2024-03-18T16:26:31.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.712+0000 7f7157e00640 1 -- 172.21.15.102:0/3690663222 wait complete. 2024-03-18T16:26:31.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.714+0000 7f7157e00640 1 Processor -- start 2024-03-18T16:26:31.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.715+0000 7f7157e00640 1 -- start start 2024-03-18T16:26:31.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.716+0000 7f7157e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 0x7f71581384a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:31.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.716+0000 7f7157e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7158141650 0x7f715815b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:31.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.716+0000 7f7157e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7158158420 0x7f715815ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:31.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.716+0000 7f7157e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7158159090 con 0x7f7158141650 2024-03-18T16:26:31.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.716+0000 7f7157e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7158158db0 con 0x7f71580c4fa0 2024-03-18T16:26:31.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.716+0000 7f7157e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f71580c63c0 con 0x7f7158158420 2024-03-18T16:26:31.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.717+0000 7f7156a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 0x7f71581384a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:31.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.717+0000 7f7157400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7158158420 0x7f715815ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:31.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.717+0000 7f7156a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 0x7f71581384a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35790/0 (socket says 172.21.15.102:35790) 2024-03-18T16:26:31.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.717+0000 7f7156a00640 1 -- 172.21.15.102:0/1515808803 learned_addr learned my addr 172.21.15.102:0/1515808803 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:31.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.717+0000 7f7156000640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7158141650 0x7f715815b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.720+0000 7f7156a00640 1 -- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7158158420 msgr2=0x7f715815ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.720+0000 7f7156a00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7158158420 0x7f715815ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.720+0000 7f7156a00640 1 -- 172.21.15.102:0/1515808803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7158141650 msgr2=0x7f715815b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.720+0000 7f7156a00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7158141650 0x7f715815b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:31.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.720+0000 7f7156a00640 1 -- 172.21.15.102:0/1515808803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71581ec1c0 con 0x7f71580c4fa0 2024-03-18T16:26:31.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.723+0000 7f7156a00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 0x7f71581384a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f714800c440 tx=0x7f714800c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.726+0000 7f714ea00640 1 -- 172.21.15.102:0/1515808803 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148017070 con 0x7f71580c4fa0 2024-03-18T16:26:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.726+0000 7f714ea00640 1 -- 172.21.15.102:0/1515808803 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71480035f0 con 0x7f71580c4fa0 2024-03-18T16:26:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.726+0000 7f714ea00640 1 -- 172.21.15.102:0/1515808803 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71480039e0 con 0x7f71580c4fa0 2024-03-18T16:26:31.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.726+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71581ec4a0 con 0x7f71580c4fa0 2024-03-18T16:26:31.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.727+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71581eca60 con 0x7f71580c4fa0 2024-03-18T16:26:31.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.728+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7158074fd0 con 0x7f71580c4fa0 2024-03-18T16:26:31.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.748+0000 7f714ea00640 1 -- 172.21.15.102:0/1515808803 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f714801a070 con 0x7f71580c4fa0 2024-03-18T16:26:31.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.750+0000 7f714ea00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f713806bc20 0x7f713806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:31.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.750+0000 7f7156000640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f713806bc20 0x7f713806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:31.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.751+0000 7f7156000640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f713806bc20 0x7f713806e0e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f715813d0a0 tx=0x7f7144009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:31.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:31.993+0000 7f714ea00640 1 -- 172.21.15.102:0/1515808803 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f71480567e0 con 0x7f71580c4fa0 2024-03-18T16:26:32.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.011+0000 7f714ea00640 1 -- 172.21.15.102:0/1515808803 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7148021020 con 0x7f71580c4fa0 2024-03-18T16:26:32.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.343+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7158155e30 con 0x7f71580c4fa0 2024-03-18T16:26:32.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.358+0000 7f714ea00640 1 -- 172.21.15.102:0/1515808803 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f71480010c0 con 0x7f71580c4fa0 2024-03-18T16:26:32.360 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:32.361 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:32.361 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:32.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f713806bc20 msgr2=0x7f713806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:32.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f713806bc20 0x7f713806e0e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f715813d0a0 tx=0x7f7144009190 comp rx=0 tx=0).stop 2024-03-18T16:26:32.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 msgr2=0x7f71581384a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:32.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 0x7f71581384a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f714800c440 tx=0x7f714800c900 comp rx=0 tx=0).stop 2024-03-18T16:26:32.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 shutdown_connections 2024-03-18T16:26:32.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f713806bc20 0x7f713806e0e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:32.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7158158420 0x7f715815ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:32.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7158141650 0x7f715815b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:32.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 --2- 172.21.15.102:0/1515808803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71580c4fa0 0x7f71581384a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:32.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 >> 172.21.15.102:0/1515808803 conn(0x7f71580cb890 msgr2=0x7f7158156290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:32.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 shutdown_connections 2024-03-18T16:26:32.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:32.361+0000 7f7157e00640 1 -- 172.21.15.102:0/1515808803 wait complete. 2024-03-18T16:26:32.985 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:32.985 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:33.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:34.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.157+0000 7fa579a00640 1 Processor -- start 2024-03-18T16:26:34.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.157+0000 7fa579a00640 1 -- start start 2024-03-18T16:26:34.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.158+0000 7fa579a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa57406de50 0x7fa57415ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:34.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.158+0000 7fa579a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa57406bf60 con 0x7fa574154150 2024-03-18T16:26:34.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.158+0000 7fa579a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5740c94b0 con 0x7fa57406b470 2024-03-18T16:26:34.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.158+0000 7fa579a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa5740ca0a0 con 0x7fa57406de50 2024-03-18T16:26:34.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.159+0000 7fa573e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa57406b470 0x7fa57406b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:55836/0 (socket says 172.21.15.102:55836) 2024-03-18T16:26:34.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.159+0000 7fa573e00640 1 -- 172.21.15.102:0/269743450 learned_addr learned my addr 172.21.15.102:0/269743450 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:34.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.159+0000 7fa573400640 1 --2- 172.21.15.102:0/269743450 >> v2:172.21.15.102:3300/0 conn(0x7fa57406de50 0x7fa57415ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:34.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.162+0000 7fa572a00640 1 -- 172.21.15.102:0/269743450 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3266535146 0 0) 0x7fa5740c94b0 con 0x7fa57406b470 2024-03-18T16:26:34.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.162+0000 7fa572a00640 1 -- 172.21.15.102:0/269743450 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa558003520 con 0x7fa57406b470 2024-03-18T16:26:34.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.162+0000 7fa573400640 1 -- 172.21.15.102:0/269743450 >> v1:172.21.15.154:6789/0 conn(0x7fa57406b470 legacy=0x7fa57406b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:34.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.162+0000 7fa573400640 1 -- 172.21.15.102:0/269743450 >> v1:172.21.15.102:6789/0 conn(0x7fa574154150 legacy=0x7fa5740c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:34.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.162+0000 7fa573400640 1 -- 172.21.15.102:0/269743450 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5740c9770 con 0x7fa57406de50 2024-03-18T16:26:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.166+0000 7fa573400640 1 --2- 172.21.15.102:0/269743450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa57406de50 0x7fa57415ad60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa5600097a0 tx=0x7fa56002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9d603f43bf114e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.170+0000 7fa572a00640 1 -- 172.21.15.102:0/269743450 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560035020 con 0x7fa57406de50 2024-03-18T16:26:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.170+0000 7fa572a00640 1 -- 172.21.15.102:0/269743450 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa560031830 con 0x7fa57406de50 2024-03-18T16:26:34.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.170+0000 7fa572a00640 1 -- 172.21.15.102:0/269743450 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560031c00 con 0x7fa57406de50 2024-03-18T16:26:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.171+0000 7fa579a00640 1 -- 172.21.15.102:0/269743450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa57406de50 msgr2=0x7fa57415ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:34.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.171+0000 7fa579a00640 1 --2- 172.21.15.102:0/269743450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa57406de50 0x7fa57415ad60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa5600097a0 tx=0x7fa56002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:26:34.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.172+0000 7fa579a00640 1 -- 172.21.15.102:0/269743450 shutdown_connections 2024-03-18T16:26:34.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.172+0000 7fa579a00640 1 --2- 172.21.15.102:0/269743450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa57406de50 0x7fa57415ad60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:34.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.172+0000 7fa579a00640 1 -- 172.21.15.102:0/269743450 >> 172.21.15.102:0/269743450 conn(0x7fa5740727e0 msgr2=0x7fa574072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:34.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.172+0000 7fa579a00640 1 -- 172.21.15.102:0/269743450 shutdown_connections 2024-03-18T16:26:34.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.172+0000 7fa579a00640 1 -- 172.21.15.102:0/269743450 wait complete. 2024-03-18T16:26:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.174+0000 7fa579a00640 1 Processor -- start 2024-03-18T16:26:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.174+0000 7fa579a00640 1 -- start start 2024-03-18T16:26:34.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.174+0000 7fa579a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa57406b470 0x7fa5741efd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:34.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.175+0000 7fa579a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa57406de50 0x7fa57414f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:34.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.175+0000 7fa579a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa574154150 0x7fa5741e0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:34.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.175+0000 7fa579a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa5740c58a0 con 0x7fa57406de50 2024-03-18T16:26:34.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.175+0000 7fa579a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5740c9d30 con 0x7fa574154150 2024-03-18T16:26:34.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.175+0000 7fa579a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa5740c5a20 con 0x7fa57406b470 2024-03-18T16:26:34.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.175+0000 7fa579000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa574154150 0x7fa5741e0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:34.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.175+0000 7fa573e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa57406b470 0x7fa5741efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:34.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.175+0000 7fa573400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa57406de50 0x7fa57414f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:34.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.176+0000 7fa579000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa574154150 0x7fa5741e0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35794/0 (socket says 172.21.15.102:35794) 2024-03-18T16:26:34.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.176+0000 7fa579000640 1 -- 172.21.15.102:0/1388996595 learned_addr learned my addr 172.21.15.102:0/1388996595 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:34.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.179+0000 7fa579000640 1 -- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa57406b470 msgr2=0x7fa5741efd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:34.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.179+0000 7fa579000640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa57406b470 0x7fa5741efd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:34.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.179+0000 7fa579000640 1 -- 172.21.15.102:0/1388996595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa57406de50 msgr2=0x7fa57414f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:34.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.179+0000 7fa579000640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa57406de50 0x7fa57414f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:34.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.179+0000 7fa579000640 1 -- 172.21.15.102:0/1388996595 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5741566c0 con 0x7fa574154150 2024-03-18T16:26:34.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.182+0000 7fa579000640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa574154150 0x7fa5741e0720 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00b990 tx=0x7fa56c00be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.186+0000 7fa56be00640 1 -- 172.21.15.102:0/1388996595 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c019070 con 0x7fa574154150 2024-03-18T16:26:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.186+0000 7fa56be00640 1 -- 172.21.15.102:0/1388996595 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa56c00cba0 con 0x7fa574154150 2024-03-18T16:26:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.186+0000 7fa56be00640 1 -- 172.21.15.102:0/1388996595 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c014440 con 0x7fa574154150 2024-03-18T16:26:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.186+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa574157c40 con 0x7fa574154150 2024-03-18T16:26:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.186+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa574156c40 con 0x7fa574154150 2024-03-18T16:26:34.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.188+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5740c45b0 con 0x7fa574154150 2024-03-18T16:26:34.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.448+0000 7fa56be00640 1 -- 172.21.15.102:0/1388996595 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa56c01c070 con 0x7fa574154150 2024-03-18T16:26:34.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.451+0000 7fa56be00640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa55406bc20 0x7fa55406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.451+0000 7fa573e00640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa55406bc20 0x7fa55406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.451+0000 7fa56be00640 1 -- 172.21.15.102:0/1388996595 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa56c08af70 con 0x7fa574154150 2024-03-18T16:26:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.452+0000 7fa573e00640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa55406bc20 0x7fa55406e0e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa564005880 tx=0x7fa564009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:34.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.461+0000 7fa56be00640 1 -- 172.21.15.102:0/1388996595 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa5740c45b0 con 0x7fa574154150 2024-03-18T16:26:34.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.692+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa574074fd0 con 0x7fa574154150 2024-03-18T16:26:34.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.705+0000 7fa56be00640 1 -- 172.21.15.102:0/1388996595 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa56c023020 con 0x7fa574154150 2024-03-18T16:26:34.707 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:34.707 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:34.707 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.708+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa55406bc20 msgr2=0x7fa55406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.708+0000 7fa579a00640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa55406bc20 0x7fa55406e0e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa564005880 tx=0x7fa564009040 comp rx=0 tx=0).stop 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.708+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa574154150 msgr2=0x7fa5741e0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.708+0000 7fa579a00640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa574154150 0x7fa5741e0720 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00b990 tx=0x7fa56c00be50 comp rx=0 tx=0).stop 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.709+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 shutdown_connections 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.709+0000 7fa579a00640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa55406bc20 0x7fa55406e0e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.709+0000 7fa579a00640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa574154150 0x7fa5741e0720 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.709+0000 7fa579a00640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa57406de50 0x7fa57414f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.709+0000 7fa579a00640 1 --2- 172.21.15.102:0/1388996595 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa57406b470 0x7fa5741efd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.709+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 >> 172.21.15.102:0/1388996595 conn(0x7fa5740727e0 msgr2=0x7fa5740c7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:34.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.709+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 shutdown_connections 2024-03-18T16:26:34.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:34.709+0000 7fa579a00640 1 -- 172.21.15.102:0/1388996595 wait complete. 2024-03-18T16:26:35.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:35.339 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:36.340 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:36.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.515+0000 7fe61d000640 1 Processor -- start 2024-03-18T16:26:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.515+0000 7fe61d000640 1 -- start start 2024-03-18T16:26:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.516+0000 7fe61d000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe61815a310 0x7fe61806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:36.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.516+0000 7fe61d000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe61815a6f0 0x7fe61815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:36.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.516+0000 7fe61d000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6180ca500 con 0x7fe61815baa0 2024-03-18T16:26:36.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.516+0000 7fe61d000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe6180cd410 con 0x7fe61815a310 2024-03-18T16:26:36.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.516+0000 7fe61d000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe61815b080 con 0x7fe61815a6f0 2024-03-18T16:26:36.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.517+0000 7fe617400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe61815a310 0x7fe61806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.517+0000 7fe617400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe61815a310 0x7fe61806d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42164/0 (socket says 172.21.15.102:42164) 2024-03-18T16:26:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.517+0000 7fe617400640 1 -- 172.21.15.102:0/3468606209 learned_addr learned my addr 172.21.15.102:0/3468606209 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.517+0000 7fe616a00640 1 --2- 172.21.15.102:0/3468606209 >> v2:172.21.15.154:3300/0 conn(0x7fe61815a6f0 0x7fe61815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.520+0000 7fe617400640 1 -- 172.21.15.102:0/3468606209 >> v2:172.21.15.154:3300/0 conn(0x7fe61815a6f0 msgr2=0x7fe61815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.520+0000 7fe617400640 1 --2- 172.21.15.102:0/3468606209 >> v2:172.21.15.154:3300/0 conn(0x7fe61815a6f0 0x7fe61815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.520+0000 7fe617400640 1 -- 172.21.15.102:0/3468606209 >> v1:172.21.15.154:6789/0 conn(0x7fe61815baa0 legacy=0x7fe61815bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.520+0000 7fe617400640 1 -- 172.21.15.102:0/3468606209 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6180ca7a0 con 0x7fe61815a310 2024-03-18T16:26:36.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.524+0000 7fe617400640 1 --2- 172.21.15.102:0/3468606209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe61815a310 0x7fe61806d3e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe6180d0c80 tx=0x7fe60802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0b224339c2db91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:36.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.529+0000 7fe616000640 1 -- 172.21.15.102:0/3468606209 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe608034070 con 0x7fe61815a310 2024-03-18T16:26:36.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.529+0000 7fe616000640 1 -- 172.21.15.102:0/3468606209 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe608030860 con 0x7fe61815a310 2024-03-18T16:26:36.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.529+0000 7fe616000640 1 -- 172.21.15.102:0/3468606209 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe608030c30 con 0x7fe61815a310 2024-03-18T16:26:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.530+0000 7fe61d000640 1 -- 172.21.15.102:0/3468606209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe61815a310 msgr2=0x7fe61806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:36.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.530+0000 7fe61d000640 1 --2- 172.21.15.102:0/3468606209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe61815a310 0x7fe61806d3e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe6180d0c80 tx=0x7fe60802b9a0 comp rx=0 tx=0).stop 2024-03-18T16:26:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.531+0000 7fe61d000640 1 -- 172.21.15.102:0/3468606209 shutdown_connections 2024-03-18T16:26:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.531+0000 7fe61d000640 1 --2- 172.21.15.102:0/3468606209 >> v2:172.21.15.154:3300/0 conn(0x7fe61815a6f0 0x7fe61815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.531+0000 7fe61d000640 1 --2- 172.21.15.102:0/3468606209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe61815a310 0x7fe61806d3e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.531+0000 7fe61d000640 1 -- 172.21.15.102:0/3468606209 >> 172.21.15.102:0/3468606209 conn(0x7fe61808c1a0 msgr2=0x7fe618066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.531+0000 7fe61d000640 1 -- 172.21.15.102:0/3468606209 shutdown_connections 2024-03-18T16:26:36.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.531+0000 7fe61d000640 1 -- 172.21.15.102:0/3468606209 wait complete. 2024-03-18T16:26:36.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.533+0000 7fe61d000640 1 Processor -- start 2024-03-18T16:26:36.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.533+0000 7fe61d000640 1 -- start start 2024-03-18T16:26:36.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.534+0000 7fe61d000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe61815a310 0x7fe618157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:36.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.534+0000 7fe61d000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61815a6f0 0x7fe6181591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:36.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.534+0000 7fe61d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe61815baa0 0x7fe61815d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:36.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.534+0000 7fe61d000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe6180cd130 con 0x7fe61815baa0 2024-03-18T16:26:36.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.534+0000 7fe61d000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe6180cce50 con 0x7fe61815a6f0 2024-03-18T16:26:36.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.534+0000 7fe61d000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe61806d920 con 0x7fe61815a310 2024-03-18T16:26:36.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.535+0000 7fe617400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe61815a310 0x7fe618157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:36.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.535+0000 7fe616a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61815a6f0 0x7fe6181591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:36.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.535+0000 7fe617400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe61815a310 0x7fe618157400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41504/0 (socket says 172.21.15.102:41504) 2024-03-18T16:26:36.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.535+0000 7fe617400640 1 -- 172.21.15.102:0/3977830770 learned_addr learned my addr 172.21.15.102:0/3977830770 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:36.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.535+0000 7fe617e00640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe61815baa0 0x7fe61815d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.538+0000 7fe617400640 1 -- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61815a6f0 msgr2=0x7fe6181591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.538+0000 7fe617400640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61815a6f0 0x7fe6181591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.538+0000 7fe617400640 1 -- 172.21.15.102:0/3977830770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe61815baa0 msgr2=0x7fe61815d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.538+0000 7fe617400640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe61815baa0 0x7fe61815d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:36.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.539+0000 7fe617400640 1 -- 172.21.15.102:0/3977830770 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6180c7230 con 0x7fe61815a310 2024-03-18T16:26:36.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.541+0000 7fe617400640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe61815a310 0x7fe618157400 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe608000c00 tx=0x7fe608004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.545+0000 7fe60f400640 1 -- 172.21.15.102:0/3977830770 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe608048070 con 0x7fe61815a310 2024-03-18T16:26:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.545+0000 7fe60f400640 1 -- 172.21.15.102:0/3977830770 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe608030660 con 0x7fe61815a310 2024-03-18T16:26:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.545+0000 7fe60f400640 1 -- 172.21.15.102:0/3977830770 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe608034040 con 0x7fe61815a310 2024-03-18T16:26:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.545+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe618158320 con 0x7fe61815a310 2024-03-18T16:26:36.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.546+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe618158cd0 con 0x7fe61815a310 2024-03-18T16:26:36.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.547+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe618074fd0 con 0x7fe61815a310 2024-03-18T16:26:36.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.808+0000 7fe60f400640 1 -- 172.21.15.102:0/3977830770 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe60804b070 con 0x7fe61815a310 2024-03-18T16:26:36.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.809+0000 7fe60f400640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f806bc20 0x7fe5f806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:36.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.809+0000 7fe60f400640 1 -- 172.21.15.102:0/3977830770 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe6080acf30 con 0x7fe61815a310 2024-03-18T16:26:36.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.810+0000 7fe616a00640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f806bc20 0x7fe5f806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:36.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.810+0000 7fe616a00640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f806bc20 0x7fe5f806e0e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe6181e6350 tx=0x7fe604009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:36.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:36.820+0000 7fe60f400640 1 -- 172.21.15.102:0/3977830770 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe608052080 con 0x7fe61815a310 2024-03-18T16:26:37.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.156+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6180c7670 con 0x7fe61815a310 2024-03-18T16:26:37.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.214+0000 7fe60f400640 1 -- 172.21.15.102:0/3977830770 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe60807b0e0 con 0x7fe61815a310 2024-03-18T16:26:37.217 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:37.217 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:37.217 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:37.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f806bc20 msgr2=0x7fe5f806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:37.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f806bc20 0x7fe5f806e0e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe6181e6350 tx=0x7fe604009190 comp rx=0 tx=0).stop 2024-03-18T16:26:37.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe61815a310 msgr2=0x7fe618157400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:37.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe61815a310 0x7fe618157400 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe608000c00 tx=0x7fe608004730 comp rx=0 tx=0).stop 2024-03-18T16:26:37.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 shutdown_connections 2024-03-18T16:26:37.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5f806bc20 0x7fe5f806e0e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:37.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe61815baa0 0x7fe61815d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:37.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61815a6f0 0x7fe6181591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:37.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 --2- 172.21.15.102:0/3977830770 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe61815a310 0x7fe618157400 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:37.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 >> 172.21.15.102:0/3977830770 conn(0x7fe61808c1a0 msgr2=0x7fe618151fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:37.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 shutdown_connections 2024-03-18T16:26:37.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:37.218+0000 7fe61d000640 1 -- 172.21.15.102:0/3977830770 wait complete. 2024-03-18T16:26:37.849 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:37.849 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:38.851 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:39.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.030+0000 7f585ce00640 1 Processor -- start 2024-03-18T16:26:39.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.030+0000 7f585ce00640 1 -- start start 2024-03-18T16:26:39.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.031+0000 7f585ce00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f585806b470 0x7f585815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:39.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.031+0000 7f585ce00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f585806de50 0x7f585806b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:39.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.031+0000 7f585ce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806bd90 0x7f5858154150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:39.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.031+0000 7f585ce00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f58580ca0a0 con 0x7f585806b470 2024-03-18T16:26:39.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.031+0000 7f585ce00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f58580c94b0 con 0x7f585806de50 2024-03-18T16:26:39.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.031+0000 7f585ce00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f58580c9770 con 0x7f585806bd90 2024-03-18T16:26:39.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.032+0000 7f5857e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806bd90 0x7f5858154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:39.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.032+0000 7f5857400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f585806b470 0x7f585815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:39.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.032+0000 7f584fe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f585806de50 0x7f585806b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:39.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.033+0000 7f5857e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806bd90 0x7f5858154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41506/0 (socket says 172.21.15.102:41506) 2024-03-18T16:26:39.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.033+0000 7f5857e00640 1 -- 172.21.15.102:0/1549246044 learned_addr learned my addr 172.21.15.102:0/1549246044 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:39.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.035+0000 7f5857e00640 1 -- 172.21.15.102:0/1549246044 >> v2:172.21.15.154:3300/0 conn(0x7f585806de50 msgr2=0x7f585806b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:39.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.035+0000 7f5857e00640 1 --2- 172.21.15.102:0/1549246044 >> v2:172.21.15.154:3300/0 conn(0x7f585806de50 0x7f585806b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.035+0000 7f5857e00640 1 -- 172.21.15.102:0/1549246044 >> v2:172.21.15.102:3300/0 conn(0x7f585806b470 msgr2=0x7f585815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:39.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.035+0000 7f5857e00640 1 --2- 172.21.15.102:0/1549246044 >> v2:172.21.15.102:3300/0 conn(0x7f585806b470 0x7f585815ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.035+0000 7f5857e00640 1 -- 172.21.15.102:0/1549246044 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58580c9a50 con 0x7f585806bd90 2024-03-18T16:26:39.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.038+0000 7f5857e00640 1 --2- 172.21.15.102:0/1549246044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806bd90 0x7f5858154150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f585806c1e0 tx=0x7f585002bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=824ea404e8ac0801 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:39.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.041+0000 7f5856a00640 1 -- 172.21.15.102:0/1549246044 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585002ca80 con 0x7f585806bd90 2024-03-18T16:26:39.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.041+0000 7f5856a00640 1 -- 172.21.15.102:0/1549246044 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f585002cc20 con 0x7f585806bd90 2024-03-18T16:26:39.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.041+0000 7f5856a00640 1 -- 172.21.15.102:0/1549246044 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58500343f0 con 0x7f585806bd90 2024-03-18T16:26:39.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.042+0000 7f585ce00640 1 -- 172.21.15.102:0/1549246044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806bd90 msgr2=0x7f5858154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:39.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.042+0000 7f585ce00640 1 --2- 172.21.15.102:0/1549246044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806bd90 0x7f5858154150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f585806c1e0 tx=0x7f585002bee0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.042+0000 7f585ce00640 1 -- 172.21.15.102:0/1549246044 shutdown_connections 2024-03-18T16:26:39.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.042+0000 7f585ce00640 1 --2- 172.21.15.102:0/1549246044 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806bd90 0x7f5858154150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.042+0000 7f585ce00640 1 --2- 172.21.15.102:0/1549246044 >> v2:172.21.15.154:3300/0 conn(0x7f585806de50 0x7f585806b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.042+0000 7f585ce00640 1 --2- 172.21.15.102:0/1549246044 >> v2:172.21.15.102:3300/0 conn(0x7f585806b470 0x7f585815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.042+0000 7f585ce00640 1 -- 172.21.15.102:0/1549246044 >> 172.21.15.102:0/1549246044 conn(0x7f58580727e0 msgr2=0x7f5858072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:39.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.042+0000 7f585ce00640 1 -- 172.21.15.102:0/1549246044 shutdown_connections 2024-03-18T16:26:39.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.042+0000 7f585ce00640 1 -- 172.21.15.102:0/1549246044 wait complete. 2024-03-18T16:26:39.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.044+0000 7f585ce00640 1 Processor -- start 2024-03-18T16:26:39.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.045+0000 7f585ce00640 1 -- start start 2024-03-18T16:26:39.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.046+0000 7f585ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f585806b470 0x7f5858156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:39.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.046+0000 7f585ce00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f585806bd90 0x7f5858157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:39.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.046+0000 7f585ce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806de50 0x7f58581f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:39.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.046+0000 7f585ce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5858154810 con 0x7f585806b470 2024-03-18T16:26:39.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.046+0000 7f585ce00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5858154690 con 0x7f585806bd90 2024-03-18T16:26:39.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.046+0000 7f585ce00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5858154990 con 0x7f585806de50 2024-03-18T16:26:39.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.047+0000 7f5857400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f585806b470 0x7f5858156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:39.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.047+0000 7f584fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f585806bd90 0x7f5858157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:39.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.047+0000 7f5857e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806de50 0x7f58581f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:39.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.047+0000 7f5857e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806de50 0x7f58581f16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41516/0 (socket says 172.21.15.102:41516) 2024-03-18T16:26:39.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.047+0000 7f5857e00640 1 -- 172.21.15.102:0/3730197442 learned_addr learned my addr 172.21.15.102:0/3730197442 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:39.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.051+0000 7f5857e00640 1 -- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f585806bd90 msgr2=0x7f5858157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:39.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.051+0000 7f5857e00640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f585806bd90 0x7f5858157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.051+0000 7f5857e00640 1 -- 172.21.15.102:0/3730197442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f585806b470 msgr2=0x7f5858156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:39.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.051+0000 7f5857e00640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f585806b470 0x7f5858156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.051+0000 7f5857e00640 1 -- 172.21.15.102:0/3730197442 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58581e1740 con 0x7f585806de50 2024-03-18T16:26:39.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.053+0000 7f5857e00640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806de50 0x7f58581f16d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f585002c4f0 tx=0x7f5850004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.057+0000 7f584f400640 1 -- 172.21.15.102:0/3730197442 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850043020 con 0x7f585806de50 2024-03-18T16:26:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.057+0000 7f584f400640 1 -- 172.21.15.102:0/3730197442 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f585003e450 con 0x7f585806de50 2024-03-18T16:26:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.057+0000 7f584f400640 1 -- 172.21.15.102:0/3730197442 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850009510 con 0x7f585806de50 2024-03-18T16:26:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.057+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58581f0bf0 con 0x7f585806de50 2024-03-18T16:26:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.058+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58580c3ca0 con 0x7f585806de50 2024-03-18T16:26:39.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.059+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5858074fd0 con 0x7f585806de50 2024-03-18T16:26:39.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.079+0000 7f584f400640 1 -- 172.21.15.102:0/3730197442 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5850046020 con 0x7f585806de50 2024-03-18T16:26:39.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.081+0000 7f584f400640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f583406bc20 0x7f583406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:39.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.081+0000 7f5857400640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f583406bc20 0x7f583406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:39.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.081+0000 7f584f400640 1 -- 172.21.15.102:0/3730197442 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5850079e40 con 0x7f585806de50 2024-03-18T16:26:39.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.082+0000 7f5857400640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f583406bc20 0x7f583406e0e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5848009a20 tx=0x7f5848009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:39.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.332+0000 7f584f400640 1 -- 172.21.15.102:0/3730197442 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f585004d020 con 0x7f585806de50 2024-03-18T16:26:39.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.797+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f585806d470 con 0x7f585806de50 2024-03-18T16:26:39.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.810+0000 7f584f400640 1 -- 172.21.15.102:0/3730197442 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f585007bda0 con 0x7f585806de50 2024-03-18T16:26:39.812 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:39.812 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:39.812 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:39.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f583406bc20 msgr2=0x7f583406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:39.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f583406bc20 0x7f583406e0e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5848009a20 tx=0x7f5848009190 comp rx=0 tx=0).stop 2024-03-18T16:26:39.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806de50 msgr2=0x7f58581f16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:39.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806de50 0x7f58581f16d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f585002c4f0 tx=0x7f5850004a40 comp rx=0 tx=0).stop 2024-03-18T16:26:39.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 shutdown_connections 2024-03-18T16:26:39.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f583406bc20 0x7f583406e0e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f585806de50 0x7f58581f16d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f585806bd90 0x7f5858157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 --2- 172.21.15.102:0/3730197442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f585806b470 0x7f5858156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:39.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.812+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 >> 172.21.15.102:0/3730197442 conn(0x7f58580727e0 msgr2=0x7f58580c7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:39.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.813+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 shutdown_connections 2024-03-18T16:26:39.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:39.813+0000 7f585ce00640 1 -- 172.21.15.102:0/3730197442 wait complete. 2024-03-18T16:26:40.441 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:40.441 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:41.443 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:41.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.640+0000 7f08ee400640 1 Processor -- start 2024-03-18T16:26:41.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.640+0000 7f08ee400640 1 -- start start 2024-03-18T16:26:41.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.642+0000 7f08ee400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f08e8154920 0x7f08e80cd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.643+0000 7f08ee400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e80c9ed0 0x7f08e80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.643+0000 7f08ee400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08e80c78a0 con 0x7f08e815af20 2024-03-18T16:26:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.643+0000 7f08ee400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f08e80c7b80 con 0x7f08e8154920 2024-03-18T16:26:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.643+0000 7f08ee400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f08e80c7e60 con 0x7f08e80c9ed0 2024-03-18T16:26:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.644+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e80c9ed0 0x7f08e80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.645+0000 7f08e7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e80c9ed0 0x7f08e80cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:55776/0 (socket says 172.21.15.102:55776) 2024-03-18T16:26:41.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.645+0000 7f08e7e00640 1 -- 172.21.15.102:0/1181374907 learned_addr learned my addr 172.21.15.102:0/1181374907 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:41.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.646+0000 7f08e7400640 1 --2- 172.21.15.102:0/1181374907 >> v2:172.21.15.154:3300/0 conn(0x7f08e8154920 0x7f08e80cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:41.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.648+0000 7f08e7e00640 1 -- 172.21.15.102:0/1181374907 >> v2:172.21.15.154:3300/0 conn(0x7f08e8154920 msgr2=0x7f08e80cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:41.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.648+0000 7f08e7e00640 1 --2- 172.21.15.102:0/1181374907 >> v2:172.21.15.154:3300/0 conn(0x7f08e8154920 0x7f08e80cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:41.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.648+0000 7f08e7e00640 1 -- 172.21.15.102:0/1181374907 >> v1:172.21.15.102:6789/0 conn(0x7f08e815af20 legacy=0x7f08e80c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:41.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.648+0000 7f08e7e00640 1 -- 172.21.15.102:0/1181374907 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e80c8140 con 0x7f08e80c9ed0 2024-03-18T16:26:41.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.651+0000 7f08e7e00640 1 --2- 172.21.15.102:0/1181374907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e80c9ed0 0x7f08e80cd7c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009e50 tx=0x7f08e002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=593c11c1087446eb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:41.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.655+0000 7f08e6000640 1 -- 172.21.15.102:0/1181374907 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e002ca70 con 0x7f08e80c9ed0 2024-03-18T16:26:41.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.656+0000 7f08e6000640 1 -- 172.21.15.102:0/1181374907 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e002cc10 con 0x7f08e80c9ed0 2024-03-18T16:26:41.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.656+0000 7f08e6000640 1 -- 172.21.15.102:0/1181374907 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e00343f0 con 0x7f08e80c9ed0 2024-03-18T16:26:41.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.657+0000 7f08ee400640 1 -- 172.21.15.102:0/1181374907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e80c9ed0 msgr2=0x7f08e80cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:41.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.657+0000 7f08ee400640 1 --2- 172.21.15.102:0/1181374907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e80c9ed0 0x7f08e80cd7c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009e50 tx=0x7f08e002bed0 comp rx=0 tx=0).stop 2024-03-18T16:26:41.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.657+0000 7f08ee400640 1 -- 172.21.15.102:0/1181374907 shutdown_connections 2024-03-18T16:26:41.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.657+0000 7f08ee400640 1 --2- 172.21.15.102:0/1181374907 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e80c9ed0 0x7f08e80cd7c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:41.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.657+0000 7f08ee400640 1 --2- 172.21.15.102:0/1181374907 >> v2:172.21.15.154:3300/0 conn(0x7f08e8154920 0x7f08e80cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:41.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.657+0000 7f08ee400640 1 -- 172.21.15.102:0/1181374907 >> 172.21.15.102:0/1181374907 conn(0x7f08e8072e30 msgr2=0x7f08e8073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:41.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.657+0000 7f08ee400640 1 -- 172.21.15.102:0/1181374907 shutdown_connections 2024-03-18T16:26:41.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.657+0000 7f08ee400640 1 -- 172.21.15.102:0/1181374907 wait complete. 2024-03-18T16:26:41.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.660+0000 7f08ee400640 1 Processor -- start 2024-03-18T16:26:41.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.660+0000 7f08ee400640 1 -- start start 2024-03-18T16:26:41.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.662+0000 7f08ee400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e80c9ed0 0x7f08e815e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:41.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.662+0000 7f08ee400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e8154920 0x7f08e8158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:41.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.662+0000 7f08ee400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e815af20 0x7f08e8159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:41.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.662+0000 7f08ee400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f08e8070f70 con 0x7f08e80c9ed0 2024-03-18T16:26:41.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.662+0000 7f08ee400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08e815b3c0 con 0x7f08e8154920 2024-03-18T16:26:41.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.662+0000 7f08ee400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f08e80710f0 con 0x7f08e815af20 2024-03-18T16:26:41.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.663+0000 7f08e6a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e8154920 0x7f08e8158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:41.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.664+0000 7f08e6a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e8154920 0x7f08e8158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49314/0 (socket says 172.21.15.102:49314) 2024-03-18T16:26:41.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.664+0000 7f08e6a00640 1 -- 172.21.15.102:0/3629266106 learned_addr learned my addr 172.21.15.102:0/3629266106 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:41.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.665+0000 7f08e7400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e80c9ed0 0x7f08e815e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:41.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.667+0000 7f08e7e00640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e815af20 0x7f08e8159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:41.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.668+0000 7f08e6a00640 1 -- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e815af20 msgr2=0x7f08e8159030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:41.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.668+0000 7f08e6a00640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e815af20 0x7f08e8159030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:41.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.668+0000 7f08e6a00640 1 -- 172.21.15.102:0/3629266106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e80c9ed0 msgr2=0x7f08e815e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:41.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.668+0000 7f08e6a00640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e80c9ed0 0x7f08e815e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:41.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.668+0000 7f08e6a00640 1 -- 172.21.15.102:0/3629266106 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e8155b40 con 0x7f08e8154920 2024-03-18T16:26:41.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.671+0000 7f08e6a00640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e8154920 0x7f08e8158af0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f08d400c410 tx=0x7f08d400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.675+0000 7f08df400640 1 -- 172.21.15.102:0/3629266106 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d4017070 con 0x7f08e8154920 2024-03-18T16:26:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.675+0000 7f08df400640 1 -- 172.21.15.102:0/3629266106 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08d40035f0 con 0x7f08e8154920 2024-03-18T16:26:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.675+0000 7f08df400640 1 -- 172.21.15.102:0/3629266106 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d40039e0 con 0x7f08e8154920 2024-03-18T16:26:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.675+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08e8155e20 con 0x7f08e8154920 2024-03-18T16:26:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.675+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08e81563e0 con 0x7f08e8154920 2024-03-18T16:26:41.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.677+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08e8074fd0 con 0x7f08e8154920 2024-03-18T16:26:41.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.696+0000 7f08df400640 1 -- 172.21.15.102:0/3629266106 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f08d401a070 con 0x7f08e8154920 2024-03-18T16:26:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.698+0000 7f08df400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f08c806bc20 0x7f08c806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:41.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.698+0000 7f08e7400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f08c806bc20 0x7f08c806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:41.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.699+0000 7f08e7400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f08c806bc20 0x7f08c806e0e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009aa0 tx=0x7f08d8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:41.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.937+0000 7f08df400640 1 -- 172.21.15.102:0/3629266106 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f08d4056820 con 0x7f08e8154920 2024-03-18T16:26:41.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:41.949+0000 7f08df400640 1 -- 172.21.15.102:0/3629266106 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f08d4021020 con 0x7f08e8154920 2024-03-18T16:26:42.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.186+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08e80cbf00 con 0x7f08e8154920 2024-03-18T16:26:42.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.202+0000 7f08df400640 1 -- 172.21.15.102:0/3629266106 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f08d4058780 con 0x7f08e8154920 2024-03-18T16:26:42.204 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:42.204 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:42.204 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.204+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f08c806bc20 msgr2=0x7f08c806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f08c806bc20 0x7f08c806e0e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f08d8009aa0 tx=0x7f08d8009190 comp rx=0 tx=0).stop 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e8154920 msgr2=0x7f08e8158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e8154920 0x7f08e8158af0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f08d400c410 tx=0x7f08d400c8d0 comp rx=0 tx=0).stop 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 shutdown_connections 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f08c806bc20 0x7f08c806e0e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f08e815af20 0x7f08e8159030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08e8154920 0x7f08e8158af0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 --2- 172.21.15.102:0/3629266106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08e80c9ed0 0x7f08e815e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:42.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 >> 172.21.15.102:0/3629266106 conn(0x7f08e8072e30 msgr2=0x7f08e8073210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:42.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 shutdown_connections 2024-03-18T16:26:42.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:42.205+0000 7f08ee400640 1 -- 172.21.15.102:0/3629266106 wait complete. 2024-03-18T16:26:42.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:42.826 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:43.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:44.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.006+0000 7fbd6ae00640 1 Processor -- start 2024-03-18T16:26:44.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.006+0000 7fbd6ae00640 1 -- start start 2024-03-18T16:26:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.007+0000 7fbd6ae00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd6415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.007+0000 7fbd6ae00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbd640c4fa0 0x7fbd640c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.007+0000 7fbd6ae00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd640c72d0 con 0x7fbd64141650 2024-03-18T16:26:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.007+0000 7fbd6ae00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbd6415c8e0 con 0x7fbd640c4fa0 2024-03-18T16:26:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.007+0000 7fbd6ae00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbd64159370 con 0x7fbd64158420 2024-03-18T16:26:44.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.008+0000 7fbd69000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd6415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:44.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.008+0000 7fbd6a400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbd640c4fa0 0x7fbd640c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:44.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.009+0000 7fbd69a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fbd64141650 0x7fbd640c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44900/0 (socket says 172.21.15.102:44900) 2024-03-18T16:26:44.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.009+0000 7fbd69a00640 1 -- 172.21.15.102:0/70522737 learned_addr learned my addr 172.21.15.102:0/70522737 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:44.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.012+0000 7fbd69000640 1 -- 172.21.15.102:0/70522737 >> v2:172.21.15.154:3300/0 conn(0x7fbd640c4fa0 msgr2=0x7fbd640c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:44.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.012+0000 7fbd69000640 1 --2- 172.21.15.102:0/70522737 >> v2:172.21.15.154:3300/0 conn(0x7fbd640c4fa0 0x7fbd640c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.012+0000 7fbd69000640 1 -- 172.21.15.102:0/70522737 >> v1:172.21.15.154:6789/0 conn(0x7fbd64141650 legacy=0x7fbd640c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:44.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.012+0000 7fbd69000640 1 -- 172.21.15.102:0/70522737 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd64158840 con 0x7fbd64158420 2024-03-18T16:26:44.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.015+0000 7fbd69000640 1 --2- 172.21.15.102:0/70522737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd6415c3a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd500097a0 tx=0x7fbd5002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf3a068c78229b74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:44.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.019+0000 7fbd63e00640 1 -- 172.21.15.102:0/70522737 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50035020 con 0x7fbd64158420 2024-03-18T16:26:44.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.019+0000 7fbd63e00640 1 -- 172.21.15.102:0/70522737 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd50031830 con 0x7fbd64158420 2024-03-18T16:26:44.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.019+0000 7fbd63e00640 1 -- 172.21.15.102:0/70522737 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50031c00 con 0x7fbd64158420 2024-03-18T16:26:44.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.020+0000 7fbd6ae00640 1 -- 172.21.15.102:0/70522737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 msgr2=0x7fbd6415c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:44.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.020+0000 7fbd6ae00640 1 --2- 172.21.15.102:0/70522737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd6415c3a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd500097a0 tx=0x7fbd5002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.020+0000 7fbd6ae00640 1 -- 172.21.15.102:0/70522737 shutdown_connections 2024-03-18T16:26:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.020+0000 7fbd6ae00640 1 --2- 172.21.15.102:0/70522737 >> v2:172.21.15.154:3300/0 conn(0x7fbd640c4fa0 0x7fbd640c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.020+0000 7fbd6ae00640 1 --2- 172.21.15.102:0/70522737 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd6415c3a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.020+0000 7fbd6ae00640 1 -- 172.21.15.102:0/70522737 >> 172.21.15.102:0/70522737 conn(0x7fbd640cb890 msgr2=0x7fbd640cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.020+0000 7fbd6ae00640 1 -- 172.21.15.102:0/70522737 shutdown_connections 2024-03-18T16:26:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.020+0000 7fbd6ae00640 1 -- 172.21.15.102:0/70522737 wait complete. 2024-03-18T16:26:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.023+0000 7fbd6ae00640 1 Processor -- start 2024-03-18T16:26:44.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.023+0000 7fbd6ae00640 1 -- start start 2024-03-18T16:26:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.025+0000 7fbd6ae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd640c4fa0 0x7fbd6413fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.025+0000 7fbd6ae00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd64141650 0x7fbd641f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.025+0000 7fbd6ae00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd64151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.025+0000 7fbd6ae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64159090 con 0x7fbd640c4fa0 2024-03-18T16:26:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.025+0000 7fbd6ae00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64158db0 con 0x7fbd64141650 2024-03-18T16:26:44.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.025+0000 7fbd6ae00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbd640c63c0 con 0x7fbd64158420 2024-03-18T16:26:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.025+0000 7fbd69a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd640c4fa0 0x7fbd6413fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.025+0000 7fbd69000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd64141650 0x7fbd641f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.025+0000 7fbd6a400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd64151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.026+0000 7fbd69a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd640c4fa0 0x7fbd6413fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38546/0 (socket says 172.21.15.102:38546) 2024-03-18T16:26:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.026+0000 7fbd69a00640 1 -- 172.21.15.102:0/3414695287 learned_addr learned my addr 172.21.15.102:0/3414695287 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:44.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.026+0000 7fbd69000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd64141650 0x7fbd641f0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49332/0 (socket says 172.21.15.102:49332) 2024-03-18T16:26:44.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.030+0000 7fbd6a400640 1 -- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd64141650 msgr2=0x7fbd641f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:44.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.030+0000 7fbd6a400640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd64141650 0x7fbd641f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.030+0000 7fbd6a400640 1 -- 172.21.15.102:0/3414695287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd640c4fa0 msgr2=0x7fbd6413fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:44.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.030+0000 7fbd6a400640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd640c4fa0 0x7fbd6413fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.030+0000 7fbd6a400640 1 -- 172.21.15.102:0/3414695287 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd641f03c0 con 0x7fbd64158420 2024-03-18T16:26:44.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.032+0000 7fbd6a400640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd64151550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00efc0 tx=0x7fbd5c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:44.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.037+0000 7fbd61600640 1 -- 172.21.15.102:0/3414695287 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c018020 con 0x7fbd64158420 2024-03-18T16:26:44.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.037+0000 7fbd61600640 1 -- 172.21.15.102:0/3414695287 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd5c014480 con 0x7fbd64158420 2024-03-18T16:26:44.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.037+0000 7fbd61600640 1 -- 172.21.15.102:0/3414695287 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c014870 con 0x7fbd64158420 2024-03-18T16:26:44.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.037+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd641f4650 con 0x7fbd64158420 2024-03-18T16:26:44.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.037+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd64159860 con 0x7fbd64158420 2024-03-18T16:26:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.039+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd64074fd0 con 0x7fbd64158420 2024-03-18T16:26:44.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.328+0000 7fbd61600640 1 -- 172.21.15.102:0/3414695287 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbd5c005d10 con 0x7fbd64158420 2024-03-18T16:26:44.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.329+0000 7fbd61600640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd4406bc20 0x7fbd4406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:44.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.330+0000 7fbd69a00640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd4406bc20 0x7fbd4406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:44.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.331+0000 7fbd61600640 1 -- 172.21.15.102:0/3414695287 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fbd5c058260 con 0x7fbd64158420 2024-03-18T16:26:44.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.331+0000 7fbd69a00640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd4406bc20 0x7fbd4406e0e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbd58004e40 tx=0x7fbd58007f10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:44.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.342+0000 7fbd61600640 1 -- 172.21.15.102:0/3414695287 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbd5c022030 con 0x7fbd64158420 2024-03-18T16:26:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.593+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd640b7f00 con 0x7fbd64158420 2024-03-18T16:26:44.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.607+0000 7fbd61600640 1 -- 172.21.15.102:0/3414695287 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fbd5c010040 con 0x7fbd64158420 2024-03-18T16:26:44.609 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:44.609 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:44.609 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:44.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.609+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd4406bc20 msgr2=0x7fbd4406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.609+0000 7fbd6ae00640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd4406bc20 0x7fbd4406e0e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbd58004e40 tx=0x7fbd58007f10 comp rx=0 tx=0).stop 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.609+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 msgr2=0x7fbd64151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.609+0000 7fbd6ae00640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd64151550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00efc0 tx=0x7fbd5c00c490 comp rx=0 tx=0).stop 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.610+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 shutdown_connections 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.610+0000 7fbd6ae00640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbd4406bc20 0x7fbd4406e0e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.610+0000 7fbd6ae00640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbd64158420 0x7fbd64151550 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.610+0000 7fbd6ae00640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd64141650 0x7fbd641f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.610+0000 7fbd6ae00640 1 --2- 172.21.15.102:0/3414695287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd640c4fa0 0x7fbd6413fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.610+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 >> 172.21.15.102:0/3414695287 conn(0x7fbd640cb890 msgr2=0x7fbd6413ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.610+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 shutdown_connections 2024-03-18T16:26:44.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:44.610+0000 7fbd6ae00640 1 -- 172.21.15.102:0/3414695287 wait complete. 2024-03-18T16:26:45.233 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:45.233 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:46.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:46.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.411+0000 7fcbce800640 1 Processor -- start 2024-03-18T16:26:46.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.411+0000 7fcbce800640 1 -- start start 2024-03-18T16:26:46.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.412+0000 7fcbce800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcbc80ce2f0 0x7fcbc8165bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:46.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.412+0000 7fcbce800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc80ca5c0 con 0x7fcbc8154770 2024-03-18T16:26:46.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.412+0000 7fcbce800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc80ca860 con 0x7fcbc815e290 2024-03-18T16:26:46.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.412+0000 7fcbce800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fcbc815e7f0 con 0x7fcbc80ce2f0 2024-03-18T16:26:46.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.413+0000 7fcbcca00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fcbc815e290 0x7fcbc80cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44914/0 (socket says 172.21.15.102:44914) 2024-03-18T16:26:46.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.413+0000 7fcbcca00640 1 -- 172.21.15.102:0/4162281431 learned_addr learned my addr 172.21.15.102:0/4162281431 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:46.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.413+0000 7fcbcde00640 1 --2- 172.21.15.102:0/4162281431 >> v2:172.21.15.102:3300/0 conn(0x7fcbc80ce2f0 0x7fcbc8165bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:46.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.416+0000 7fcbc7e00640 1 -- 172.21.15.102:0/4162281431 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8767049 0 0) 0x7fcbc80ca860 con 0x7fcbc815e290 2024-03-18T16:26:46.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.416+0000 7fcbc7e00640 1 -- 172.21.15.102:0/4162281431 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbb0003520 con 0x7fcbc815e290 2024-03-18T16:26:46.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.417+0000 7fcbcde00640 1 -- 172.21.15.102:0/4162281431 >> v1:172.21.15.154:6789/0 conn(0x7fcbc815e290 legacy=0x7fcbc80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:46.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.417+0000 7fcbcde00640 1 -- 172.21.15.102:0/4162281431 >> v1:172.21.15.102:6789/0 conn(0x7fcbc8154770 legacy=0x7fcbc815ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:46.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.417+0000 7fcbcde00640 1 -- 172.21.15.102:0/4162281431 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc815a420 con 0x7fcbc80ce2f0 2024-03-18T16:26:46.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.421+0000 7fcbcde00640 1 --2- 172.21.15.102:0/4162281431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbc80ce2f0 0x7fcbc8165bc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009d90 tx=0x7fcbc002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8cb20831c5dddbb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:46.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.427+0000 7fcbc7e00640 1 -- 172.21.15.102:0/4162281431 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0039070 con 0x7fcbc80ce2f0 2024-03-18T16:26:46.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.427+0000 7fcbc7e00640 1 -- 172.21.15.102:0/4162281431 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc002cbd0 con 0x7fcbc80ce2f0 2024-03-18T16:26:46.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.427+0000 7fcbc7e00640 1 -- 172.21.15.102:0/4162281431 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0034440 con 0x7fcbc80ce2f0 2024-03-18T16:26:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.428+0000 7fcbce800640 1 -- 172.21.15.102:0/4162281431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbc80ce2f0 msgr2=0x7fcbc8165bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.428+0000 7fcbce800640 1 --2- 172.21.15.102:0/4162281431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbc80ce2f0 0x7fcbc8165bc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009d90 tx=0x7fcbc002be20 comp rx=0 tx=0).stop 2024-03-18T16:26:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.428+0000 7fcbce800640 1 -- 172.21.15.102:0/4162281431 shutdown_connections 2024-03-18T16:26:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.428+0000 7fcbce800640 1 --2- 172.21.15.102:0/4162281431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbc80ce2f0 0x7fcbc8165bc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.428+0000 7fcbce800640 1 -- 172.21.15.102:0/4162281431 >> 172.21.15.102:0/4162281431 conn(0x7fcbc8066f80 msgr2=0x7fcbc8067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.428+0000 7fcbce800640 1 -- 172.21.15.102:0/4162281431 shutdown_connections 2024-03-18T16:26:46.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.428+0000 7fcbce800640 1 -- 172.21.15.102:0/4162281431 wait complete. 2024-03-18T16:26:46.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.430+0000 7fcbce800640 1 Processor -- start 2024-03-18T16:26:46.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.430+0000 7fcbce800640 1 -- start start 2024-03-18T16:26:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.431+0000 7fcbce800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbc80ce2f0 0x7fcbc80b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.431+0000 7fcbce800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbc8154770 0x7fcbc8159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.431+0000 7fcbce800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbc815e290 0x7fcbc81f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.431+0000 7fcbce800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc80cdf60 con 0x7fcbc815e290 2024-03-18T16:26:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.431+0000 7fcbce800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc815a790 con 0x7fcbc80ce2f0 2024-03-18T16:26:46.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.431+0000 7fcbce800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc80cab40 con 0x7fcbc8154770 2024-03-18T16:26:46.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.432+0000 7fcbcd400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbc80ce2f0 0x7fcbc80b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:46.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.432+0000 7fcbcca00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbc8154770 0x7fcbc8159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:46.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.432+0000 7fcbcde00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbc815e290 0x7fcbc81f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:46.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.432+0000 7fcbcd400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbc80ce2f0 0x7fcbc80b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49336/0 (socket says 172.21.15.102:49336) 2024-03-18T16:26:46.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.432+0000 7fcbcd400640 1 -- 172.21.15.102:0/1438588768 learned_addr learned my addr 172.21.15.102:0/1438588768 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:46.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.436+0000 7fcbcca00640 1 -- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbc80ce2f0 msgr2=0x7fcbc80b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:46.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.436+0000 7fcbcca00640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbc80ce2f0 0x7fcbc80b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:46.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.436+0000 7fcbcca00640 1 -- 172.21.15.102:0/1438588768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbc815e290 msgr2=0x7fcbc81f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:46.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.436+0000 7fcbcca00640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbc815e290 0x7fcbc81f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:46.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.436+0000 7fcbcca00640 1 -- 172.21.15.102:0/1438588768 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc8154e20 con 0x7fcbc8154770 2024-03-18T16:26:46.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.438+0000 7fcbcca00640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbc8154770 0x7fcbc8159720 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400c3c0 tx=0x7fcbb4001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.442+0000 7fcbc5600640 1 -- 172.21.15.102:0/1438588768 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4017070 con 0x7fcbc8154770 2024-03-18T16:26:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.442+0000 7fcbc5600640 1 -- 172.21.15.102:0/1438588768 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb4005ce0 con 0x7fcbc8154770 2024-03-18T16:26:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.442+0000 7fcbc5600640 1 -- 172.21.15.102:0/1438588768 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb40072d0 con 0x7fcbc8154770 2024-03-18T16:26:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.442+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc8155100 con 0x7fcbc8154770 2024-03-18T16:26:46.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.442+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc81556c0 con 0x7fcbc8154770 2024-03-18T16:26:46.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.443+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc8074fd0 con 0x7fcbc8154770 2024-03-18T16:26:46.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.704+0000 7fcbc5600640 1 -- 172.21.15.102:0/1438588768 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcbb401a070 con 0x7fcbc8154770 2024-03-18T16:26:46.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.705+0000 7fcbc5600640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcba806bc20 0x7fcba806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:46.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.705+0000 7fcbcd400640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcba806bc20 0x7fcba806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:46.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.706+0000 7fcbcd400640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcba806bc20 0x7fcba806e0e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc0018b0 tx=0x7fcbbc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:46.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.707+0000 7fcbc5600640 1 -- 172.21.15.102:0/1438588768 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fcbb40567f0 con 0x7fcbc8154770 2024-03-18T16:26:46.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.718+0000 7fcbc5600640 1 -- 172.21.15.102:0/1438588768 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcbb4058750 con 0x7fcbc8154770 2024-03-18T16:26:46.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:46.986+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbc80cd110 con 0x7fcbc8154770 2024-03-18T16:26:47.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.000+0000 7fcbc5600640 1 -- 172.21.15.102:0/1438588768 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fcbb405c7f0 con 0x7fcbc8154770 2024-03-18T16:26:47.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:47.002 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:47.003 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:47.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcba806bc20 msgr2=0x7fcba806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:47.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcba806bc20 0x7fcba806e0e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc0018b0 tx=0x7fcbbc009040 comp rx=0 tx=0).stop 2024-03-18T16:26:47.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbc8154770 msgr2=0x7fcbc8159720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:47.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbc8154770 0x7fcbc8159720 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400c3c0 tx=0x7fcbb4001e70 comp rx=0 tx=0).stop 2024-03-18T16:26:47.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 shutdown_connections 2024-03-18T16:26:47.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcba806bc20 0x7fcba806e0e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:47.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbc815e290 0x7fcbc81f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:47.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcbc8154770 0x7fcbc8159720 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:47.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 --2- 172.21.15.102:0/1438588768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcbc80ce2f0 0x7fcbc80b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:47.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.003+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 >> 172.21.15.102:0/1438588768 conn(0x7fcbc8066f80 msgr2=0x7fcbc814e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:47.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.004+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 shutdown_connections 2024-03-18T16:26:47.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:47.004+0000 7fcbce800640 1 -- 172.21.15.102:0/1438588768 wait complete. 2024-03-18T16:26:47.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:47.636 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:48.637 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:48.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.811+0000 7f951d200640 1 Processor -- start 2024-03-18T16:26:48.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.811+0000 7f951d200640 1 -- start start 2024-03-18T16:26:48.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.812+0000 7f951d200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f95180cca00 0x7f951815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:48.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.812+0000 7f951d200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95180cabd0 con 0x7f95180cfe70 2024-03-18T16:26:48.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.812+0000 7f951d200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95180caea0 con 0x7f95180d0250 2024-03-18T16:26:48.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.812+0000 7f951d200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f95180cb180 con 0x7f95180cca00 2024-03-18T16:26:48.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.813+0000 7f9517400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f95180cfe70 0x7f95180c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43222/0 (socket says 172.21.15.102:43222) 2024-03-18T16:26:48.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.813+0000 7f9517400640 1 -- 172.21.15.102:0/3890997072 learned_addr learned my addr 172.21.15.102:0/3890997072 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:48.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.813+0000 7f9517e00640 1 --2- 172.21.15.102:0/3890997072 >> v2:172.21.15.154:3300/0 conn(0x7f95180cca00 0x7f951815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:48.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.816+0000 7f9517e00640 1 -- 172.21.15.102:0/3890997072 >> v1:172.21.15.154:6789/0 conn(0x7f95180d0250 legacy=0x7f95180cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:48.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.816+0000 7f9517e00640 1 -- 172.21.15.102:0/3890997072 >> v1:172.21.15.102:6789/0 conn(0x7f95180cfe70 legacy=0x7f95180c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:48.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.816+0000 7f9517e00640 1 -- 172.21.15.102:0/3890997072 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95180cb460 con 0x7f95180cca00 2024-03-18T16:26:48.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.820+0000 7f9517e00640 1 --2- 172.21.15.102:0/3890997072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cca00 0x7f951815ae40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9510009d90 tx=0x7f951002bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37f80cae1c317101 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:48.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.824+0000 7f9516000640 1 -- 172.21.15.102:0/3890997072 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9510039070 con 0x7f95180cca00 2024-03-18T16:26:48.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.824+0000 7f9516000640 1 -- 172.21.15.102:0/3890997072 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f951002cd80 con 0x7f95180cca00 2024-03-18T16:26:48.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.824+0000 7f9516000640 1 -- 172.21.15.102:0/3890997072 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9510034660 con 0x7f95180cca00 2024-03-18T16:26:48.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.825+0000 7f951d200640 1 -- 172.21.15.102:0/3890997072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cca00 msgr2=0x7f951815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:48.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.825+0000 7f951d200640 1 --2- 172.21.15.102:0/3890997072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cca00 0x7f951815ae40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9510009d90 tx=0x7f951002bfd0 comp rx=0 tx=0).stop 2024-03-18T16:26:48.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.825+0000 7f951d200640 1 -- 172.21.15.102:0/3890997072 shutdown_connections 2024-03-18T16:26:48.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.825+0000 7f951d200640 1 --2- 172.21.15.102:0/3890997072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cca00 0x7f951815ae40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:48.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.826+0000 7f951d200640 1 -- 172.21.15.102:0/3890997072 >> 172.21.15.102:0/3890997072 conn(0x7f951808c3e0 msgr2=0x7f951806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:48.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.826+0000 7f951d200640 1 -- 172.21.15.102:0/3890997072 shutdown_connections 2024-03-18T16:26:48.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.826+0000 7f951d200640 1 -- 172.21.15.102:0/3890997072 wait complete. 2024-03-18T16:26:48.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.827+0000 7f951d200640 1 Processor -- start 2024-03-18T16:26:48.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.828+0000 7f951d200640 1 -- start start 2024-03-18T16:26:48.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.829+0000 7f951d200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95180cca00 0x7f95181f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:48.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.829+0000 7f951d200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cfe70 0x7f9518159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:48.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.829+0000 7f951d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95180d0250 0x7f9518158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:48.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.829+0000 7f951d200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f95180cbd00 con 0x7f95180d0250 2024-03-18T16:26:48.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.829+0000 7f951d200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f95180cba20 con 0x7f95180cfe70 2024-03-18T16:26:48.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.829+0000 7f951d200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f95180c8330 con 0x7f95180cca00 2024-03-18T16:26:48.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.830+0000 7f9516a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cfe70 0x7f9518159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:48.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.830+0000 7f9517400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95180cca00 0x7f95181f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:48.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.830+0000 7f9517e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95180d0250 0x7f9518158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:48.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.830+0000 7f9516a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cfe70 0x7f9518159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49344/0 (socket says 172.21.15.102:49344) 2024-03-18T16:26:48.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.830+0000 7f9516a00640 1 -- 172.21.15.102:0/4255614529 learned_addr learned my addr 172.21.15.102:0/4255614529 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:48.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.833+0000 7f9516a00640 1 -- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95180cca00 msgr2=0x7f95181f0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:48.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.833+0000 7f9516a00640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95180cca00 0x7f95181f0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:48.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.834+0000 7f9516a00640 1 -- 172.21.15.102:0/4255614529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95180d0250 msgr2=0x7f9518158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:48.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.834+0000 7f9516a00640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95180d0250 0x7f9518158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:48.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.834+0000 7f9516a00640 1 -- 172.21.15.102:0/4255614529 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95181535f0 con 0x7f95180cfe70 2024-03-18T16:26:48.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.836+0000 7f9516a00640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cfe70 0x7f9518159b00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f950400c470 tx=0x7f950400c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:48.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.840+0000 7f950f400640 1 -- 172.21.15.102:0/4255614529 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95040052d0 con 0x7f95180cfe70 2024-03-18T16:26:48.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.840+0000 7f950f400640 1 -- 172.21.15.102:0/4255614529 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9504005490 con 0x7f95180cfe70 2024-03-18T16:26:48.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.840+0000 7f950f400640 1 -- 172.21.15.102:0/4255614529 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504007260 con 0x7f95180cfe70 2024-03-18T16:26:48.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.840+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9518153890 con 0x7f95180cfe70 2024-03-18T16:26:48.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.840+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9518153b50 con 0x7f95180cfe70 2024-03-18T16:26:48.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.842+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f951806cb60 con 0x7f95180cfe70 2024-03-18T16:26:48.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.860+0000 7f950f400640 1 -- 172.21.15.102:0/4255614529 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9504007400 con 0x7f95180cfe70 2024-03-18T16:26:48.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.862+0000 7f950f400640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f806bc50 0x7f94f806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:48.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.862+0000 7f9517400640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f806bc50 0x7f94f806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:48.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:48.863+0000 7f9517400640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f806bc50 0x7f94f806e110 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9508007fd0 tx=0x7f9508007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:49.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.106+0000 7f950f400640 1 -- 172.21.15.102:0/4255614529 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9504056860 con 0x7f95180cfe70 2024-03-18T16:26:49.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.117+0000 7f950f400640 1 -- 172.21.15.102:0/4255614529 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f95040587c0 con 0x7f95180cfe70 2024-03-18T16:26:49.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.383+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f951815b460 con 0x7f95180cfe70 2024-03-18T16:26:49.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.415+0000 7f950f400640 1 -- 172.21.15.102:0/4255614529 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f950405c860 con 0x7f95180cfe70 2024-03-18T16:26:49.418 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:49.418 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:49.418 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f806bc50 msgr2=0x7f94f806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f806bc50 0x7f94f806e110 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9508007fd0 tx=0x7f9508007e80 comp rx=0 tx=0).stop 2024-03-18T16:26:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cfe70 msgr2=0x7f9518159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cfe70 0x7f9518159b00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f950400c470 tx=0x7f950400c930 comp rx=0 tx=0).stop 2024-03-18T16:26:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 shutdown_connections 2024-03-18T16:26:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f806bc50 0x7f94f806e110 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95180d0250 0x7f9518158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95180cfe70 0x7f9518159b00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 --2- 172.21.15.102:0/4255614529 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f95180cca00 0x7f95181f0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:49.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 >> 172.21.15.102:0/4255614529 conn(0x7f951808c3e0 msgr2=0x7f95180cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:49.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 shutdown_connections 2024-03-18T16:26:49.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:49.419+0000 7f951d200640 1 -- 172.21.15.102:0/4255614529 wait complete. 2024-03-18T16:26:50.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:50.050 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:51.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.228+0000 7fa313e00640 1 Processor -- start 2024-03-18T16:26:51.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.228+0000 7fa313e00640 1 -- start start 2024-03-18T16:26:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.229+0000 7fa313e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa31415a310 0x7fa31406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.229+0000 7fa313e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa31415a6f0 0x7fa31415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.229+0000 7fa313e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3140ca500 con 0x7fa31415baa0 2024-03-18T16:26:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.229+0000 7fa313e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa3140cd410 con 0x7fa31415a6f0 2024-03-18T16:26:51.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.229+0000 7fa313e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa31415b080 con 0x7fa31415a310 2024-03-18T16:26:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.230+0000 7fa312a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa31415a310 0x7fa31406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.230+0000 7fa312000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa31415a6f0 0x7fa31415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.230+0000 7fa312a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa31415a310 0x7fa31406d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41810/0 (socket says 172.21.15.102:41810) 2024-03-18T16:26:51.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.230+0000 7fa312a00640 1 -- 172.21.15.102:0/1894059045 learned_addr learned my addr 172.21.15.102:0/1894059045 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.233+0000 7fa312a00640 1 -- 172.21.15.102:0/1894059045 >> v2:172.21.15.102:3300/0 conn(0x7fa31415a6f0 msgr2=0x7fa31415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.233+0000 7fa312a00640 1 --2- 172.21.15.102:0/1894059045 >> v2:172.21.15.102:3300/0 conn(0x7fa31415a6f0 0x7fa31415ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.233+0000 7fa312a00640 1 -- 172.21.15.102:0/1894059045 >> v1:172.21.15.154:6789/0 conn(0x7fa31415baa0 legacy=0x7fa31415bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:51.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.233+0000 7fa312a00640 1 -- 172.21.15.102:0/1894059045 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3140ca7a0 con 0x7fa31415a310 2024-03-18T16:26:51.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.239+0000 7fa312a00640 1 --2- 172.21.15.102:0/1894059045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 0x7fa31406d3e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa314067db0 tx=0x7fa30c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc9564261571c753 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:51.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.241+0000 7fa311600640 1 -- 172.21.15.102:0/1894059045 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c034070 con 0x7fa31415a310 2024-03-18T16:26:51.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.243+0000 7fa311600640 1 -- 172.21.15.102:0/1894059045 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa30c030860 con 0x7fa31415a310 2024-03-18T16:26:51.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.243+0000 7fa311600640 1 -- 172.21.15.102:0/1894059045 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c030c30 con 0x7fa31415a310 2024-03-18T16:26:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.244+0000 7fa313e00640 1 -- 172.21.15.102:0/1894059045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 msgr2=0x7fa31406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.244+0000 7fa313e00640 1 --2- 172.21.15.102:0/1894059045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 0x7fa31406d3e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa314067db0 tx=0x7fa30c02b9a0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.244+0000 7fa313e00640 1 -- 172.21.15.102:0/1894059045 shutdown_connections 2024-03-18T16:26:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.244+0000 7fa313e00640 1 --2- 172.21.15.102:0/1894059045 >> v2:172.21.15.102:3300/0 conn(0x7fa31415a6f0 0x7fa31415ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.244+0000 7fa313e00640 1 --2- 172.21.15.102:0/1894059045 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 0x7fa31406d3e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.244+0000 7fa313e00640 1 -- 172.21.15.102:0/1894059045 >> 172.21.15.102:0/1894059045 conn(0x7fa31408c1a0 msgr2=0x7fa314066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.244+0000 7fa313e00640 1 -- 172.21.15.102:0/1894059045 shutdown_connections 2024-03-18T16:26:51.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.244+0000 7fa313e00640 1 -- 172.21.15.102:0/1894059045 wait complete. 2024-03-18T16:26:51.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.247+0000 7fa313e00640 1 Processor -- start 2024-03-18T16:26:51.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.247+0000 7fa313e00640 1 -- start start 2024-03-18T16:26:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.248+0000 7fa313e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 0x7fa31414e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.248+0000 7fa313e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa31415a6f0 0x7fa314152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.248+0000 7fa313e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa31415baa0 0x7fa314153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.248+0000 7fa313e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa3140cd130 con 0x7fa31415a6f0 2024-03-18T16:26:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.248+0000 7fa313e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3140cce50 con 0x7fa31415a310 2024-03-18T16:26:51.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.248+0000 7fa313e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa31406d920 con 0x7fa31415baa0 2024-03-18T16:26:51.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.249+0000 7fa312a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 0x7fa31414e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:51.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.249+0000 7fa313400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa31415baa0 0x7fa314153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:51.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.249+0000 7fa312a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 0x7fa31414e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41822/0 (socket says 172.21.15.102:41822) 2024-03-18T16:26:51.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.249+0000 7fa312a00640 1 -- 172.21.15.102:0/853815273 learned_addr learned my addr 172.21.15.102:0/853815273 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:51.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.253+0000 7fa312a00640 1 -- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa31415baa0 msgr2=0x7fa314153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:51.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.253+0000 7fa312a00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa31415baa0 0x7fa314153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.253+0000 7fa312a00640 1 -- 172.21.15.102:0/853815273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa31415a6f0 msgr2=0x7fa314152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:51.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.253+0000 7fa312a00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa31415a6f0 0x7fa314152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.253+0000 7fa312a00640 1 -- 172.21.15.102:0/853815273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3141f4650 con 0x7fa31415a310 2024-03-18T16:26:51.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.256+0000 7fa312a00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 0x7fa31414e1a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009960 tx=0x7fa30c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.259+0000 7fa30aa00640 1 -- 172.21.15.102:0/853815273 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c048070 con 0x7fa31415a310 2024-03-18T16:26:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.259+0000 7fa30aa00640 1 -- 172.21.15.102:0/853815273 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa30c030660 con 0x7fa31415a310 2024-03-18T16:26:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.259+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3140c6fb0 con 0x7fa31415a310 2024-03-18T16:26:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.259+0000 7fa30aa00640 1 -- 172.21.15.102:0/853815273 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c034040 con 0x7fa31415a310 2024-03-18T16:26:51.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.259+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3140c7570 con 0x7fa31415a310 2024-03-18T16:26:51.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.261+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa314074fd0 con 0x7fa31415a310 2024-03-18T16:26:51.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.523+0000 7fa30aa00640 1 -- 172.21.15.102:0/853815273 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa30c04b070 con 0x7fa31415a310 2024-03-18T16:26:51.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.525+0000 7fa30aa00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa2f006bc00 0x7fa2f006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:51.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.525+0000 7fa312000640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa2f006bc00 0x7fa2f006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:51.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.526+0000 7fa312000640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa2f006bc00 0x7fa2f006e0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc006a90 tx=0x7fa2fc006a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:51.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.526+0000 7fa30aa00640 1 -- 172.21.15.102:0/853815273 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa30c030810 con 0x7fa31415a310 2024-03-18T16:26:51.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.539+0000 7fa30aa00640 1 -- 172.21.15.102:0/853815273 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa30c052020 con 0x7fa31415a310 2024-03-18T16:26:51.719 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:26:51.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:26:51.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.833+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa31406df20 con 0x7fa31415a310 2024-03-18T16:26:51.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.845+0000 7fa30aa00640 1 -- 172.21.15.102:0/853815273 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa30c07b4f0 con 0x7fa31415a310 2024-03-18T16:26:51.847 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:51.848 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:51.848 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa2f006bc00 msgr2=0x7fa2f006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa2f006bc00 0x7fa2f006e0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc006a90 tx=0x7fa2fc006a20 comp rx=0 tx=0).stop 2024-03-18T16:26:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 msgr2=0x7fa31414e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:51.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 0x7fa31414e1a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009960 tx=0x7fa30c004730 comp rx=0 tx=0).stop 2024-03-18T16:26:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 shutdown_connections 2024-03-18T16:26:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa2f006bc00 0x7fa2f006e0c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa31415baa0 0x7fa314153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa31415a6f0 0x7fa314152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 --2- 172.21.15.102:0/853815273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa31415a310 0x7fa31414e1a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 >> 172.21.15.102:0/853815273 conn(0x7fa31408c1a0 msgr2=0x7fa31415d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 shutdown_connections 2024-03-18T16:26:51.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:51.848+0000 7fa313e00640 1 -- 172.21.15.102:0/853815273 wait complete. 2024-03-18T16:26:52.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:52.472 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:53.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:53.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.645+0000 7f81ffe00640 1 Processor -- start 2024-03-18T16:26:53.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.646+0000 7f81ffe00640 1 -- start start 2024-03-18T16:26:53.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.647+0000 7f81ffe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f820006b470 0x7f820015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.647+0000 7f81ffe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820006bf60 con 0x7f8200154150 2024-03-18T16:26:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.647+0000 7f81ffe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82000c94b0 con 0x7f820006de50 2024-03-18T16:26:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.647+0000 7f81ffe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f82000ca0a0 con 0x7f820006b470 2024-03-18T16:26:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.648+0000 7f81fea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f820006b470 0x7f820015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.648+0000 7f81ff400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8200154150 0x7f82000c5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37760/0 (socket says 172.21.15.102:37760) 2024-03-18T16:26:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.648+0000 7f81ff400640 1 -- 172.21.15.102:0/3508485481 learned_addr learned my addr 172.21.15.102:0/3508485481 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:53.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.651+0000 7f81fea00640 1 -- 172.21.15.102:0/3508485481 >> v1:172.21.15.154:6789/0 conn(0x7f820006de50 legacy=0x7f820006b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:53.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.651+0000 7f81fea00640 1 -- 172.21.15.102:0/3508485481 >> v1:172.21.15.102:6789/0 conn(0x7f8200154150 legacy=0x7f82000c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:53.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.651+0000 7f81fea00640 1 -- 172.21.15.102:0/3508485481 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82000c9770 con 0x7f820006b470 2024-03-18T16:26:53.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.655+0000 7f81fea00640 1 --2- 172.21.15.102:0/3508485481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 0x7f820015ad60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f81f00097a0 tx=0x7f81f002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ec921ac92610d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:53.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.659+0000 7f81fd600640 1 -- 172.21.15.102:0/3508485481 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0035020 con 0x7f820006b470 2024-03-18T16:26:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.659+0000 7f81fd600640 1 -- 172.21.15.102:0/3508485481 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f0031830 con 0x7f820006b470 2024-03-18T16:26:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.659+0000 7f81fd600640 1 -- 172.21.15.102:0/3508485481 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0031c00 con 0x7f820006b470 2024-03-18T16:26:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.659+0000 7f81ffe00640 1 -- 172.21.15.102:0/3508485481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 msgr2=0x7f820015ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.659+0000 7f81ffe00640 1 --2- 172.21.15.102:0/3508485481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 0x7f820015ad60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f81f00097a0 tx=0x7f81f002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:26:53.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.660+0000 7f81ffe00640 1 -- 172.21.15.102:0/3508485481 shutdown_connections 2024-03-18T16:26:53.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.660+0000 7f81ffe00640 1 --2- 172.21.15.102:0/3508485481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 0x7f820015ad60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:53.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.660+0000 7f81ffe00640 1 -- 172.21.15.102:0/3508485481 >> 172.21.15.102:0/3508485481 conn(0x7f82000727e0 msgr2=0x7f8200072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:53.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.660+0000 7f81ffe00640 1 -- 172.21.15.102:0/3508485481 shutdown_connections 2024-03-18T16:26:53.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.660+0000 7f81ffe00640 1 -- 172.21.15.102:0/3508485481 wait complete. 2024-03-18T16:26:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.662+0000 7f81ffe00640 1 Processor -- start 2024-03-18T16:26:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.663+0000 7f81ffe00640 1 -- start start 2024-03-18T16:26:53.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.664+0000 7f81ffe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 0x7f8200157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:53.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.664+0000 7f81ffe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f820006de50 0x7f820015e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:53.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.664+0000 7f81ffe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8200154150 0x7f820015c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:53.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.664+0000 7f81ffe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f82000c58a0 con 0x7f8200154150 2024-03-18T16:26:53.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.664+0000 7f81ffe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82000c9d30 con 0x7f820006b470 2024-03-18T16:26:53.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.664+0000 7f81ffe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f82000c5a20 con 0x7f820006de50 2024-03-18T16:26:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.665+0000 7f81fea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 0x7f8200157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.665+0000 7f81fe000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f820006de50 0x7f820015e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.665+0000 7f81ff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8200154150 0x7f820015c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.665+0000 7f81fea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 0x7f8200157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41846/0 (socket says 172.21.15.102:41846) 2024-03-18T16:26:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.665+0000 7f81fea00640 1 -- 172.21.15.102:0/569469383 learned_addr learned my addr 172.21.15.102:0/569469383 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:53.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.669+0000 7f81fea00640 1 -- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f820006de50 msgr2=0x7f820015e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:53.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.669+0000 7f81fea00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f820006de50 0x7f820015e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:53.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.669+0000 7f81fea00640 1 -- 172.21.15.102:0/569469383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8200154150 msgr2=0x7f820015c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:53.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.669+0000 7f81fea00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8200154150 0x7f820015c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:53.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.669+0000 7f81fea00640 1 -- 172.21.15.102:0/569469383 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820015e610 con 0x7f820006b470 2024-03-18T16:26:53.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.672+0000 7f81fea00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 0x7f8200157750 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f81f0009890 tx=0x7f81f00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:53.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.674+0000 7f81f6a00640 1 -- 172.21.15.102:0/569469383 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0035040 con 0x7f820006b470 2024-03-18T16:26:53.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.674+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820015b6f0 con 0x7f820006b470 2024-03-18T16:26:53.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.674+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820015bcb0 con 0x7f820006b470 2024-03-18T16:26:53.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.676+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8200074fd0 con 0x7f820006b470 2024-03-18T16:26:53.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.937+0000 7f81f6a00640 1 -- 172.21.15.102:0/569469383 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f0031830 con 0x7f820006b470 2024-03-18T16:26:53.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.937+0000 7f81f6a00640 1 -- 172.21.15.102:0/569469383 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f003ac30 con 0x7f820006b470 2024-03-18T16:26:53.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.939+0000 7f81f6a00640 1 -- 172.21.15.102:0/569469383 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f81f0038020 con 0x7f820006b470 2024-03-18T16:26:53.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.941+0000 7f81f6a00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81e006bc00 0x7f81e006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:53.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.941+0000 7f81fe000640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81e006bc00 0x7f81e006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:53.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.942+0000 7f81fe000640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81e006bc00 0x7f81e006e0c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f820015a260 tx=0x7f81ec007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:53.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.944+0000 7f81f6a00640 1 -- 172.21.15.102:0/569469383 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f81f00791d0 con 0x7f820006b470 2024-03-18T16:26:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:53.956+0000 7f81f6a00640 1 -- 172.21.15.102:0/569469383 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f81f007b130 con 0x7f820006b470 2024-03-18T16:26:54.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.293+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f820006baa0 con 0x7f820006b470 2024-03-18T16:26:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.306+0000 7f81f6a00640 1 -- 172.21.15.102:0/569469383 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f81f007f1d0 con 0x7f820006b470 2024-03-18T16:26:54.309 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:54.309 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81e006bc00 msgr2=0x7f81e006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81e006bc00 0x7f81e006e0c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f820015a260 tx=0x7f81ec007e30 comp rx=0 tx=0).stop 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 msgr2=0x7f8200157750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 0x7f8200157750 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f81f0009890 tx=0x7f81f00046c0 comp rx=0 tx=0).stop 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 shutdown_connections 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81e006bc00 0x7f81e006e0c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8200154150 0x7f820015c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f820006de50 0x7f820015e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 --2- 172.21.15.102:0/569469383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820006b470 0x7f8200157750 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.308+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 >> 172.21.15.102:0/569469383 conn(0x7f82000727e0 msgr2=0x7f8200072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.309+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 shutdown_connections 2024-03-18T16:26:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:54.309+0000 7f81ffe00640 1 -- 172.21.15.102:0/569469383 wait complete. 2024-03-18T16:26:54.940 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:54.940 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:55.941 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:56.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.108+0000 7f8dc2e00640 1 Processor -- start 2024-03-18T16:26:56.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.108+0000 7f8dc2e00640 1 -- start start 2024-03-18T16:26:56.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.109+0000 7f8dc2e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8dbc0d0250 0x7f8dbc15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:56.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.109+0000 7f8dc2e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8dbc0cc2f0 0x7f8dbc0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:56.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.109+0000 7f8dc2e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dbc0ccc80 con 0x7f8dbc0cfe70 2024-03-18T16:26:56.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.109+0000 7f8dc2e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8dbc0cabd0 con 0x7f8dbc0cc2f0 2024-03-18T16:26:56.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.109+0000 7f8dc2e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8dbc0caea0 con 0x7f8dbc0d0250 2024-03-18T16:26:56.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.110+0000 7f8dc1000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8dbc0d0250 0x7f8dbc15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:56.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.110+0000 7f8dc2400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8dbc0cc2f0 0x7f8dbc0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:56.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.110+0000 7f8dc1a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8dbc0cfe70 0x7f8dbc0c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37774/0 (socket says 172.21.15.102:37774) 2024-03-18T16:26:56.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.110+0000 7f8dc1a00640 1 -- 172.21.15.102:0/2911450013 learned_addr learned my addr 172.21.15.102:0/2911450013 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:26:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.113+0000 7f8dc1000640 1 -- 172.21.15.102:0/2911450013 >> v2:172.21.15.102:3300/0 conn(0x7f8dbc0cc2f0 msgr2=0x7f8dbc0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.113+0000 7f8dc1000640 1 --2- 172.21.15.102:0/2911450013 >> v2:172.21.15.102:3300/0 conn(0x7f8dbc0cc2f0 0x7f8dbc0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.113+0000 7f8dc1000640 1 -- 172.21.15.102:0/2911450013 >> v1:172.21.15.102:6789/0 conn(0x7f8dbc0cfe70 legacy=0x7f8dbc0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.113+0000 7f8dc1000640 1 -- 172.21.15.102:0/2911450013 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dbc0cb180 con 0x7f8dbc0d0250 2024-03-18T16:26:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.115+0000 7f8dc1000640 1 --2- 172.21.15.102:0/2911450013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dbc0d0250 0x7f8dbc15ae40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc0c6370 tx=0x7f8da802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9c12dc32c4562ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.118+0000 7f8dbbe00640 1 -- 172.21.15.102:0/2911450013 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8034070 con 0x7f8dbc0d0250 2024-03-18T16:26:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.120+0000 7f8dbbe00640 1 -- 172.21.15.102:0/2911450013 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da8030610 con 0x7f8dbc0d0250 2024-03-18T16:26:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.120+0000 7f8dbbe00640 1 -- 172.21.15.102:0/2911450013 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da80309e0 con 0x7f8dbc0d0250 2024-03-18T16:26:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.121+0000 7f8dc2e00640 1 -- 172.21.15.102:0/2911450013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dbc0d0250 msgr2=0x7f8dbc15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.121+0000 7f8dc2e00640 1 --2- 172.21.15.102:0/2911450013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dbc0d0250 0x7f8dbc15ae40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc0c6370 tx=0x7f8da802b960 comp rx=0 tx=0).stop 2024-03-18T16:26:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.121+0000 7f8dc2e00640 1 -- 172.21.15.102:0/2911450013 shutdown_connections 2024-03-18T16:26:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.121+0000 7f8dc2e00640 1 --2- 172.21.15.102:0/2911450013 >> v2:172.21.15.102:3300/0 conn(0x7f8dbc0cc2f0 0x7f8dbc0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.121+0000 7f8dc2e00640 1 --2- 172.21.15.102:0/2911450013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dbc0d0250 0x7f8dbc15ae40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.121+0000 7f8dc2e00640 1 -- 172.21.15.102:0/2911450013 >> 172.21.15.102:0/2911450013 conn(0x7f8dbc08c3e0 msgr2=0x7f8dbc06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.121+0000 7f8dc2e00640 1 -- 172.21.15.102:0/2911450013 shutdown_connections 2024-03-18T16:26:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.121+0000 7f8dc2e00640 1 -- 172.21.15.102:0/2911450013 wait complete. 2024-03-18T16:26:56.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.123+0000 7f8dc2e00640 1 Processor -- start 2024-03-18T16:26:56.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.123+0000 7f8dc2e00640 1 -- start start 2024-03-18T16:26:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.124+0000 7f8dc2e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc0cc2f0 0x7f8dbc0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.124+0000 7f8dc2e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dbc0cfe70 0x7f8dbc1efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.124+0000 7f8dc2e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dbc0d0250 0x7f8dbc1500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.124+0000 7f8dc2e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc0cba20 con 0x7f8dbc0cc2f0 2024-03-18T16:26:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.124+0000 7f8dc2e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc0cb740 con 0x7f8dbc0cfe70 2024-03-18T16:26:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.124+0000 7f8dc2e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8dbc0cbd00 con 0x7f8dbc0d0250 2024-03-18T16:26:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.125+0000 7f8dc1000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dbc0cfe70 0x7f8dbc1efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.125+0000 7f8dc2400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dbc0d0250 0x7f8dbc1500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.125+0000 7f8dc1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc0cc2f0 0x7f8dbc0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.125+0000 7f8dc2400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dbc0d0250 0x7f8dbc1500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38212/0 (socket says 172.21.15.102:38212) 2024-03-18T16:26:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.125+0000 7f8dc2400640 1 -- 172.21.15.102:0/3947851103 learned_addr learned my addr 172.21.15.102:0/3947851103 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:56.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.128+0000 7f8dc2400640 1 -- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dbc0cfe70 msgr2=0x7f8dbc1efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:56.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.128+0000 7f8dc2400640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dbc0cfe70 0x7f8dbc1efcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:56.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.128+0000 7f8dc2400640 1 -- 172.21.15.102:0/3947851103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc0cc2f0 msgr2=0x7f8dbc0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:56.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.128+0000 7f8dc2400640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc0cc2f0 0x7f8dbc0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:56.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.129+0000 7f8dc2400640 1 -- 172.21.15.102:0/3947851103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dbc156970 con 0x7f8dbc0d0250 2024-03-18T16:26:56.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.131+0000 7f8dc2400640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dbc0d0250 0x7f8dbc1500c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009dd0 tx=0x7f8db400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.135+0000 7f8db9600640 1 -- 172.21.15.102:0/3947851103 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4018020 con 0x7f8dbc0d0250 2024-03-18T16:26:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.135+0000 7f8db9600640 1 -- 172.21.15.102:0/3947851103 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db4014480 con 0x7f8dbc0d0250 2024-03-18T16:26:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.135+0000 7f8db9600640 1 -- 172.21.15.102:0/3947851103 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4014870 con 0x7f8dbc0d0250 2024-03-18T16:26:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.135+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dbc1e03b0 con 0x7f8dbc0d0250 2024-03-18T16:26:56.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.135+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dbc0c4b40 con 0x7f8dbc0d0250 2024-03-18T16:26:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.137+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dbc074fd0 con 0x7f8dbc0d0250 2024-03-18T16:26:56.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.156+0000 7f8db9600640 1 -- 172.21.15.102:0/3947851103 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8db4005d10 con 0x7f8dbc0d0250 2024-03-18T16:26:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.157+0000 7f8db9600640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8d9c06bc20 0x7f8d9c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.158+0000 7f8dc1a00640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8d9c06bc20 0x7f8d9c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.158+0000 7f8db9600640 1 -- 172.21.15.102:0/3947851103 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8db4058120 con 0x7f8dbc0d0250 2024-03-18T16:26:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.158+0000 7f8dc1a00640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8d9c06bc20 0x7f8d9c06e0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8db0007fd0 tx=0x7f8db0007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:56.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.411+0000 7f8db9600640 1 -- 172.21.15.102:0/3947851103 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8db4022070 con 0x7f8dbc0d0250 2024-03-18T16:26:56.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.642+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dbc159c20 con 0x7f8dbc0d0250 2024-03-18T16:26:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.659+0000 7f8db9600640 1 -- 172.21.15.102:0/3947851103 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8db4010040 con 0x7f8dbc0d0250 2024-03-18T16:26:56.662 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:56.662 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:56.662 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8d9c06bc20 msgr2=0x7f8d9c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8d9c06bc20 0x7f8d9c06e0e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8db0007fd0 tx=0x7f8db0007e80 comp rx=0 tx=0).stop 2024-03-18T16:26:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dbc0d0250 msgr2=0x7f8dbc1500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dbc0d0250 0x7f8dbc1500c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009dd0 tx=0x7f8db400c490 comp rx=0 tx=0).stop 2024-03-18T16:26:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 shutdown_connections 2024-03-18T16:26:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8d9c06bc20 0x7f8d9c06e0e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8dbc0d0250 0x7f8dbc1500c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:56.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8dbc0cfe70 0x7f8dbc1efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:56.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 --2- 172.21.15.102:0/3947851103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc0cc2f0 0x7f8dbc0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:56.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 >> 172.21.15.102:0/3947851103 conn(0x7f8dbc08c3e0 msgr2=0x7f8dbc06c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:56.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 shutdown_connections 2024-03-18T16:26:56.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:56.661+0000 7f8dc2e00640 1 -- 172.21.15.102:0/3947851103 wait complete. 2024-03-18T16:26:57.283 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:57.284 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:26:58.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:26:58.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.457+0000 7ff1bbe00640 1 Processor -- start 2024-03-18T16:26:58.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.457+0000 7ff1bbe00640 1 -- start start 2024-03-18T16:26:58.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.458+0000 7ff1bbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff1bc0cfe70 0x7ff1bc15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:58.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.458+0000 7ff1bbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1bc0cc2f0 0x7ff1bc0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:58.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.458+0000 7ff1bbe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1bc0ccc80 con 0x7ff1bc0d0250 2024-03-18T16:26:58.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.458+0000 7ff1bbe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff1bc0cabd0 con 0x7ff1bc0cfe70 2024-03-18T16:26:58.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.458+0000 7ff1bbe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff1bc0caea0 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.459+0000 7ff1bb400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1bc0cc2f0 0x7ff1bc0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:58.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.459+0000 7ff1baa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff1bc0cfe70 0x7ff1bc15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:58.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.460+0000 7ff1bb400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1bc0cc2f0 0x7ff1bc0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41878/0 (socket says 172.21.15.102:41878) 2024-03-18T16:26:58.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.460+0000 7ff1bb400640 1 -- 172.21.15.102:0/3654663314 learned_addr learned my addr 172.21.15.102:0/3654663314 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:58.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.463+0000 7ff1baa00640 1 -- 172.21.15.102:0/3654663314 >> v2:172.21.15.154:3300/0 conn(0x7ff1bc0cc2f0 msgr2=0x7ff1bc0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:26:58.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.463+0000 7ff1baa00640 1 --2- 172.21.15.102:0/3654663314 >> v2:172.21.15.154:3300/0 conn(0x7ff1bc0cc2f0 0x7ff1bc0cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:58.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.463+0000 7ff1baa00640 1 -- 172.21.15.102:0/3654663314 >> v1:172.21.15.154:6789/0 conn(0x7ff1bc0d0250 legacy=0x7ff1bc0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:58.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.463+0000 7ff1baa00640 1 -- 172.21.15.102:0/3654663314 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc0cb180 con 0x7ff1bc0cfe70 2024-03-18T16:26:58.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.466+0000 7ff1baa00640 1 --2- 172.21.15.102:0/3654663314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1bc0cfe70 0x7ff1bc15ae40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0ca160 tx=0x7ff1ac02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f280314d324c7223 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.471+0000 7ff1b9600640 1 -- 172.21.15.102:0/3654663314 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac034070 con 0x7ff1bc0cfe70 2024-03-18T16:26:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.471+0000 7ff1b9600640 1 -- 172.21.15.102:0/3654663314 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ac030610 con 0x7ff1bc0cfe70 2024-03-18T16:26:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.471+0000 7ff1b9600640 1 -- 172.21.15.102:0/3654663314 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac0309e0 con 0x7ff1bc0cfe70 2024-03-18T16:26:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.472+0000 7ff1bbe00640 1 -- 172.21.15.102:0/3654663314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1bc0cfe70 msgr2=0x7ff1bc15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.472+0000 7ff1bbe00640 1 --2- 172.21.15.102:0/3654663314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1bc0cfe70 0x7ff1bc15ae40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0ca160 tx=0x7ff1ac02b960 comp rx=0 tx=0).stop 2024-03-18T16:26:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.472+0000 7ff1bbe00640 1 -- 172.21.15.102:0/3654663314 shutdown_connections 2024-03-18T16:26:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.472+0000 7ff1bbe00640 1 --2- 172.21.15.102:0/3654663314 >> v2:172.21.15.154:3300/0 conn(0x7ff1bc0cc2f0 0x7ff1bc0cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.472+0000 7ff1bbe00640 1 --2- 172.21.15.102:0/3654663314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1bc0cfe70 0x7ff1bc15ae40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.472+0000 7ff1bbe00640 1 -- 172.21.15.102:0/3654663314 >> 172.21.15.102:0/3654663314 conn(0x7ff1bc08c3e0 msgr2=0x7ff1bc06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.472+0000 7ff1bbe00640 1 -- 172.21.15.102:0/3654663314 shutdown_connections 2024-03-18T16:26:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.472+0000 7ff1bbe00640 1 -- 172.21.15.102:0/3654663314 wait complete. 2024-03-18T16:26:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.474+0000 7ff1bbe00640 1 Processor -- start 2024-03-18T16:26:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.474+0000 7ff1bbe00640 1 -- start start 2024-03-18T16:26:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.475+0000 7ff1bbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1bc0cc2f0 0x7ff1bc1e3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.475+0000 7ff1bbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1bc0cfe70 0x7ff1bc0c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.475+0000 7ff1bbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1bc0d0250 0x7ff1bc1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.475+0000 7ff1bbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff1bc0cba20 con 0x7ff1bc0d0250 2024-03-18T16:26:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.475+0000 7ff1bbe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1bc0cb740 con 0x7ff1bc0cfe70 2024-03-18T16:26:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.475+0000 7ff1bbe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff1bc0cbd00 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.476+0000 7ff1baa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1bc0cc2f0 0x7ff1bc1e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:58.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.476+0000 7ff1baa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1bc0cc2f0 0x7ff1bc1e3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38218/0 (socket says 172.21.15.102:38218) 2024-03-18T16:26:58.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.476+0000 7ff1baa00640 1 -- 172.21.15.102:0/730819189 learned_addr learned my addr 172.21.15.102:0/730819189 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:26:58.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.477+0000 7ff1bb400640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1bc0d0250 0x7ff1bc1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:58.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.478+0000 7ff1ba000640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1bc0cfe70 0x7ff1bc0c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.480+0000 7ff1baa00640 1 -- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1bc0cfe70 msgr2=0x7ff1bc0c5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.480+0000 7ff1baa00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1bc0cfe70 0x7ff1bc0c5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.480+0000 7ff1baa00640 1 -- 172.21.15.102:0/730819189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1bc0d0250 msgr2=0x7ff1bc1552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.480+0000 7ff1baa00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1bc0d0250 0x7ff1bc1552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.480+0000 7ff1baa00640 1 -- 172.21.15.102:0/730819189 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc1f0320 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.483+0000 7ff1baa00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1bc0cc2f0 0x7ff1bc1e3160 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac009890 tx=0x7ff1ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:58.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.487+0000 7ff1b2a00640 1 -- 172.21.15.102:0/730819189 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac034040 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.487+0000 7ff1b2a00640 1 -- 172.21.15.102:0/730819189 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ac039910 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.487+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1bc1f05e0 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.488+0000 7ff1b2a00640 1 -- 172.21.15.102:0/730819189 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac039d00 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.488+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1bc1f0e80 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.489+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1bc074fd0 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.510+0000 7ff1b2a00640 1 -- 172.21.15.102:0/730819189 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff1ac037070 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.511+0000 7ff1b2a00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff19c06bc50 0x7ff19c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:26:58.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.511+0000 7ff1b2a00640 1 -- 172.21.15.102:0/730819189 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff1ac079860 con 0x7ff1bc0cc2f0 2024-03-18T16:26:58.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.511+0000 7ff1ba000640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff19c06bc50 0x7ff19c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:26:58.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.512+0000 7ff1ba000640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff19c06bc50 0x7ff19c06e110 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0c5cf0 tx=0x7ff1a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:26:58.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:58.760+0000 7ff1b2a00640 1 -- 172.21.15.102:0/730819189 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff1ac07b7c0 con 0x7ff1bc0cc2f0 2024-03-18T16:26:59.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.008+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1bc15a790 con 0x7ff1bc0cc2f0 2024-03-18T16:26:59.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.137+0000 7ff1b2a00640 1 -- 172.21.15.102:0/730819189 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff1ac07f860 con 0x7ff1bc0cc2f0 2024-03-18T16:26:59.141 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:26:59.141 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:26:59.141 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:26:59.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff19c06bc50 msgr2=0x7ff19c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:59.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff19c06bc50 0x7ff19c06e110 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc0c5cf0 tx=0x7ff1a8009040 comp rx=0 tx=0).stop 2024-03-18T16:26:59.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1bc0cc2f0 msgr2=0x7ff1bc1e3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:26:59.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1bc0cc2f0 0x7ff1bc1e3160 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac009890 tx=0x7ff1ac0046c0 comp rx=0 tx=0).stop 2024-03-18T16:26:59.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 shutdown_connections 2024-03-18T16:26:59.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff19c06bc50 0x7ff19c06e110 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:59.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff1bc0d0250 0x7ff1bc1552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:59.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1bc0cfe70 0x7ff1bc0c5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:59.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 --2- 172.21.15.102:0/730819189 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff1bc0cc2f0 0x7ff1bc1e3160 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:26:59.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 >> 172.21.15.102:0/730819189 conn(0x7ff1bc08c3e0 msgr2=0x7ff1bc158860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:26:59.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.141+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 shutdown_connections 2024-03-18T16:26:59.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:26:59.142+0000 7ff1bbe00640 1 -- 172.21.15.102:0/730819189 wait complete. 2024-03-18T16:26:59.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:26:59.772 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:00.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:00.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.947+0000 7fa5dfe00640 1 Processor -- start 2024-03-18T16:27:00.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.948+0000 7fa5dfe00640 1 -- start start 2024-03-18T16:27:00.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.949+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e00c49a0 0x7fa5e0072110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:00.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.949+0000 7fa5dfe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5e006f200 con 0x7fa5e0154810 2024-03-18T16:27:00.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.949+0000 7fa5dfe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5e006f4e0 con 0x7fa5e015d9a0 2024-03-18T16:27:00.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.949+0000 7fa5dfe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e006f7c0 con 0x7fa5e00c49a0 2024-03-18T16:27:00.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.950+0000 7fa5df400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa5e015d9a0 0x7fa5e015de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:45070/0 (socket says 172.21.15.102:45070) 2024-03-18T16:27:00.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.950+0000 7fa5df400640 1 -- 172.21.15.102:0/3213583941 learned_addr learned my addr 172.21.15.102:0/3213583941 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:00.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.950+0000 7fa5de000640 1 --2- 172.21.15.102:0/3213583941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e00c49a0 0x7fa5e0072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:00.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.953+0000 7fa5dd600640 1 -- 172.21.15.102:0/3213583941 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 382083494 0 0) 0x7fa5e006f4e0 con 0x7fa5e015d9a0 2024-03-18T16:27:00.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.953+0000 7fa5dd600640 1 -- 172.21.15.102:0/3213583941 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5c4003520 con 0x7fa5e015d9a0 2024-03-18T16:27:00.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.953+0000 7fa5de000640 1 -- 172.21.15.102:0/3213583941 >> v1:172.21.15.154:6789/0 conn(0x7fa5e015d9a0 legacy=0x7fa5e015de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:00.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.953+0000 7fa5de000640 1 -- 172.21.15.102:0/3213583941 >> v1:172.21.15.102:6789/0 conn(0x7fa5e0154810 legacy=0x7fa5e00c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:00.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.953+0000 7fa5de000640 1 -- 172.21.15.102:0/3213583941 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e006faa0 con 0x7fa5e00c49a0 2024-03-18T16:27:00.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.956+0000 7fa5de000640 1 --2- 172.21.15.102:0/3213583941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e00c49a0 0x7fa5e0072110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0097d0 tx=0x7fa5cc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f9d89bc610700b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.961+0000 7fa5dd600640 1 -- 172.21.15.102:0/3213583941 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc035020 con 0x7fa5e00c49a0 2024-03-18T16:27:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.961+0000 7fa5dd600640 1 -- 172.21.15.102:0/3213583941 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5cc031810 con 0x7fa5e00c49a0 2024-03-18T16:27:00.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.961+0000 7fa5dd600640 1 -- 172.21.15.102:0/3213583941 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc031be0 con 0x7fa5e00c49a0 2024-03-18T16:27:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.962+0000 7fa5dfe00640 1 -- 172.21.15.102:0/3213583941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e00c49a0 msgr2=0x7fa5e0072110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.962+0000 7fa5dfe00640 1 --2- 172.21.15.102:0/3213583941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e00c49a0 0x7fa5e0072110 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0097d0 tx=0x7fa5cc02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:27:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.962+0000 7fa5dfe00640 1 -- 172.21.15.102:0/3213583941 shutdown_connections 2024-03-18T16:27:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.962+0000 7fa5dfe00640 1 --2- 172.21.15.102:0/3213583941 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e00c49a0 0x7fa5e0072110 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.962+0000 7fa5dfe00640 1 -- 172.21.15.102:0/3213583941 >> 172.21.15.102:0/3213583941 conn(0x7fa5e00cb890 msgr2=0x7fa5e00cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.962+0000 7fa5dfe00640 1 -- 172.21.15.102:0/3213583941 shutdown_connections 2024-03-18T16:27:00.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.962+0000 7fa5dfe00640 1 -- 172.21.15.102:0/3213583941 wait complete. 2024-03-18T16:27:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.965+0000 7fa5dfe00640 1 Processor -- start 2024-03-18T16:27:00.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.965+0000 7fa5dfe00640 1 -- start start 2024-03-18T16:27:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.966+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5e00c49a0 0x7fa5e00c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.966+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e0154810 0x7fa5e00c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.966+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e015d9a0 0x7fa5e01e47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.966+0000 7fa5dfe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e00cc6a0 con 0x7fa5e00c49a0 2024-03-18T16:27:00.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.966+0000 7fa5dfe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e006c910 con 0x7fa5e0154810 2024-03-18T16:27:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.966+0000 7fa5dfe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e015e560 con 0x7fa5e015d9a0 2024-03-18T16:27:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.967+0000 7fa5de000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e0154810 0x7fa5e00c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.967+0000 7fa5dea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5e00c49a0 0x7fa5e00c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:00.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.967+0000 7fa5df400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e015d9a0 0x7fa5e01e47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.967+0000 7fa5de000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e0154810 0x7fa5e00c9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41144/0 (socket says 172.21.15.102:41144) 2024-03-18T16:27:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.967+0000 7fa5de000640 1 -- 172.21.15.102:0/1730892905 learned_addr learned my addr 172.21.15.102:0/1730892905 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:00.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.967+0000 7fa5df400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e015d9a0 0x7fa5e01e47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52370/0 (socket says 172.21.15.102:52370) 2024-03-18T16:27:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.971+0000 7fa5df400640 1 -- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e0154810 msgr2=0x7fa5e00c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.971+0000 7fa5df400640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e0154810 0x7fa5e00c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.971+0000 7fa5df400640 1 -- 172.21.15.102:0/1730892905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5e00c49a0 msgr2=0x7fa5e00c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.971+0000 7fa5df400640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5e00c49a0 0x7fa5e00c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:00.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.971+0000 7fa5df400640 1 -- 172.21.15.102:0/1730892905 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e01f4160 con 0x7fa5e015d9a0 2024-03-18T16:27:00.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.973+0000 7fa5df400640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e015d9a0 0x7fa5e01e47b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800caa0 tx=0x7fa5d800cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.978+0000 7fa5d7400640 1 -- 172.21.15.102:0/1730892905 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8017070 con 0x7fa5e015d9a0 2024-03-18T16:27:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.978+0000 7fa5d7400640 1 -- 172.21.15.102:0/1730892905 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d800aae0 con 0x7fa5e015d9a0 2024-03-18T16:27:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.978+0000 7fa5d7400640 1 -- 172.21.15.102:0/1730892905 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8012400 con 0x7fa5e015d9a0 2024-03-18T16:27:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.978+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5e01f0c10 con 0x7fa5e015d9a0 2024-03-18T16:27:00.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.978+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5e01efe50 con 0x7fa5e015d9a0 2024-03-18T16:27:00.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:00.979+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5e0074fd0 con 0x7fa5e015d9a0 2024-03-18T16:27:01.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.243+0000 7fa5d7400640 1 -- 172.21.15.102:0/1730892905 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa5d801a070 con 0x7fa5e015d9a0 2024-03-18T16:27:01.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.245+0000 7fa5d7400640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa5c006bbd0 0x7fa5c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:01.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.246+0000 7fa5dea00640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa5c006bbd0 0x7fa5c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:01.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.246+0000 7fa5d7400640 1 -- 172.21.15.102:0/1730892905 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa5d8057050 con 0x7fa5e015d9a0 2024-03-18T16:27:01.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.247+0000 7fa5dea00640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa5c006bbd0 0x7fa5c006e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0001780 tx=0x7fa5d0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:01.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.256+0000 7fa5d7400640 1 -- 172.21.15.102:0/1730892905 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa5d8058fb0 con 0x7fa5e015d9a0 2024-03-18T16:27:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.635+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5e006c010 con 0x7fa5e015d9a0 2024-03-18T16:27:01.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.650+0000 7fa5d7400640 1 -- 172.21.15.102:0/1730892905 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa5d805d050 con 0x7fa5e015d9a0 2024-03-18T16:27:01.652 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:01.652 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:01.653 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:01.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa5c006bbd0 msgr2=0x7fa5c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa5c006bbd0 0x7fa5c006e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0001780 tx=0x7fa5d0009040 comp rx=0 tx=0).stop 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e015d9a0 msgr2=0x7fa5e01e47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e015d9a0 0x7fa5e01e47b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800caa0 tx=0x7fa5d800cea0 comp rx=0 tx=0).stop 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 shutdown_connections 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa5c006bbd0 0x7fa5c006e090 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa5e015d9a0 0x7fa5e01e47b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5e0154810 0x7fa5e00c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 --2- 172.21.15.102:0/1730892905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5e00c49a0 0x7fa5e00c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.652+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 >> 172.21.15.102:0/1730892905 conn(0x7fa5e00cb890 msgr2=0x7fa5e0153980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.653+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 shutdown_connections 2024-03-18T16:27:01.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:01.653+0000 7fa5dfe00640 1 -- 172.21.15.102:0/1730892905 wait complete. 2024-03-18T16:27:02.275 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:02.275 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:03.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:03.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.448+0000 7fb05e400640 1 Processor -- start 2024-03-18T16:27:03.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.448+0000 7fb05e400640 1 -- start start 2024-03-18T16:27:03.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.449+0000 7fb05e400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 0x7fb058158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:03.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.449+0000 7fb05e400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb05815c3a0 0x7fb0580c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:03.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.449+0000 7fb05e400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb058158ad0 con 0x7fb0580c4fa0 2024-03-18T16:27:03.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.449+0000 7fb05e400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb058159370 con 0x7fb05815c3a0 2024-03-18T16:27:03.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.449+0000 7fb05e400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb058158db0 con 0x7fb058141650 2024-03-18T16:27:03.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.450+0000 7fb05d000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 0x7fb058158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:03.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.450+0000 7fb05da00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb0580c4fa0 0x7fb05806f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:45072/0 (socket says 172.21.15.102:45072) 2024-03-18T16:27:03.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.450+0000 7fb05da00640 1 -- 172.21.15.102:0/1529523734 learned_addr learned my addr 172.21.15.102:0/1529523734 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:03.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.451+0000 7fb057e00640 1 --2- 172.21.15.102:0/1529523734 >> v2:172.21.15.154:3300/0 conn(0x7fb05815c3a0 0x7fb0580c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:03.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.453+0000 7fb05d000640 1 -- 172.21.15.102:0/1529523734 >> v2:172.21.15.154:3300/0 conn(0x7fb05815c3a0 msgr2=0x7fb0580c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:03.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.453+0000 7fb05d000640 1 --2- 172.21.15.102:0/1529523734 >> v2:172.21.15.154:3300/0 conn(0x7fb05815c3a0 0x7fb0580c6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:03.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.453+0000 7fb05d000640 1 -- 172.21.15.102:0/1529523734 >> v1:172.21.15.154:6789/0 conn(0x7fb0580c4fa0 legacy=0x7fb05806f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:03.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.453+0000 7fb05d000640 1 -- 172.21.15.102:0/1529523734 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb058159090 con 0x7fb058141650 2024-03-18T16:27:03.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.455+0000 7fb05d000640 1 --2- 172.21.15.102:0/1529523734 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 0x7fb058158420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0097a0 tx=0x7fb04c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21891f325c9a4026 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:03.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.460+0000 7fb057400640 1 -- 172.21.15.102:0/1529523734 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c035020 con 0x7fb058141650 2024-03-18T16:27:03.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.460+0000 7fb057400640 1 -- 172.21.15.102:0/1529523734 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04c031830 con 0x7fb058141650 2024-03-18T16:27:03.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.460+0000 7fb057400640 1 -- 172.21.15.102:0/1529523734 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c031c00 con 0x7fb058141650 2024-03-18T16:27:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.461+0000 7fb05e400640 1 -- 172.21.15.102:0/1529523734 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 msgr2=0x7fb058158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.461+0000 7fb05e400640 1 --2- 172.21.15.102:0/1529523734 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 0x7fb058158420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0097a0 tx=0x7fb04c02b8d0 comp rx=0 tx=0).stop 2024-03-18T16:27:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.461+0000 7fb05e400640 1 -- 172.21.15.102:0/1529523734 shutdown_connections 2024-03-18T16:27:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.461+0000 7fb05e400640 1 --2- 172.21.15.102:0/1529523734 >> v2:172.21.15.154:3300/0 conn(0x7fb05815c3a0 0x7fb0580c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.461+0000 7fb05e400640 1 --2- 172.21.15.102:0/1529523734 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 0x7fb058158420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.461+0000 7fb05e400640 1 -- 172.21.15.102:0/1529523734 >> 172.21.15.102:0/1529523734 conn(0x7fb0580cb890 msgr2=0x7fb0580cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.461+0000 7fb05e400640 1 -- 172.21.15.102:0/1529523734 shutdown_connections 2024-03-18T16:27:03.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.461+0000 7fb05e400640 1 -- 172.21.15.102:0/1529523734 wait complete. 2024-03-18T16:27:03.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.464+0000 7fb05e400640 1 Processor -- start 2024-03-18T16:27:03.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.464+0000 7fb05e400640 1 -- start start 2024-03-18T16:27:03.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.465+0000 7fb05e400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0580c4fa0 0x7fb0581f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:03.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.465+0000 7fb05e400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 0x7fb0581f0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:03.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.465+0000 7fb05e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05815c3a0 0x7fb0581f0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:03.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.466+0000 7fb05e400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb05815c7c0 con 0x7fb05815c3a0 2024-03-18T16:27:03.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.466+0000 7fb05e400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb0580c66a0 con 0x7fb0580c4fa0 2024-03-18T16:27:03.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.466+0000 7fb05e400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb058141a30 con 0x7fb058141650 2024-03-18T16:27:03.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.466+0000 7fb05d000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0580c4fa0 0x7fb0581f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:03.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.467+0000 7fb05da00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05815c3a0 0x7fb0581f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:03.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.467+0000 7fb057e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 0x7fb0581f0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:03.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.467+0000 7fb05d000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0580c4fa0 0x7fb0581f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41168/0 (socket says 172.21.15.102:41168) 2024-03-18T16:27:03.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.467+0000 7fb05d000640 1 -- 172.21.15.102:0/3221097635 learned_addr learned my addr 172.21.15.102:0/3221097635 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.470+0000 7fb05d000640 1 -- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 msgr2=0x7fb0581f0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.470+0000 7fb05d000640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 0x7fb0581f0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.470+0000 7fb05d000640 1 -- 172.21.15.102:0/3221097635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05815c3a0 msgr2=0x7fb0581f0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.470+0000 7fb05d000640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05815c3a0 0x7fb0581f0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.470+0000 7fb05d000640 1 -- 172.21.15.102:0/3221097635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb058157c20 con 0x7fb0580c4fa0 2024-03-18T16:27:03.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.473+0000 7fb05d000640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0580c4fa0 0x7fb0581f3b30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb04c02bee0 tx=0x7fb04c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:03.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.477+0000 7fb054c00640 1 -- 172.21.15.102:0/3221097635 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c041070 con 0x7fb0580c4fa0 2024-03-18T16:27:03.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.477+0000 7fb054c00640 1 -- 172.21.15.102:0/3221097635 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04c03aba0 con 0x7fb0580c4fa0 2024-03-18T16:27:03.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.477+0000 7fb054c00640 1 -- 172.21.15.102:0/3221097635 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04c035040 con 0x7fb0580c4fa0 2024-03-18T16:27:03.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.477+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0581f44d0 con 0x7fb0580c4fa0 2024-03-18T16:27:03.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.477+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb058157550 con 0x7fb0580c4fa0 2024-03-18T16:27:03.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.479+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb058074fd0 con 0x7fb0580c4fa0 2024-03-18T16:27:03.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.514+0000 7fb054c00640 1 -- 172.21.15.102:0/3221097635 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb04c038020 con 0x7fb0580c4fa0 2024-03-18T16:27:03.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.516+0000 7fb054c00640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb03806bc20 0x7fb03806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:03.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.516+0000 7fb057e00640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb03806bc20 0x7fb03806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:03.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.517+0000 7fb057e00640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb03806bc20 0x7fb03806e0e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb0580cb830 tx=0x7fb044008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:03.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.757+0000 7fb054c00640 1 -- 172.21.15.102:0/3221097635 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb04c0317c0 con 0x7fb0580c4fa0 2024-03-18T16:27:03.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:03.775+0000 7fb054c00640 1 -- 172.21.15.102:0/3221097635 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb04c07b230 con 0x7fb0580c4fa0 2024-03-18T16:27:04.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.022+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0580b7f00 con 0x7fb0580c4fa0 2024-03-18T16:27:04.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.043+0000 7fb054c00640 1 -- 172.21.15.102:0/3221097635 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb04c07f2d0 con 0x7fb0580c4fa0 2024-03-18T16:27:04.045 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:04.045 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:04.045 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb03806bc20 msgr2=0x7fb03806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb03806bc20 0x7fb03806e0e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb0580cb830 tx=0x7fb044008040 comp rx=0 tx=0).stop 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0580c4fa0 msgr2=0x7fb0581f3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0580c4fa0 0x7fb0581f3b30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb04c02bee0 tx=0x7fb04c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 shutdown_connections 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb03806bc20 0x7fb03806e0e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb05815c3a0 0x7fb0581f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb058141650 0x7fb0581f0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 --2- 172.21.15.102:0/3221097635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0580c4fa0 0x7fb0581f3b30 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 >> 172.21.15.102:0/3221097635 conn(0x7fb0580cb890 msgr2=0x7fb05813fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 shutdown_connections 2024-03-18T16:27:04.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:04.046+0000 7fb05e400640 1 -- 172.21.15.102:0/3221097635 wait complete. 2024-03-18T16:27:04.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:04.669 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:05.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:05.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.838+0000 7fe5cbe00640 1 Processor -- start 2024-03-18T16:27:05.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.838+0000 7fe5cbe00640 1 -- start start 2024-03-18T16:27:05.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.839+0000 7fe5cbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc141650 0x7fe5cc158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:05.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.839+0000 7fe5cbe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5cc0c72d0 con 0x7fe5cc0c4fa0 2024-03-18T16:27:05.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.839+0000 7fe5cbe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5cc158ad0 con 0x7fe5cc15c3a0 2024-03-18T16:27:05.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.839+0000 7fe5cbe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe5cc159370 con 0x7fe5cc141650 2024-03-18T16:27:05.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.840+0000 7fe5caa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc141650 0x7fe5cc158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:05.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.840+0000 7fe5ca000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe5cc15c3a0 0x7fe5cc0c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:45088/0 (socket says 172.21.15.102:45088) 2024-03-18T16:27:05.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.840+0000 7fe5ca000640 1 -- 172.21.15.102:0/2447752962 learned_addr learned my addr 172.21.15.102:0/2447752962 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:05.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.843+0000 7fe5caa00640 1 -- 172.21.15.102:0/2447752962 >> v1:172.21.15.154:6789/0 conn(0x7fe5cc15c3a0 legacy=0x7fe5cc0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:05.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.843+0000 7fe5caa00640 1 -- 172.21.15.102:0/2447752962 >> v1:172.21.15.102:6789/0 conn(0x7fe5cc0c4fa0 legacy=0x7fe5cc162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:05.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.843+0000 7fe5caa00640 1 -- 172.21.15.102:0/2447752962 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc158db0 con 0x7fe5cc141650 2024-03-18T16:27:05.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.845+0000 7fe5caa00640 1 --2- 172.21.15.102:0/2447752962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc141650 0x7fe5cc158420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0097d0 tx=0x7fe5bc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a69e049c860c22d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:05.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.849+0000 7fe5c9600640 1 -- 172.21.15.102:0/2447752962 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc035020 con 0x7fe5cc141650 2024-03-18T16:27:05.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.849+0000 7fe5c9600640 1 -- 172.21.15.102:0/2447752962 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc031890 con 0x7fe5cc141650 2024-03-18T16:27:05.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.849+0000 7fe5c9600640 1 -- 172.21.15.102:0/2447752962 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc031c60 con 0x7fe5cc141650 2024-03-18T16:27:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.850+0000 7fe5cbe00640 1 -- 172.21.15.102:0/2447752962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc141650 msgr2=0x7fe5cc158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.850+0000 7fe5cbe00640 1 --2- 172.21.15.102:0/2447752962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc141650 0x7fe5cc158420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0097d0 tx=0x7fe5bc02b930 comp rx=0 tx=0).stop 2024-03-18T16:27:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.850+0000 7fe5cbe00640 1 -- 172.21.15.102:0/2447752962 shutdown_connections 2024-03-18T16:27:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.850+0000 7fe5cbe00640 1 --2- 172.21.15.102:0/2447752962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc141650 0x7fe5cc158420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.850+0000 7fe5cbe00640 1 -- 172.21.15.102:0/2447752962 >> 172.21.15.102:0/2447752962 conn(0x7fe5cc0cb890 msgr2=0x7fe5cc0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.850+0000 7fe5cbe00640 1 -- 172.21.15.102:0/2447752962 shutdown_connections 2024-03-18T16:27:05.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.850+0000 7fe5cbe00640 1 -- 172.21.15.102:0/2447752962 wait complete. 2024-03-18T16:27:05.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.852+0000 7fe5cbe00640 1 Processor -- start 2024-03-18T16:27:05.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.852+0000 7fe5cbe00640 1 -- start start 2024-03-18T16:27:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.853+0000 7fe5cbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc0c4fa0 0x7fe5cc13e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.853+0000 7fe5cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc141650 0x7fe5cc1e5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.853+0000 7fe5cbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5cc15c3a0 0x7fe5cc1e58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.853+0000 7fe5cbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc0c66a0 con 0x7fe5cc141650 2024-03-18T16:27:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.853+0000 7fe5cbe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc0c63c0 con 0x7fe5cc15c3a0 2024-03-18T16:27:05.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.853+0000 7fe5cbe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe5cc15c7c0 con 0x7fe5cc0c4fa0 2024-03-18T16:27:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.853+0000 7fe5caa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc0c4fa0 0x7fe5cc13e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.854+0000 7fe5cb400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5cc15c3a0 0x7fe5cc1e58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.854+0000 7fe5ca000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc141650 0x7fe5cc1e5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.854+0000 7fe5caa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc0c4fa0 0x7fe5cc13e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52412/0 (socket says 172.21.15.102:52412) 2024-03-18T16:27:05.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.854+0000 7fe5caa00640 1 -- 172.21.15.102:0/4235320927 learned_addr learned my addr 172.21.15.102:0/4235320927 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:05.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.859+0000 7fe5caa00640 1 -- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5cc15c3a0 msgr2=0x7fe5cc1e58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:05.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.859+0000 7fe5caa00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5cc15c3a0 0x7fe5cc1e58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:05.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.859+0000 7fe5caa00640 1 -- 172.21.15.102:0/4235320927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc141650 msgr2=0x7fe5cc1e5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:05.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.859+0000 7fe5caa00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc141650 0x7fe5cc1e5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:05.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.859+0000 7fe5caa00640 1 -- 172.21.15.102:0/4235320927 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc1f4570 con 0x7fe5cc0c4fa0 2024-03-18T16:27:05.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.861+0000 7fe5caa00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc0c4fa0 0x7fe5cc13e520 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0098f0 tx=0x7fe5bc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.864+0000 7fe5c2a00640 1 -- 172.21.15.102:0/4235320927 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc035040 con 0x7fe5cc0c4fa0 2024-03-18T16:27:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.864+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5cc157890 con 0x7fe5cc0c4fa0 2024-03-18T16:27:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.864+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5cc1e4e50 con 0x7fe5cc0c4fa0 2024-03-18T16:27:05.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.865+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5cc074fd0 con 0x7fe5cc0c4fa0 2024-03-18T16:27:05.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.866+0000 7fe5c2a00640 1 -- 172.21.15.102:0/4235320927 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc031750 con 0x7fe5cc0c4fa0 2024-03-18T16:27:05.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:05.866+0000 7fe5c2a00640 1 -- 172.21.15.102:0/4235320927 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc046680 con 0x7fe5cc0c4fa0 2024-03-18T16:27:06.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.125+0000 7fe5c2a00640 1 -- 172.21.15.102:0/4235320927 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe5bc038020 con 0x7fe5cc0c4fa0 2024-03-18T16:27:06.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.127+0000 7fe5c2a00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5ac06bbd0 0x7fe5ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:06.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.128+0000 7fe5ca000640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5ac06bbd0 0x7fe5ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.129+0000 7fe5c2a00640 1 -- 172.21.15.102:0/4235320927 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe5bc079e80 con 0x7fe5cc0c4fa0 2024-03-18T16:27:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.129+0000 7fe5ca000640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5ac06bbd0 0x7fe5ac06e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc140c20 tx=0x7fe5b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:06.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.140+0000 7fe5c2a00640 1 -- 172.21.15.102:0/4235320927 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe5bc054020 con 0x7fe5cc0c4fa0 2024-03-18T16:27:06.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.377+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5cc0b7f00 con 0x7fe5cc0c4fa0 2024-03-18T16:27:06.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.390+0000 7fe5c2a00640 1 -- 172.21.15.102:0/4235320927 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe5bc02d040 con 0x7fe5cc0c4fa0 2024-03-18T16:27:06.392 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:06.392 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:06.392 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:06.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5ac06bbd0 msgr2=0x7fe5ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:06.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5ac06bbd0 0x7fe5ac06e090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc140c20 tx=0x7fe5b8009040 comp rx=0 tx=0).stop 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc0c4fa0 msgr2=0x7fe5cc13e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc0c4fa0 0x7fe5cc13e520 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0098f0 tx=0x7fe5bc0046c0 comp rx=0 tx=0).stop 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 shutdown_connections 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe5ac06bbd0 0x7fe5ac06e090 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5cc15c3a0 0x7fe5cc1e58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5cc141650 0x7fe5cc1e5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 --2- 172.21.15.102:0/4235320927 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe5cc0c4fa0 0x7fe5cc13e520 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0098f0 tx=0x7fe5bc0046c0 comp rx=0 tx=0).stop 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.392+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 >> 172.21.15.102:0/4235320927 conn(0x7fe5cc0cb890 msgr2=0x7fe5cc13ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.393+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 shutdown_connections 2024-03-18T16:27:06.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:06.393+0000 7fe5cbe00640 1 -- 172.21.15.102:0/4235320927 wait complete. 2024-03-18T16:27:07.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:07.016 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:08.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:08.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.190+0000 7f97ede00640 1 Processor -- start 2024-03-18T16:27:08.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.190+0000 7f97ede00640 1 -- start start 2024-03-18T16:27:08.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.191+0000 7f97ede00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f97e806b470 0x7f97e815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:08.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.191+0000 7f97ede00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97e8154150 0x7f97e8154580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:08.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.191+0000 7f97ede00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97e806bf60 con 0x7f97e806de50 2024-03-18T16:27:08.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.191+0000 7f97ede00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f97e80ca0a0 con 0x7f97e806b470 2024-03-18T16:27:08.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.191+0000 7f97ede00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f97e80c94b0 con 0x7f97e8154150 2024-03-18T16:27:08.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.192+0000 7f97ed400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97e8154150 0x7f97e8154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:08.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.192+0000 7f97eca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f97e806b470 0x7f97e815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:08.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.192+0000 7f97ed400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97e8154150 0x7f97e8154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41182/0 (socket says 172.21.15.102:41182) 2024-03-18T16:27:08.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.192+0000 7f97ed400640 1 -- 172.21.15.102:0/372963108 learned_addr learned my addr 172.21.15.102:0/372963108 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:08.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.194+0000 7f97ed400640 1 -- 172.21.15.102:0/372963108 >> v2:172.21.15.102:3300/0 conn(0x7f97e806b470 msgr2=0x7f97e815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:08.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.194+0000 7f97ed400640 1 --2- 172.21.15.102:0/372963108 >> v2:172.21.15.102:3300/0 conn(0x7f97e806b470 0x7f97e815ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.194+0000 7f97ed400640 1 -- 172.21.15.102:0/372963108 >> v1:172.21.15.102:6789/0 conn(0x7f97e806de50 legacy=0x7f97e806b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:08.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.194+0000 7f97ed400640 1 -- 172.21.15.102:0/372963108 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e80c9770 con 0x7f97e8154150 2024-03-18T16:27:08.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.197+0000 7f97ed400640 1 --2- 172.21.15.102:0/372963108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8154150 0x7f97e8154580 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f97e806ee50 tx=0x7f97e002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9996d55294c54a6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:08.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.201+0000 7f97e7400640 1 -- 172.21.15.102:0/372963108 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e002cb20 con 0x7f97e8154150 2024-03-18T16:27:08.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.201+0000 7f97e7400640 1 -- 172.21.15.102:0/372963108 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e002ccc0 con 0x7f97e8154150 2024-03-18T16:27:08.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.201+0000 7f97e7400640 1 -- 172.21.15.102:0/372963108 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0034610 con 0x7f97e8154150 2024-03-18T16:27:08.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.202+0000 7f97ede00640 1 -- 172.21.15.102:0/372963108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8154150 msgr2=0x7f97e8154580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:08.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.202+0000 7f97ede00640 1 --2- 172.21.15.102:0/372963108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8154150 0x7f97e8154580 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f97e806ee50 tx=0x7f97e002bf80 comp rx=0 tx=0).stop 2024-03-18T16:27:08.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.202+0000 7f97ede00640 1 -- 172.21.15.102:0/372963108 shutdown_connections 2024-03-18T16:27:08.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.202+0000 7f97ede00640 1 --2- 172.21.15.102:0/372963108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e8154150 0x7f97e8154580 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.202+0000 7f97ede00640 1 --2- 172.21.15.102:0/372963108 >> v2:172.21.15.102:3300/0 conn(0x7f97e806b470 0x7f97e815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.202+0000 7f97ede00640 1 -- 172.21.15.102:0/372963108 >> 172.21.15.102:0/372963108 conn(0x7f97e80727e0 msgr2=0x7f97e8072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:08.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.202+0000 7f97ede00640 1 -- 172.21.15.102:0/372963108 shutdown_connections 2024-03-18T16:27:08.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.202+0000 7f97ede00640 1 -- 172.21.15.102:0/372963108 wait complete. 2024-03-18T16:27:08.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.204+0000 7f97ede00640 1 Processor -- start 2024-03-18T16:27:08.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.205+0000 7f97ede00640 1 -- start start 2024-03-18T16:27:08.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.206+0000 7f97ede00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806b470 0x7f97e80c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:08.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.206+0000 7f97ede00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e806de50 0x7f97e815b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:08.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.206+0000 7f97ede00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e8154150 0x7f97e815bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:08.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.206+0000 7f97ede00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f97e80c5190 con 0x7f97e8154150 2024-03-18T16:27:08.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.206+0000 7f97ede00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97e80c9d30 con 0x7f97e806b470 2024-03-18T16:27:08.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.206+0000 7f97ede00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f97e80c5310 con 0x7f97e806de50 2024-03-18T16:27:08.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.207+0000 7f97e7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e806de50 0x7f97e815b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:08.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.207+0000 7f97eca00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806b470 0x7f97e80c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:08.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.207+0000 7f97ed400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e8154150 0x7f97e815bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:08.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.207+0000 7f97e7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e806de50 0x7f97e815b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52418/0 (socket says 172.21.15.102:52418) 2024-03-18T16:27:08.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.207+0000 7f97e7e00640 1 -- 172.21.15.102:0/3359376932 learned_addr learned my addr 172.21.15.102:0/3359376932 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:08.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.211+0000 7f97e7e00640 1 -- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806b470 msgr2=0x7f97e80c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:08.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.211+0000 7f97e7e00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806b470 0x7f97e80c7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.211+0000 7f97e7e00640 1 -- 172.21.15.102:0/3359376932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e8154150 msgr2=0x7f97e815bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:08.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.211+0000 7f97e7e00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e8154150 0x7f97e815bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.211+0000 7f97e7e00640 1 -- 172.21.15.102:0/3359376932 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e81efc80 con 0x7f97e806de50 2024-03-18T16:27:08.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.213+0000 7f97e7e00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e806de50 0x7f97e815b650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c430 tx=0x7f97d400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:08.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.217+0000 7f97e4c00640 1 -- 172.21.15.102:0/3359376932 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d40052d0 con 0x7f97e806de50 2024-03-18T16:27:08.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.217+0000 7f97e4c00640 1 -- 172.21.15.102:0/3359376932 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d4005490 con 0x7f97e806de50 2024-03-18T16:27:08.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.217+0000 7f97e4c00640 1 -- 172.21.15.102:0/3359376932 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4007260 con 0x7f97e806de50 2024-03-18T16:27:08.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.217+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e81eff00 con 0x7f97e806de50 2024-03-18T16:27:08.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.217+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e81f04c0 con 0x7f97e806de50 2024-03-18T16:27:08.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.218+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97e8074fd0 con 0x7f97e806de50 2024-03-18T16:27:08.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.238+0000 7f97e4c00640 1 -- 172.21.15.102:0/3359376932 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f97d4007400 con 0x7f97e806de50 2024-03-18T16:27:08.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.239+0000 7f97e4c00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 0x7f97c806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:08.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.240+0000 7f97eca00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 0x7f97c806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:08.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.240+0000 7f97e4c00640 1 -- 172.21.15.102:0/3359376932 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f97d4056910 con 0x7f97e806de50 2024-03-18T16:27:08.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.240+0000 7f97eca00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 0x7f97c806e110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009b40 tx=0x7f97dc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:08.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.494+0000 7f97e4c00640 1 -- 172.21.15.102:0/3359376932 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f97e81efc80 con 0x7f97e806de50 2024-03-18T16:27:08.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.725+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97e80735e0 con 0x7f97e806de50 2024-03-18T16:27:08.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.740+0000 7f97e4c00640 1 -- 172.21.15.102:0/3359376932 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f97d4058870 con 0x7f97e806de50 2024-03-18T16:27:08.742 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:08.743 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:08.743 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:08.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.743+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 msgr2=0x7f97c806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:08.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.743+0000 7f97ede00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 0x7f97c806e110 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009b40 tx=0x7f97dc009190 comp rx=0 tx=0).stop 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.743+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e806de50 msgr2=0x7f97e815b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.743+0000 7f97ede00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e806de50 0x7f97e815b650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c430 tx=0x7f97d400c8f0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.743+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 shutdown_connections 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.743+0000 7f97ede00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f97c806bc50 0x7f97c806e110 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.744+0000 7f97ede00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e8154150 0x7f97e815bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.744+0000 7f97ede00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f97e806de50 0x7f97e815b650 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.744+0000 7f97ede00640 1 --2- 172.21.15.102:0/3359376932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e806b470 0x7f97e80c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.744+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 >> 172.21.15.102:0/3359376932 conn(0x7f97e80727e0 msgr2=0x7f97e80c56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.744+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 shutdown_connections 2024-03-18T16:27:08.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:08.744+0000 7f97ede00640 1 -- 172.21.15.102:0/3359376932 wait complete. 2024-03-18T16:27:09.366 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:09.366 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:10.367 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:10.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.547+0000 7fa950a00640 1 Processor -- start 2024-03-18T16:27:10.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.547+0000 7fa950a00640 1 -- start start 2024-03-18T16:27:10.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.549+0000 7fa950a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0ca960 0x7fa94c06c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:10.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.549+0000 7fa950a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa94c0cad40 0x7fa94c15e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:10.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.549+0000 7fa950a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa94c15e670 con 0x7fa94c15a1a0 2024-03-18T16:27:10.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.549+0000 7fa950a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa94c0c8680 con 0x7fa94c0cad40 2024-03-18T16:27:10.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.549+0000 7fa950a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa94c070cb0 con 0x7fa94c0ca960 2024-03-18T16:27:10.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.550+0000 7fa94a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0ca960 0x7fa94c06c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:10.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.550+0000 7fa949e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa94c0cad40 0x7fa94c15e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:10.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.550+0000 7fa94b200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa94c15a1a0 0x7fa94c0705a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45932/0 (socket says 172.21.15.102:45932) 2024-03-18T16:27:10.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.550+0000 7fa94b200640 1 -- 172.21.15.102:0/361094952 learned_addr learned my addr 172.21.15.102:0/361094952 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:10.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.552+0000 7fa949e00640 1 -- 172.21.15.102:0/361094952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0ca960 msgr2=0x7fa94c06c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:10.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.552+0000 7fa949e00640 1 --2- 172.21.15.102:0/361094952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0ca960 0x7fa94c06c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:10.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.552+0000 7fa949e00640 1 -- 172.21.15.102:0/361094952 >> v1:172.21.15.102:6789/0 conn(0x7fa94c15a1a0 legacy=0x7fa94c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:10.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.552+0000 7fa949e00640 1 -- 172.21.15.102:0/361094952 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c06c9c0 con 0x7fa94c0cad40 2024-03-18T16:27:10.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.554+0000 7fa949e00640 1 --2- 172.21.15.102:0/361094952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0cad40 0x7fa94c15e130 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa9380097a0 tx=0x7fa93802b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31316e5303bf12c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:10.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.558+0000 7fa949400640 1 -- 172.21.15.102:0/361094952 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938034070 con 0x7fa94c0cad40 2024-03-18T16:27:10.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.558+0000 7fa949400640 1 -- 172.21.15.102:0/361094952 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa938030800 con 0x7fa94c0cad40 2024-03-18T16:27:10.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.558+0000 7fa949400640 1 -- 172.21.15.102:0/361094952 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa938030bd0 con 0x7fa94c0cad40 2024-03-18T16:27:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.559+0000 7fa950a00640 1 -- 172.21.15.102:0/361094952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0cad40 msgr2=0x7fa94c15e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.559+0000 7fa950a00640 1 --2- 172.21.15.102:0/361094952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0cad40 0x7fa94c15e130 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa9380097a0 tx=0x7fa93802b940 comp rx=0 tx=0).stop 2024-03-18T16:27:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.559+0000 7fa950a00640 1 -- 172.21.15.102:0/361094952 shutdown_connections 2024-03-18T16:27:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.559+0000 7fa950a00640 1 --2- 172.21.15.102:0/361094952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0cad40 0x7fa94c15e130 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.559+0000 7fa950a00640 1 --2- 172.21.15.102:0/361094952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0ca960 0x7fa94c06c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.559+0000 7fa950a00640 1 -- 172.21.15.102:0/361094952 >> 172.21.15.102:0/361094952 conn(0x7fa94c067190 msgr2=0x7fa94c0675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.559+0000 7fa950a00640 1 -- 172.21.15.102:0/361094952 shutdown_connections 2024-03-18T16:27:10.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.559+0000 7fa950a00640 1 -- 172.21.15.102:0/361094952 wait complete. 2024-03-18T16:27:10.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.562+0000 7fa950a00640 1 Processor -- start 2024-03-18T16:27:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.562+0000 7fa950a00640 1 -- start start 2024-03-18T16:27:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.563+0000 7fa950a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0ca960 0x7fa94c1e3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.563+0000 7fa950a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0cad40 0x7fa94c1e4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.563+0000 7fa950a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa94c15a1a0 0x7fa94c13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:10.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.564+0000 7fa950a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c15d930 con 0x7fa94c15a1a0 2024-03-18T16:27:10.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.564+0000 7fa950a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c06cf80 con 0x7fa94c0ca960 2024-03-18T16:27:10.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.564+0000 7fa950a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa94c15dc10 con 0x7fa94c0cad40 2024-03-18T16:27:10.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.564+0000 7fa949e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0cad40 0x7fa94c1e4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:10.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.564+0000 7fa94a800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0ca960 0x7fa94c1e3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.564+0000 7fa94b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa94c15a1a0 0x7fa94c13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.565+0000 7fa949e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0cad40 0x7fa94c1e4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52446/0 (socket says 172.21.15.102:52446) 2024-03-18T16:27:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.565+0000 7fa949e00640 1 -- 172.21.15.102:0/3635362276 learned_addr learned my addr 172.21.15.102:0/3635362276 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:10.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.568+0000 7fa94a800640 1 -- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0cad40 msgr2=0x7fa94c1e4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:10.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.568+0000 7fa94a800640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0cad40 0x7fa94c1e4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:10.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.568+0000 7fa94a800640 1 -- 172.21.15.102:0/3635362276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa94c15a1a0 msgr2=0x7fa94c13f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:10.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.568+0000 7fa94a800640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa94c15a1a0 0x7fa94c13f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:10.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.568+0000 7fa94a800640 1 -- 172.21.15.102:0/3635362276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c13ff20 con 0x7fa94c0ca960 2024-03-18T16:27:10.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.570+0000 7fa94a800640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0ca960 0x7fa94c1e3590 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00e880 tx=0x7fa93c00ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:10.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.575+0000 7fa942a00640 1 -- 172.21.15.102:0/3635362276 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c017070 con 0x7fa94c0ca960 2024-03-18T16:27:10.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.575+0000 7fa942a00640 1 -- 172.21.15.102:0/3635362276 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa93c00c940 con 0x7fa94c0ca960 2024-03-18T16:27:10.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.575+0000 7fa942a00640 1 -- 172.21.15.102:0/3635362276 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c00cd30 con 0x7fa94c0ca960 2024-03-18T16:27:10.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.575+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c141640 con 0x7fa94c0ca960 2024-03-18T16:27:10.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.575+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c141c00 con 0x7fa94c0ca960 2024-03-18T16:27:10.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.576+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa94c074fd0 con 0x7fa94c0ca960 2024-03-18T16:27:10.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.835+0000 7fa942a00640 1 -- 172.21.15.102:0/3635362276 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa93c01a070 con 0x7fa94c0ca960 2024-03-18T16:27:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.837+0000 7fa942a00640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa92c06bbd0 0x7fa92c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.838+0000 7fa949e00640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa92c06bbd0 0x7fa92c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:10.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.838+0000 7fa942a00640 1 -- 172.21.15.102:0/3635362276 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa93c08b4f0 con 0x7fa94c0ca960 2024-03-18T16:27:10.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.839+0000 7fa949e00640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa92c06bbd0 0x7fa92c06e090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa938009560 tx=0x7fa93803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:10.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:10.847+0000 7fa942a00640 1 -- 172.21.15.102:0/3635362276 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa93c059610 con 0x7fa94c0ca960 2024-03-18T16:27:11.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.086+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa94c0c8c50 con 0x7fa94c0ca960 2024-03-18T16:27:11.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.122+0000 7fa942a00640 1 -- 172.21.15.102:0/3635362276 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa93c05d6b0 con 0x7fa94c0ca960 2024-03-18T16:27:11.124 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:11.124 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:11.124 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa92c06bbd0 msgr2=0x7fa92c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa92c06bbd0 0x7fa92c06e090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa938009560 tx=0x7fa93803b040 comp rx=0 tx=0).stop 2024-03-18T16:27:11.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0ca960 msgr2=0x7fa94c1e3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0ca960 0x7fa94c1e3590 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00e880 tx=0x7fa93c00ed40 comp rx=0 tx=0).stop 2024-03-18T16:27:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 shutdown_connections 2024-03-18T16:27:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa92c06bbd0 0x7fa92c06e090 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa94c15a1a0 0x7fa94c13f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa94c0cad40 0x7fa94c1e4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 --2- 172.21.15.102:0/3635362276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa94c0ca960 0x7fa94c1e3590 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.125+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 >> 172.21.15.102:0/3635362276 conn(0x7fa94c067190 msgr2=0x7fa94c071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.126+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 shutdown_connections 2024-03-18T16:27:11.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:11.126+0000 7fa950a00640 1 -- 172.21.15.102:0/3635362276 wait complete. 2024-03-18T16:27:11.749 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:11.749 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:12.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.940+0000 7f3634e00640 1 Processor -- start 2024-03-18T16:27:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.940+0000 7f3634e00640 1 -- start start 2024-03-18T16:27:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.942+0000 7f3634e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630158c00 0x7f363006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.942+0000 7f3634e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3630071700 0x7f3630169110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.942+0000 7f3634e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36300c4190 con 0x7f363006fd80 2024-03-18T16:27:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.942+0000 7f3634e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3630158fe0 con 0x7f3630071700 2024-03-18T16:27:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.942+0000 7f3634e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f363006f3b0 con 0x7f3630158c00 2024-03-18T16:27:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.943+0000 7f362fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3630071700 0x7f3630169110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.943+0000 7f362f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630158c00 0x7f363006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.944+0000 7f362f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630158c00 0x7f363006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58542/0 (socket says 172.21.15.102:58542) 2024-03-18T16:27:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.944+0000 7f362f400640 1 -- 172.21.15.102:0/269816842 learned_addr learned my addr 172.21.15.102:0/269816842 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:12.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.947+0000 7f362f400640 1 -- 172.21.15.102:0/269816842 >> v2:172.21.15.102:3300/0 conn(0x7f3630071700 msgr2=0x7f3630169110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:12.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.947+0000 7f362f400640 1 --2- 172.21.15.102:0/269816842 >> v2:172.21.15.102:3300/0 conn(0x7f3630071700 0x7f3630169110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:12.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.947+0000 7f362f400640 1 -- 172.21.15.102:0/269816842 >> v1:172.21.15.102:6789/0 conn(0x7f363006fd80 legacy=0x7f3630072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:12.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.947+0000 7f362f400640 1 -- 172.21.15.102:0/269816842 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363006f060 con 0x7f3630158c00 2024-03-18T16:27:12.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.950+0000 7f362f400640 1 --2- 172.21.15.102:0/269816842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630158c00 0x7f363006f840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3620009870 tx=0x7f362002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e33bb1c23b3b634c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.957+0000 7f362e000640 1 -- 172.21.15.102:0/269816842 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620034070 con 0x7f3630158c00 2024-03-18T16:27:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.957+0000 7f362e000640 1 -- 172.21.15.102:0/269816842 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3620030840 con 0x7f3630158c00 2024-03-18T16:27:12.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.957+0000 7f362e000640 1 -- 172.21.15.102:0/269816842 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620030c10 con 0x7f3630158c00 2024-03-18T16:27:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.958+0000 7f3634e00640 1 -- 172.21.15.102:0/269816842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630158c00 msgr2=0x7f363006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.958+0000 7f3634e00640 1 --2- 172.21.15.102:0/269816842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630158c00 0x7f363006f840 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3620009870 tx=0x7f362002b980 comp rx=0 tx=0).stop 2024-03-18T16:27:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.958+0000 7f3634e00640 1 -- 172.21.15.102:0/269816842 shutdown_connections 2024-03-18T16:27:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.958+0000 7f3634e00640 1 --2- 172.21.15.102:0/269816842 >> v2:172.21.15.102:3300/0 conn(0x7f3630071700 0x7f3630169110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.958+0000 7f3634e00640 1 --2- 172.21.15.102:0/269816842 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630158c00 0x7f363006f840 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.958+0000 7f3634e00640 1 -- 172.21.15.102:0/269816842 >> 172.21.15.102:0/269816842 conn(0x7f36300cb890 msgr2=0x7f36300cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.958+0000 7f3634e00640 1 -- 172.21.15.102:0/269816842 shutdown_connections 2024-03-18T16:27:12.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.958+0000 7f3634e00640 1 -- 172.21.15.102:0/269816842 wait complete. 2024-03-18T16:27:12.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.960+0000 7f3634e00640 1 Processor -- start 2024-03-18T16:27:12.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.960+0000 7f3634e00640 1 -- start start 2024-03-18T16:27:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.961+0000 7f3634e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363006fd80 0x7f363015c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.961+0000 7f3634e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630071700 0x7f363014c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.961+0000 7f3634e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3630158c00 0x7f36301f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.961+0000 7f3634e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3630072270 con 0x7f363006fd80 2024-03-18T16:27:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.961+0000 7f3634e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3630071f90 con 0x7f3630158c00 2024-03-18T16:27:12.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.961+0000 7f3634e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3630072550 con 0x7f3630071700 2024-03-18T16:27:12.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.962+0000 7f362fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3630158c00 0x7f36301f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:12.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.962+0000 7f362f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363006fd80 0x7f363015c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:12.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.962+0000 7f362ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630071700 0x7f363014c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:12.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.962+0000 7f362fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3630158c00 0x7f36301f47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35346/0 (socket says 172.21.15.102:35346) 2024-03-18T16:27:12.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.962+0000 7f362fe00640 1 -- 172.21.15.102:0/2694194038 learned_addr learned my addr 172.21.15.102:0/2694194038 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:12.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.966+0000 7f362fe00640 1 -- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630071700 msgr2=0x7f363014c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:12.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.966+0000 7f362fe00640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630071700 0x7f363014c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:12.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.966+0000 7f362fe00640 1 -- 172.21.15.102:0/2694194038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363006fd80 msgr2=0x7f363015c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:12.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.966+0000 7f362fe00640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363006fd80 0x7f363015c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:12.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.966+0000 7f362fe00640 1 -- 172.21.15.102:0/2694194038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36300cac20 con 0x7f3630158c00 2024-03-18T16:27:12.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.968+0000 7f362fe00640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3630158c00 0x7f36301f47a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3628009dd0 tx=0x7f362800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:12.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.972+0000 7f3627400640 1 -- 172.21.15.102:0/2694194038 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628018020 con 0x7f3630158c00 2024-03-18T16:27:12.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.972+0000 7f3627400640 1 -- 172.21.15.102:0/2694194038 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3628014480 con 0x7f3630158c00 2024-03-18T16:27:12.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.972+0000 7f3627400640 1 -- 172.21.15.102:0/2694194038 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628014870 con 0x7f3630158c00 2024-03-18T16:27:12.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.972+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36300caec0 con 0x7f3630158c00 2024-03-18T16:27:12.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.972+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36300cb480 con 0x7f3630158c00 2024-03-18T16:27:12.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.974+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3630074fd0 con 0x7f3630158c00 2024-03-18T16:27:12.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.993+0000 7f3627400640 1 -- 172.21.15.102:0/2694194038 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3628005d10 con 0x7f3630158c00 2024-03-18T16:27:12.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.994+0000 7f3627400640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f361006bc20 0x7f361006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:12.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.995+0000 7f362f400640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f361006bc20 0x7f361006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:12.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.995+0000 7f362f400640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f361006bc20 0x7f361006e0e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3620036000 tx=0x7f3620008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:12.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:12.996+0000 7f3627400640 1 -- 172.21.15.102:0/2694194038 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3628058540 con 0x7f3630158c00 2024-03-18T16:27:13.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.249+0000 7f3627400640 1 -- 172.21.15.102:0/2694194038 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3628022070 con 0x7f3630158c00 2024-03-18T16:27:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.496+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3630073010 con 0x7f3630158c00 2024-03-18T16:27:13.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.553+0000 7f3627400640 1 -- 172.21.15.102:0/2694194038 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3628010040 con 0x7f3630158c00 2024-03-18T16:27:13.556 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:13.556 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:13.556 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f361006bc20 msgr2=0x7f361006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f361006bc20 0x7f361006e0e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3620036000 tx=0x7f3620008030 comp rx=0 tx=0).stop 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3630158c00 msgr2=0x7f36301f47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3630158c00 0x7f36301f47a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3628009dd0 tx=0x7f362800c490 comp rx=0 tx=0).stop 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 shutdown_connections 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f361006bc20 0x7f361006e0e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3630158c00 0x7f36301f47a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3630071700 0x7f363014c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 --2- 172.21.15.102:0/2694194038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363006fd80 0x7f363015c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:13.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 >> 172.21.15.102:0/2694194038 conn(0x7f36300cb890 msgr2=0x7f36300c2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:13.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 shutdown_connections 2024-03-18T16:27:13.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:13.557+0000 7f3634e00640 1 -- 172.21.15.102:0/2694194038 wait complete. 2024-03-18T16:27:14.179 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:14.179 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:15.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:15.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.363+0000 7f5d83200640 1 Processor -- start 2024-03-18T16:27:15.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.364+0000 7f5d83200640 1 -- start start 2024-03-18T16:27:15.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.365+0000 7f5d83200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:15.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.365+0000 7f5d83200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5d8415c3a0 0x7f5d840c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:15.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.365+0000 7f5d83200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d84158ad0 con 0x7f5d840c4fa0 2024-03-18T16:27:15.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.365+0000 7f5d83200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5d84159370 con 0x7f5d8415c3a0 2024-03-18T16:27:15.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.365+0000 7f5d83200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5d84158db0 con 0x7f5d84141650 2024-03-18T16:27:15.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.366+0000 7f5d81e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:15.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.366+0000 7f5d81e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58568/0 (socket says 172.21.15.102:58568) 2024-03-18T16:27:15.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.366+0000 7f5d81e00640 1 -- 172.21.15.102:0/1183441239 learned_addr learned my addr 172.21.15.102:0/1183441239 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:15.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.366+0000 7f5d81400640 1 --2- 172.21.15.102:0/1183441239 >> v2:172.21.15.102:3300/0 conn(0x7f5d8415c3a0 0x7f5d840c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.369+0000 7f5d81e00640 1 -- 172.21.15.102:0/1183441239 >> v2:172.21.15.102:3300/0 conn(0x7f5d8415c3a0 msgr2=0x7f5d840c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.369+0000 7f5d81e00640 1 --2- 172.21.15.102:0/1183441239 >> v2:172.21.15.102:3300/0 conn(0x7f5d8415c3a0 0x7f5d840c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.369+0000 7f5d81e00640 1 -- 172.21.15.102:0/1183441239 >> v1:172.21.15.102:6789/0 conn(0x7f5d840c4fa0 legacy=0x7f5d8406f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:15.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.369+0000 7f5d81e00640 1 -- 172.21.15.102:0/1183441239 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d84159090 con 0x7f5d84141650 2024-03-18T16:27:15.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.372+0000 7f5d81e00640 1 --2- 172.21.15.102:0/1183441239 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009870 tx=0x7f5d7402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e97dfee977e0eda3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:15.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.374+0000 7f5d80a00640 1 -- 172.21.15.102:0/1183441239 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74034070 con 0x7f5d84141650 2024-03-18T16:27:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.376+0000 7f5d80a00640 1 -- 172.21.15.102:0/1183441239 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d74030840 con 0x7f5d84141650 2024-03-18T16:27:15.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.376+0000 7f5d80a00640 1 -- 172.21.15.102:0/1183441239 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74030c10 con 0x7f5d84141650 2024-03-18T16:27:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.377+0000 7f5d83200640 1 -- 172.21.15.102:0/1183441239 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 msgr2=0x7f5d84158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.377+0000 7f5d83200640 1 --2- 172.21.15.102:0/1183441239 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009870 tx=0x7f5d7402b980 comp rx=0 tx=0).stop 2024-03-18T16:27:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.377+0000 7f5d83200640 1 -- 172.21.15.102:0/1183441239 shutdown_connections 2024-03-18T16:27:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.377+0000 7f5d83200640 1 --2- 172.21.15.102:0/1183441239 >> v2:172.21.15.102:3300/0 conn(0x7f5d8415c3a0 0x7f5d840c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.377+0000 7f5d83200640 1 --2- 172.21.15.102:0/1183441239 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.377+0000 7f5d83200640 1 -- 172.21.15.102:0/1183441239 >> 172.21.15.102:0/1183441239 conn(0x7f5d840cb890 msgr2=0x7f5d840cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.377+0000 7f5d83200640 1 -- 172.21.15.102:0/1183441239 shutdown_connections 2024-03-18T16:27:15.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.377+0000 7f5d83200640 1 -- 172.21.15.102:0/1183441239 wait complete. 2024-03-18T16:27:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.380+0000 7f5d83200640 1 Processor -- start 2024-03-18T16:27:15.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.380+0000 7f5d83200640 1 -- start start 2024-03-18T16:27:15.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.382+0000 7f5d83200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d840c4fa0 0x7f5d840c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.382+0000 7f5d83200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.382+0000 7f5d83200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8415c3a0 0x7f5d840c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.382+0000 7f5d83200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8415c7c0 con 0x7f5d840c4fa0 2024-03-18T16:27:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.382+0000 7f5d83200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d840c66a0 con 0x7f5d8415c3a0 2024-03-18T16:27:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.382+0000 7f5d83200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5d84141a30 con 0x7f5d84141650 2024-03-18T16:27:15.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.382+0000 7f5d81400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.383+0000 7f5d81400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58582/0 (socket says 172.21.15.102:58582) 2024-03-18T16:27:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.383+0000 7f5d82800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8415c3a0 0x7f5d840c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.383+0000 7f5d81400640 1 -- 172.21.15.102:0/1733151857 learned_addr learned my addr 172.21.15.102:0/1733151857 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:15.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.383+0000 7f5d81e00640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d840c4fa0 0x7f5d840c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.386+0000 7f5d81400640 1 -- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8415c3a0 msgr2=0x7f5d840c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.386+0000 7f5d81400640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8415c3a0 0x7f5d840c2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.386+0000 7f5d81400640 1 -- 172.21.15.102:0/1733151857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d840c4fa0 msgr2=0x7f5d840c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.386+0000 7f5d81400640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d840c4fa0 0x7f5d840c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:15.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.386+0000 7f5d81400640 1 -- 172.21.15.102:0/1733151857 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d84158660 con 0x7f5d84141650 2024-03-18T16:27:15.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.388+0000 7f5d81400640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158120 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000e870 tx=0x7f5d7000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:15.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.392+0000 7f5d7a000640 1 -- 172.21.15.102:0/1733151857 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70017070 con 0x7f5d84141650 2024-03-18T16:27:15.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.392+0000 7f5d7a000640 1 -- 172.21.15.102:0/1733151857 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d7000c940 con 0x7f5d84141650 2024-03-18T16:27:15.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.392+0000 7f5d7a000640 1 -- 172.21.15.102:0/1733151857 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7000cd30 con 0x7f5d84141650 2024-03-18T16:27:15.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.393+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d84156630 con 0x7f5d84141650 2024-03-18T16:27:15.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.393+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d84156e70 con 0x7f5d84141650 2024-03-18T16:27:15.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.394+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d84074fd0 con 0x7f5d84141650 2024-03-18T16:27:15.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.414+0000 7f5d7a000640 1 -- 172.21.15.102:0/1733151857 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5d7001a070 con 0x7f5d84141650 2024-03-18T16:27:15.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.415+0000 7f5d7a000640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5d6406bc20 0x7f5d6406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:15.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.415+0000 7f5d81e00640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5d6406bc20 0x7f5d6406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:15.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.416+0000 7f5d81e00640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5d6406bc20 0x7f5d6406e0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5d74036000 tx=0x7f5d74008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:15.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.418+0000 7f5d7a000640 1 -- 172.21.15.102:0/1733151857 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5d700575f0 con 0x7f5d84141650 2024-03-18T16:27:15.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.668+0000 7f5d7a000640 1 -- 172.21.15.102:0/1733151857 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5d84156630 con 0x7f5d84141650 2024-03-18T16:27:15.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:15.989+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d84140520 con 0x7f5d84141650 2024-03-18T16:27:16.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.002+0000 7f5d7a000640 1 -- 172.21.15.102:0/1733151857 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f5d70059550 con 0x7f5d84141650 2024-03-18T16:27:16.004 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:16.004 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:16.004 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:16.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5d6406bc20 msgr2=0x7f5d6406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:16.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5d6406bc20 0x7f5d6406e0e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5d74036000 tx=0x7f5d74008030 comp rx=0 tx=0).stop 2024-03-18T16:27:16.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 msgr2=0x7f5d84158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:16.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158120 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000e870 tx=0x7f5d7000ed30 comp rx=0 tx=0).stop 2024-03-18T16:27:16.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 shutdown_connections 2024-03-18T16:27:16.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5d6406bc20 0x7f5d6406e0e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d8415c3a0 0x7f5d840c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5d84141650 0x7f5d84158120 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 --2- 172.21.15.102:0/1733151857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5d840c4fa0 0x7f5d840c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 >> 172.21.15.102:0/1733151857 conn(0x7f5d840cb890 msgr2=0x7f5d840c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 shutdown_connections 2024-03-18T16:27:16.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:16.004+0000 7f5d83200640 1 -- 172.21.15.102:0/1733151857 wait complete. 2024-03-18T16:27:16.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:16.627 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:17.628 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:17.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.804+0000 7fa39b200640 1 Processor -- start 2024-03-18T16:27:17.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.804+0000 7fa39b200640 1 -- start start 2024-03-18T16:27:17.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.805+0000 7fa39b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c141650 0x7fa39c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:17.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.805+0000 7fa39b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa39c0c72d0 con 0x7fa39c15c3a0 2024-03-18T16:27:17.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.805+0000 7fa39b200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa39c158ad0 con 0x7fa39c0c4fa0 2024-03-18T16:27:17.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.805+0000 7fa39b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa39c159370 con 0x7fa39c141650 2024-03-18T16:27:17.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.806+0000 7fa399400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa39c15c3a0 0x7fa39c0c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32986/0 (socket says 172.21.15.102:32986) 2024-03-18T16:27:17.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.806+0000 7fa399400640 1 -- 172.21.15.102:0/477458690 learned_addr learned my addr 172.21.15.102:0/477458690 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:17.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.806+0000 7fa399e00640 1 --2- 172.21.15.102:0/477458690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c141650 0x7fa39c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:17.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.809+0000 7fa399e00640 1 -- 172.21.15.102:0/477458690 >> v1:172.21.15.154:6789/0 conn(0x7fa39c0c4fa0 legacy=0x7fa39c162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:17.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.809+0000 7fa399e00640 1 -- 172.21.15.102:0/477458690 >> v1:172.21.15.102:6789/0 conn(0x7fa39c15c3a0 legacy=0x7fa39c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:17.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.809+0000 7fa399e00640 1 -- 172.21.15.102:0/477458690 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa39c158db0 con 0x7fa39c141650 2024-03-18T16:27:17.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.811+0000 7fa399e00640 1 --2- 172.21.15.102:0/477458690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c141650 0x7fa39c158420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0097d0 tx=0x7fa38c02b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc4d13bfdfa03bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:17.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.816+0000 7fa398a00640 1 -- 172.21.15.102:0/477458690 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c034070 con 0x7fa39c141650 2024-03-18T16:27:17.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.816+0000 7fa398a00640 1 -- 172.21.15.102:0/477458690 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa38c030800 con 0x7fa39c141650 2024-03-18T16:27:17.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.816+0000 7fa398a00640 1 -- 172.21.15.102:0/477458690 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c030bd0 con 0x7fa39c141650 2024-03-18T16:27:17.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.817+0000 7fa39b200640 1 -- 172.21.15.102:0/477458690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c141650 msgr2=0x7fa39c158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:17.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.817+0000 7fa39b200640 1 --2- 172.21.15.102:0/477458690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c141650 0x7fa39c158420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0097d0 tx=0x7fa38c02b940 comp rx=0 tx=0).stop 2024-03-18T16:27:17.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.817+0000 7fa39b200640 1 -- 172.21.15.102:0/477458690 shutdown_connections 2024-03-18T16:27:17.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.817+0000 7fa39b200640 1 --2- 172.21.15.102:0/477458690 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c141650 0x7fa39c158420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:17.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.817+0000 7fa39b200640 1 -- 172.21.15.102:0/477458690 >> 172.21.15.102:0/477458690 conn(0x7fa39c0cb890 msgr2=0x7fa39c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:17.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.817+0000 7fa39b200640 1 -- 172.21.15.102:0/477458690 shutdown_connections 2024-03-18T16:27:17.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.817+0000 7fa39b200640 1 -- 172.21.15.102:0/477458690 wait complete. 2024-03-18T16:27:17.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.819+0000 7fa39b200640 1 Processor -- start 2024-03-18T16:27:17.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.819+0000 7fa39b200640 1 -- start start 2024-03-18T16:27:17.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.820+0000 7fa39b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c0c4fa0 0x7fa39c15aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:17.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.820+0000 7fa39b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa39c141650 0x7fa39c157740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:17.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.820+0000 7fa39b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa39c15c3a0 0x7fa39c154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:17.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.820+0000 7fa39b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa39c0c66a0 con 0x7fa39c15c3a0 2024-03-18T16:27:17.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.820+0000 7fa39b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa39c0c63c0 con 0x7fa39c141650 2024-03-18T16:27:17.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.820+0000 7fa39b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa39c15c7c0 con 0x7fa39c0c4fa0 2024-03-18T16:27:17.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.820+0000 7fa399e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c0c4fa0 0x7fa39c15aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.820+0000 7fa399400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa39c141650 0x7fa39c157740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.821+0000 7fa39a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa39c15c3a0 0x7fa39c154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.821+0000 7fa399400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa39c141650 0x7fa39c157740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35364/0 (socket says 172.21.15.102:35364) 2024-03-18T16:27:17.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.821+0000 7fa399400640 1 -- 172.21.15.102:0/4127594427 learned_addr learned my addr 172.21.15.102:0/4127594427 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:17.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.824+0000 7fa399400640 1 -- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c0c4fa0 msgr2=0x7fa39c15aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:17.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.824+0000 7fa399400640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c0c4fa0 0x7fa39c15aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:17.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.824+0000 7fa399400640 1 -- 172.21.15.102:0/4127594427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa39c15c3a0 msgr2=0x7fa39c154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:17.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.824+0000 7fa399400640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa39c15c3a0 0x7fa39c154dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:17.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.824+0000 7fa399400640 1 -- 172.21.15.102:0/4127594427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa39c154310 con 0x7fa39c141650 2024-03-18T16:27:17.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.827+0000 7fa399400640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa39c141650 0x7fa39c157740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c480 tx=0x7fa38800c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:17.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.831+0000 7fa392000640 1 -- 172.21.15.102:0/4127594427 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa388017070 con 0x7fa39c141650 2024-03-18T16:27:17.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.831+0000 7fa392000640 1 -- 172.21.15.102:0/4127594427 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa388005ce0 con 0x7fa39c141650 2024-03-18T16:27:17.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.831+0000 7fa392000640 1 -- 172.21.15.102:0/4127594427 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3880072f0 con 0x7fa39c141650 2024-03-18T16:27:17.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.831+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa39c0cb0f0 con 0x7fa39c141650 2024-03-18T16:27:17.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.831+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa39c158340 con 0x7fa39c141650 2024-03-18T16:27:17.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.833+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa39c0c74a0 con 0x7fa39c141650 2024-03-18T16:27:17.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.851+0000 7fa392000640 1 -- 172.21.15.102:0/4127594427 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa38801a070 con 0x7fa39c141650 2024-03-18T16:27:17.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.853+0000 7fa392000640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa37c06bc20 0x7fa37c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:17.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.853+0000 7fa399e00640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa37c06bc20 0x7fa37c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:17.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:17.854+0000 7fa399e00640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa37c06bc20 0x7fa37c06e0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa38c036000 tx=0x7fa38c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:18.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.103+0000 7fa392000640 1 -- 172.21.15.102:0/4127594427 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa388056760 con 0x7fa39c141650 2024-03-18T16:27:18.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.121+0000 7fa392000640 1 -- 172.21.15.102:0/4127594427 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa39c0c74a0 con 0x7fa39c141650 2024-03-18T16:27:18.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.407+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa39c074fd0 con 0x7fa39c141650 2024-03-18T16:27:18.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.427+0000 7fa392000640 1 -- 172.21.15.102:0/4127594427 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa3880586c0 con 0x7fa39c141650 2024-03-18T16:27:18.429 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:18.429 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:18.429 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:18.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa37c06bc20 msgr2=0x7fa37c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:18.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa37c06bc20 0x7fa37c06e0e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa38c036000 tx=0x7fa38c03b040 comp rx=0 tx=0).stop 2024-03-18T16:27:18.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa39c141650 msgr2=0x7fa39c157740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa39c141650 0x7fa39c157740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c480 tx=0x7fa38800c940 comp rx=0 tx=0).stop 2024-03-18T16:27:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 shutdown_connections 2024-03-18T16:27:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa37c06bc20 0x7fa37c06e0e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa39c15c3a0 0x7fa39c154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa39c141650 0x7fa39c157740 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 --2- 172.21.15.102:0/4127594427 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa39c0c4fa0 0x7fa39c15aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.429+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 >> 172.21.15.102:0/4127594427 conn(0x7fa39c0cb890 msgr2=0x7fa39c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.430+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 shutdown_connections 2024-03-18T16:27:18.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:18.431+0000 7fa39b200640 1 -- 172.21.15.102:0/4127594427 wait complete. 2024-03-18T16:27:19.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:19.053 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:20.054 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.228+0000 7f3bce200640 1 Processor -- start 2024-03-18T16:27:20.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.228+0000 7f3bce200640 1 -- start start 2024-03-18T16:27:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.229+0000 7f3bce200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806b470 0x7f3bc815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.229+0000 7f3bce200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3bc806de50 0x7f3bc806b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.229+0000 7f3bce200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc80c94b0 con 0x7f3bc806bd90 2024-03-18T16:27:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.229+0000 7f3bce200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc80ca0a0 con 0x7f3bc806de50 2024-03-18T16:27:20.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.229+0000 7f3bce200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc80c9770 con 0x7f3bc806b470 2024-03-18T16:27:20.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.230+0000 7f3bcce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806b470 0x7f3bc815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:20.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.230+0000 7f3bc7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3bc806de50 0x7f3bc806b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:20.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.230+0000 7f3bcd800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3bc806bd90 0x7f3bc8154150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:47650/0 (socket says 172.21.15.102:47650) 2024-03-18T16:27:20.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.230+0000 7f3bcd800640 1 -- 172.21.15.102:0/4262000685 learned_addr learned my addr 172.21.15.102:0/4262000685 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:20.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.233+0000 7f3bcce00640 1 -- 172.21.15.102:0/4262000685 >> v2:172.21.15.102:3300/0 conn(0x7f3bc806de50 msgr2=0x7f3bc806b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:20.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.233+0000 7f3bcce00640 1 --2- 172.21.15.102:0/4262000685 >> v2:172.21.15.102:3300/0 conn(0x7f3bc806de50 0x7f3bc806b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.233+0000 7f3bcce00640 1 -- 172.21.15.102:0/4262000685 >> v1:172.21.15.154:6789/0 conn(0x7f3bc806bd90 legacy=0x7f3bc8154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:20.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.233+0000 7f3bcce00640 1 -- 172.21.15.102:0/4262000685 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc80c9a50 con 0x7f3bc806b470 2024-03-18T16:27:20.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.235+0000 7f3bcce00640 1 --2- 172.21.15.102:0/4262000685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806b470 0x7f3bc815ad60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0097a0 tx=0x7f3bbc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=526be3e308ce63c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:20.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.237+0000 7f3bc7400640 1 -- 172.21.15.102:0/4262000685 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc035020 con 0x7f3bc806b470 2024-03-18T16:27:20.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.239+0000 7f3bc7400640 1 -- 172.21.15.102:0/4262000685 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bbc031810 con 0x7f3bc806b470 2024-03-18T16:27:20.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.239+0000 7f3bc7400640 1 -- 172.21.15.102:0/4262000685 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc031be0 con 0x7f3bc806b470 2024-03-18T16:27:20.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.239+0000 7f3bce200640 1 -- 172.21.15.102:0/4262000685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806b470 msgr2=0x7f3bc815ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:20.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.239+0000 7f3bce200640 1 --2- 172.21.15.102:0/4262000685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806b470 0x7f3bc815ad60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0097a0 tx=0x7f3bbc02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.240+0000 7f3bce200640 1 -- 172.21.15.102:0/4262000685 shutdown_connections 2024-03-18T16:27:20.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.240+0000 7f3bce200640 1 --2- 172.21.15.102:0/4262000685 >> v2:172.21.15.102:3300/0 conn(0x7f3bc806de50 0x7f3bc806b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.240+0000 7f3bce200640 1 --2- 172.21.15.102:0/4262000685 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806b470 0x7f3bc815ad60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.240+0000 7f3bce200640 1 -- 172.21.15.102:0/4262000685 >> 172.21.15.102:0/4262000685 conn(0x7f3bc80727e0 msgr2=0x7f3bc8072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:20.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.240+0000 7f3bce200640 1 -- 172.21.15.102:0/4262000685 shutdown_connections 2024-03-18T16:27:20.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.240+0000 7f3bce200640 1 -- 172.21.15.102:0/4262000685 wait complete. 2024-03-18T16:27:20.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.242+0000 7f3bce200640 1 Processor -- start 2024-03-18T16:27:20.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.243+0000 7f3bce200640 1 -- start start 2024-03-18T16:27:20.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.244+0000 7f3bce200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc806b470 0x7f3bc8155770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:20.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.244+0000 7f3bce200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806bd90 0x7f3bc815e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:20.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.244+0000 7f3bce200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc806de50 0x7f3bc81ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:20.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.244+0000 7f3bce200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc81549e0 con 0x7f3bc806de50 2024-03-18T16:27:20.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.244+0000 7f3bce200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc8154860 con 0x7f3bc806b470 2024-03-18T16:27:20.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.244+0000 7f3bce200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc80c5190 con 0x7f3bc806bd90 2024-03-18T16:27:20.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.245+0000 7f3bcce00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc806b470 0x7f3bc8155770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:20.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.245+0000 7f3bc7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806bd90 0x7f3bc815e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:20.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.245+0000 7f3bcd800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc806de50 0x7f3bc81ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:20.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.245+0000 7f3bcce00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc806b470 0x7f3bc8155770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35372/0 (socket says 172.21.15.102:35372) 2024-03-18T16:27:20.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.245+0000 7f3bcce00640 1 -- 172.21.15.102:0/1064378931 learned_addr learned my addr 172.21.15.102:0/1064378931 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:20.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.249+0000 7f3bcce00640 1 -- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806bd90 msgr2=0x7f3bc815e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:20.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.249+0000 7f3bcce00640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806bd90 0x7f3bc815e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.249+0000 7f3bcce00640 1 -- 172.21.15.102:0/1064378931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc806de50 msgr2=0x7f3bc81ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:20.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.249+0000 7f3bcce00640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc806de50 0x7f3bc81ef880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.249+0000 7f3bcce00640 1 -- 172.21.15.102:0/1064378931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc8155cb0 con 0x7f3bc806b470 2024-03-18T16:27:20.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.251+0000 7f3bcce00640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc806b470 0x7f3bc8155770 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0098c0 tx=0x7f3bbc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:20.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.256+0000 7f3bc4c00640 1 -- 172.21.15.102:0/1064378931 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc041070 con 0x7f3bc806b470 2024-03-18T16:27:20.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.256+0000 7f3bc4c00640 1 -- 172.21.15.102:0/1064378931 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bbc03ad00 con 0x7f3bc806b470 2024-03-18T16:27:20.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.256+0000 7f3bc4c00640 1 -- 172.21.15.102:0/1064378931 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc035040 con 0x7f3bc806b470 2024-03-18T16:27:20.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.256+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc81f0740 con 0x7f3bc806b470 2024-03-18T16:27:20.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.256+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc81f0460 con 0x7f3bc806b470 2024-03-18T16:27:20.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.257+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc8074fd0 con 0x7f3bc806b470 2024-03-18T16:27:20.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.524+0000 7f3bc4c00640 1 -- 172.21.15.102:0/1064378931 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3bbc04c060 con 0x7f3bc806b470 2024-03-18T16:27:20.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.526+0000 7f3bc4c00640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3ba806bc20 0x7f3ba806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:20.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.527+0000 7f3bc7e00640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3ba806bc20 0x7f3ba806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:20.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.528+0000 7f3bc7e00640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3ba806bc20 0x7f3ba806e0e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3bc81584d0 tx=0x7f3bb4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:20.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.529+0000 7f3bc4c00640 1 -- 172.21.15.102:0/1064378931 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3bbc078ea0 con 0x7f3bc806b470 2024-03-18T16:27:20.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.548+0000 7f3bc4c00640 1 -- 172.21.15.102:0/1064378931 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3bbc07ae00 con 0x7f3bc806b470 2024-03-18T16:27:20.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.788+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bc815ad70 con 0x7f3bc806b470 2024-03-18T16:27:20.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.801+0000 7f3bc4c00640 1 -- 172.21.15.102:0/1064378931 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3bbc07eea0 con 0x7f3bc806b470 2024-03-18T16:27:20.803 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:20.804 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:20.804 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.805+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3ba806bc20 msgr2=0x7f3ba806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.805+0000 7f3bce200640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3ba806bc20 0x7f3ba806e0e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3bc81584d0 tx=0x7f3bb4009190 comp rx=0 tx=0).stop 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.805+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc806b470 msgr2=0x7f3bc8155770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.805+0000 7f3bce200640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc806b470 0x7f3bc8155770 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc0098c0 tx=0x7f3bbc0046c0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.805+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 shutdown_connections 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.806+0000 7f3bce200640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3ba806bc20 0x7f3ba806e0e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.806+0000 7f3bce200640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc806de50 0x7f3bc81ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.806+0000 7f3bce200640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bc806bd90 0x7f3bc815e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.806+0000 7f3bce200640 1 --2- 172.21.15.102:0/1064378931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bc806b470 0x7f3bc8155770 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.806+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 >> 172.21.15.102:0/1064378931 conn(0x7f3bc80727e0 msgr2=0x7f3bc806e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.806+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 shutdown_connections 2024-03-18T16:27:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:20.806+0000 7f3bce200640 1 -- 172.21.15.102:0/1064378931 wait complete. 2024-03-18T16:27:21.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:21.429 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:21.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:27:21.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:27:22.430 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:22.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.621+0000 7fab01000640 1 Processor -- start 2024-03-18T16:27:22.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.621+0000 7fab01000640 1 -- start start 2024-03-18T16:27:22.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.622+0000 7fab01000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc141650 0x7faafc158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:22.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.622+0000 7fab01000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faafc0c4fa0 0x7faafc0c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:22.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.622+0000 7fab01000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faafc0c72d0 con 0x7faafc15c3a0 2024-03-18T16:27:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.622+0000 7fab01000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7faafc159370 con 0x7faafc0c4fa0 2024-03-18T16:27:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.622+0000 7fab01000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faafc158ad0 con 0x7faafc141650 2024-03-18T16:27:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.623+0000 7faafb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc141650 0x7faafc158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.623+0000 7faafbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faafc0c4fa0 0x7faafc0c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.624+0000 7faafb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc141650 0x7faafc158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50464/0 (socket says 172.21.15.102:50464) 2024-03-18T16:27:22.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.624+0000 7faafb400640 1 -- 172.21.15.102:0/2068104475 learned_addr learned my addr 172.21.15.102:0/2068104475 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.626+0000 7faafb400640 1 -- 172.21.15.102:0/2068104475 >> v2:172.21.15.102:3300/0 conn(0x7faafc0c4fa0 msgr2=0x7faafc0c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.626+0000 7faafb400640 1 --2- 172.21.15.102:0/2068104475 >> v2:172.21.15.102:3300/0 conn(0x7faafc0c4fa0 0x7faafc0c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.626+0000 7faafb400640 1 -- 172.21.15.102:0/2068104475 >> v1:172.21.15.102:6789/0 conn(0x7faafc15c3a0 legacy=0x7faafc0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:22.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.626+0000 7faafb400640 1 -- 172.21.15.102:0/2068104475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faafc158db0 con 0x7faafc141650 2024-03-18T16:27:22.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.628+0000 7faafb400640 1 --2- 172.21.15.102:0/2068104475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc141650 0x7faafc158420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faaec009870 tx=0x7faaec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92d08bfe27edd38f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:22.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.633+0000 7faafa000640 1 -- 172.21.15.102:0/2068104475 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec034070 con 0x7faafc141650 2024-03-18T16:27:22.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.633+0000 7faafa000640 1 -- 172.21.15.102:0/2068104475 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaec030840 con 0x7faafc141650 2024-03-18T16:27:22.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.633+0000 7faafa000640 1 -- 172.21.15.102:0/2068104475 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec030c10 con 0x7faafc141650 2024-03-18T16:27:22.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.634+0000 7fab01000640 1 -- 172.21.15.102:0/2068104475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc141650 msgr2=0x7faafc158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:22.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.634+0000 7fab01000640 1 --2- 172.21.15.102:0/2068104475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc141650 0x7faafc158420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7faaec009870 tx=0x7faaec02b980 comp rx=0 tx=0).stop 2024-03-18T16:27:22.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.635+0000 7fab01000640 1 -- 172.21.15.102:0/2068104475 shutdown_connections 2024-03-18T16:27:22.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.635+0000 7fab01000640 1 --2- 172.21.15.102:0/2068104475 >> v2:172.21.15.102:3300/0 conn(0x7faafc0c4fa0 0x7faafc0c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:22.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.635+0000 7fab01000640 1 --2- 172.21.15.102:0/2068104475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc141650 0x7faafc158420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:22.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.635+0000 7fab01000640 1 -- 172.21.15.102:0/2068104475 >> 172.21.15.102:0/2068104475 conn(0x7faafc0cb890 msgr2=0x7faafc0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:22.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.635+0000 7fab01000640 1 -- 172.21.15.102:0/2068104475 shutdown_connections 2024-03-18T16:27:22.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.635+0000 7fab01000640 1 -- 172.21.15.102:0/2068104475 wait complete. 2024-03-18T16:27:22.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.636+0000 7fab01000640 1 Processor -- start 2024-03-18T16:27:22.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.637+0000 7fab01000640 1 -- start start 2024-03-18T16:27:22.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.638+0000 7fab01000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc0c4fa0 0x7faafc13d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:22.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.638+0000 7fab01000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faafc141650 0x7faafc1e5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:22.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.638+0000 7fab01000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faafc15c3a0 0x7faafc1e7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:22.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.638+0000 7fab01000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faafc0c66a0 con 0x7faafc141650 2024-03-18T16:27:22.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.638+0000 7fab01000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faafc0c63c0 con 0x7faafc15c3a0 2024-03-18T16:27:22.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.638+0000 7fab01000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faafc15c7c0 con 0x7faafc0c4fa0 2024-03-18T16:27:22.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.639+0000 7faafb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc0c4fa0 0x7faafc13d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:22.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.639+0000 7faafaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faafc141650 0x7faafc1e5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:22.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.639+0000 7faafbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faafc15c3a0 0x7faafc1e7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:22.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.639+0000 7faafb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc0c4fa0 0x7faafc13d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50470/0 (socket says 172.21.15.102:50470) 2024-03-18T16:27:22.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.639+0000 7faafb400640 1 -- 172.21.15.102:0/1785171390 learned_addr learned my addr 172.21.15.102:0/1785171390 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:22.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.643+0000 7faafb400640 1 -- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faafc15c3a0 msgr2=0x7faafc1e7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:22.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.643+0000 7faafb400640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faafc15c3a0 0x7faafc1e7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:22.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.643+0000 7faafb400640 1 -- 172.21.15.102:0/1785171390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faafc141650 msgr2=0x7faafc1e5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:22.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.643+0000 7faafb400640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faafc141650 0x7faafc1e5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:22.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.643+0000 7faafb400640 1 -- 172.21.15.102:0/1785171390 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faafc1e4f40 con 0x7faafc0c4fa0 2024-03-18T16:27:22.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.646+0000 7faafb400640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc0c4fa0 0x7faafc13d6a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faaec009990 tx=0x7faaec004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:22.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.648+0000 7faaf3400640 1 -- 172.21.15.102:0/1785171390 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec048070 con 0x7faafc0c4fa0 2024-03-18T16:27:22.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.648+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faafc1e6bc0 con 0x7faafc0c4fa0 2024-03-18T16:27:22.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.649+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faafc06fe80 con 0x7faafc0c4fa0 2024-03-18T16:27:22.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.650+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faafc074fd0 con 0x7faafc0c4fa0 2024-03-18T16:27:22.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.651+0000 7faaf3400640 1 -- 172.21.15.102:0/1785171390 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaec039d90 con 0x7faafc0c4fa0 2024-03-18T16:27:22.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.652+0000 7faaf3400640 1 -- 172.21.15.102:0/1785171390 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec052060 con 0x7faafc0c4fa0 2024-03-18T16:27:22.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.911+0000 7faaf3400640 1 -- 172.21.15.102:0/1785171390 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faaec04b070 con 0x7faafc0c4fa0 2024-03-18T16:27:22.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.913+0000 7faaf3400640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faadc06bc00 0x7faadc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:22.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.913+0000 7faafaa00640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faadc06bc00 0x7faadc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:22.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.914+0000 7faafaa00640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faadc06bc00 0x7faadc06e0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faafc141060 tx=0x7faae8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:22.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.915+0000 7faaf3400640 1 -- 172.21.15.102:0/1785171390 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7faaec0792b0 con 0x7faafc0c4fa0 2024-03-18T16:27:22.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:22.927+0000 7faaf3400640 1 -- 172.21.15.102:0/1785171390 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faaec07b210 con 0x7faafc0c4fa0 2024-03-18T16:27:23.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.264+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faafc1e3980 con 0x7faafc0c4fa0 2024-03-18T16:27:23.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.311+0000 7faaf3400640 1 -- 172.21.15.102:0/1785171390 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7faaec07f2b0 con 0x7faafc0c4fa0 2024-03-18T16:27:23.313 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:23.313 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:23.313 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.313+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faadc06bc00 msgr2=0x7faadc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.313+0000 7fab01000640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faadc06bc00 0x7faadc06e0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faafc141060 tx=0x7faae8009040 comp rx=0 tx=0).stop 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.313+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc0c4fa0 msgr2=0x7faafc13d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.313+0000 7fab01000640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc0c4fa0 0x7faafc13d6a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faaec009990 tx=0x7faaec004730 comp rx=0 tx=0).stop 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.314+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 shutdown_connections 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.314+0000 7fab01000640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7faadc06bc00 0x7faadc06e0c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.314+0000 7fab01000640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faafc15c3a0 0x7faafc1e7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.314+0000 7fab01000640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faafc141650 0x7faafc1e5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.314+0000 7fab01000640 1 --2- 172.21.15.102:0/1785171390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faafc0c4fa0 0x7faafc13d6a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.314+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 >> 172.21.15.102:0/1785171390 conn(0x7faafc0cb890 msgr2=0x7faafc072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.314+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 shutdown_connections 2024-03-18T16:27:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:23.314+0000 7fab01000640 1 -- 172.21.15.102:0/1785171390 wait complete. 2024-03-18T16:27:23.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:23.939 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:24.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:25.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.118+0000 7f8a15800640 1 Processor -- start 2024-03-18T16:27:25.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.119+0000 7f8a15800640 1 -- start start 2024-03-18T16:27:25.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.120+0000 7f8a15800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8a10141650 0x7f8a10158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:25.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.120+0000 7f8a15800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a1015c3a0 0x7f8a100c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:25.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.120+0000 7f8a15800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a100c4fa0 0x7f8a100c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:25.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.120+0000 7f8a15800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8a10159370 con 0x7f8a1015c3a0 2024-03-18T16:27:25.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.120+0000 7f8a15800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8a10158ad0 con 0x7f8a10141650 2024-03-18T16:27:25.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.120+0000 7f8a15800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8a10158db0 con 0x7f8a100c4fa0 2024-03-18T16:27:25.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.121+0000 7f8a0fe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8a10141650 0x7f8a10158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:25.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.121+0000 7f8a14e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a100c4fa0 0x7f8a100c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:25.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.121+0000 7f8a0f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a1015c3a0 0x7f8a100c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:25.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.121+0000 7f8a0fe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8a10141650 0x7f8a10158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35516/0 (socket says 172.21.15.102:35516) 2024-03-18T16:27:25.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.121+0000 7f8a0fe00640 1 -- 172.21.15.102:0/2039101489 learned_addr learned my addr 172.21.15.102:0/2039101489 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:25.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.124+0000 7f8a0fe00640 1 -- 172.21.15.102:0/2039101489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a100c4fa0 msgr2=0x7f8a100c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:25.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.124+0000 7f8a0fe00640 1 --2- 172.21.15.102:0/2039101489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a100c4fa0 0x7f8a100c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.124+0000 7f8a0fe00640 1 -- 172.21.15.102:0/2039101489 >> v2:172.21.15.102:3300/0 conn(0x7f8a1015c3a0 msgr2=0x7f8a100c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:25.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.124+0000 7f8a0fe00640 1 --2- 172.21.15.102:0/2039101489 >> v2:172.21.15.102:3300/0 conn(0x7f8a1015c3a0 0x7f8a100c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.124+0000 7f8a0fe00640 1 -- 172.21.15.102:0/2039101489 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a10159090 con 0x7f8a10141650 2024-03-18T16:27:25.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.128+0000 7f8a0fe00640 1 --2- 172.21.15.102:0/2039101489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a10141650 0x7f8a10158420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8a10068ac0 tx=0x7f8a0002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bfee5e167547b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:25.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.133+0000 7f8a0ea00640 1 -- 172.21.15.102:0/2039101489 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00035020 con 0x7f8a10141650 2024-03-18T16:27:25.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.134+0000 7f8a0ea00640 1 -- 172.21.15.102:0/2039101489 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a00031870 con 0x7f8a10141650 2024-03-18T16:27:25.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.134+0000 7f8a0ea00640 1 -- 172.21.15.102:0/2039101489 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a00031c40 con 0x7f8a10141650 2024-03-18T16:27:25.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.134+0000 7f8a15800640 1 -- 172.21.15.102:0/2039101489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a10141650 msgr2=0x7f8a10158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:25.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.134+0000 7f8a15800640 1 --2- 172.21.15.102:0/2039101489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a10141650 0x7f8a10158420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8a10068ac0 tx=0x7f8a0002b910 comp rx=0 tx=0).stop 2024-03-18T16:27:25.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.134+0000 7f8a15800640 1 -- 172.21.15.102:0/2039101489 shutdown_connections 2024-03-18T16:27:25.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.134+0000 7f8a15800640 1 --2- 172.21.15.102:0/2039101489 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a100c4fa0 0x7f8a100c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.134+0000 7f8a15800640 1 --2- 172.21.15.102:0/2039101489 >> v2:172.21.15.102:3300/0 conn(0x7f8a1015c3a0 0x7f8a100c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.134+0000 7f8a15800640 1 --2- 172.21.15.102:0/2039101489 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a10141650 0x7f8a10158420 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.134+0000 7f8a15800640 1 -- 172.21.15.102:0/2039101489 >> 172.21.15.102:0/2039101489 conn(0x7f8a100cb890 msgr2=0x7f8a100cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:25.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.135+0000 7f8a15800640 1 -- 172.21.15.102:0/2039101489 shutdown_connections 2024-03-18T16:27:25.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.135+0000 7f8a15800640 1 -- 172.21.15.102:0/2039101489 wait complete. 2024-03-18T16:27:25.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.136+0000 7f8a15800640 1 Processor -- start 2024-03-18T16:27:25.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.137+0000 7f8a15800640 1 -- start start 2024-03-18T16:27:25.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.137+0000 7f8a15800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a100c4fa0 0x7f8a10156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:25.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.137+0000 7f8a15800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a10141650 0x7f8a101e3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:25.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.137+0000 7f8a15800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a1015c3a0 0x7f8a1015ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:25.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.137+0000 7f8a15800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8a1015c7c0 con 0x7f8a100c4fa0 2024-03-18T16:27:25.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.137+0000 7f8a15800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8a100c66a0 con 0x7f8a1015c3a0 2024-03-18T16:27:25.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.137+0000 7f8a15800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8a10141a30 con 0x7f8a10141650 2024-03-18T16:27:25.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.138+0000 7f8a14e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a1015c3a0 0x7f8a1015ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:25.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.138+0000 7f8a0f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a10141650 0x7f8a101e3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:25.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.138+0000 7f8a0fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a100c4fa0 0x7f8a10156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:25.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.138+0000 7f8a14e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a1015c3a0 0x7f8a1015ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35532/0 (socket says 172.21.15.102:35532) 2024-03-18T16:27:25.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.138+0000 7f8a14e00640 1 -- 172.21.15.102:0/1072172135 learned_addr learned my addr 172.21.15.102:0/1072172135 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:25.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.142+0000 7f8a14e00640 1 -- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a10141650 msgr2=0x7f8a101e3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:25.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.142+0000 7f8a14e00640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a10141650 0x7f8a101e3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.142+0000 7f8a14e00640 1 -- 172.21.15.102:0/1072172135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a100c4fa0 msgr2=0x7f8a10156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:25.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.142+0000 7f8a14e00640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a100c4fa0 0x7f8a10156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.142+0000 7f8a14e00640 1 -- 172.21.15.102:0/1072172135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a101f0920 con 0x7f8a1015c3a0 2024-03-18T16:27:25.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.144+0000 7f8a14e00640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a1015c3a0 0x7f8a1015ae80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800eed0 tx=0x7f8a0800b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:25.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.149+0000 7f8a07e00640 1 -- 172.21.15.102:0/1072172135 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08018020 con 0x7f8a1015c3a0 2024-03-18T16:27:25.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.149+0000 7f8a07e00640 1 -- 172.21.15.102:0/1072172135 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a0800bcc0 con 0x7f8a1015c3a0 2024-03-18T16:27:25.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.149+0000 7f8a07e00640 1 -- 172.21.15.102:0/1072172135 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08014640 con 0x7f8a1015c3a0 2024-03-18T16:27:25.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.149+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a101f0c00 con 0x7f8a1015c3a0 2024-03-18T16:27:25.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.149+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a101f11c0 con 0x7f8a1015c3a0 2024-03-18T16:27:25.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.151+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a10074fd0 con 0x7f8a1015c3a0 2024-03-18T16:27:25.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.181+0000 7f8a07e00640 1 -- 172.21.15.102:0/1072172135 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8a08005d10 con 0x7f8a1015c3a0 2024-03-18T16:27:25.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.183+0000 7f8a07e00640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f89f006bc20 0x7f89f006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:25.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.183+0000 7f8a0fe00640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f89f006bc20 0x7f89f006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:25.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.184+0000 7f8a0fe00640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f89f006bc20 0x7f89f006e0e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8a00036000 tx=0x7f8a0003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:25.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.187+0000 7f8a07e00640 1 -- 172.21.15.102:0/1072172135 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8a08057d40 con 0x7f8a1015c3a0 2024-03-18T16:27:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.451+0000 7f8a07e00640 1 -- 172.21.15.102:0/1072172135 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8a08022030 con 0x7f8a1015c3a0 2024-03-18T16:27:25.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.796+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a100c7c60 con 0x7f8a1015c3a0 2024-03-18T16:27:25.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.817+0000 7f8a07e00640 1 -- 172.21.15.102:0/1072172135 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8a08059ca0 con 0x7f8a1015c3a0 2024-03-18T16:27:25.819 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:25.819 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:25.819 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:25.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f89f006bc20 msgr2=0x7f89f006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:25.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f89f006bc20 0x7f89f006e0e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8a00036000 tx=0x7f8a0003c040 comp rx=0 tx=0).stop 2024-03-18T16:27:25.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a1015c3a0 msgr2=0x7f8a1015ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:25.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a1015c3a0 0x7f8a1015ae80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800eed0 tx=0x7f8a0800b480 comp rx=0 tx=0).stop 2024-03-18T16:27:25.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 shutdown_connections 2024-03-18T16:27:25.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f89f006bc20 0x7f89f006e0e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a1015c3a0 0x7f8a1015ae80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8a10141650 0x7f8a101e3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 --2- 172.21.15.102:0/1072172135 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a100c4fa0 0x7f8a10156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:25.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 >> 172.21.15.102:0/1072172135 conn(0x7f8a100cb890 msgr2=0x7f8a10140040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:25.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 shutdown_connections 2024-03-18T16:27:25.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:25.819+0000 7f8a15800640 1 -- 172.21.15.102:0/1072172135 wait complete. 2024-03-18T16:27:26.444 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:26.445 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:27.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:27.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.655+0000 7f62cc600640 1 Processor -- start 2024-03-18T16:27:27.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.655+0000 7f62cc600640 1 -- start start 2024-03-18T16:27:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.656+0000 7f62cc600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62c41524c0 0x7f62c4072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.656+0000 7f62cc600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158ee0 0x7f62c406e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.656+0000 7f62cc600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62c415c200 con 0x7f62c415b8e0 2024-03-18T16:27:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.656+0000 7f62cc600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f62c4159300 con 0x7f62c41524c0 2024-03-18T16:27:27.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.656+0000 7f62cc600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f62c415c4e0 con 0x7f62c4158ee0 2024-03-18T16:27:27.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.678+0000 7f62c9c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62c41524c0 0x7f62c4072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:27.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.679+0000 7f62c9c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62c41524c0 0x7f62c4072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35546/0 (socket says 172.21.15.102:35546) 2024-03-18T16:27:27.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.679+0000 7f62c9c00640 1 -- 172.21.15.102:0/1853533549 learned_addr learned my addr 172.21.15.102:0/1853533549 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:27.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.682+0000 7f62c9c00640 1 -- 172.21.15.102:0/1853533549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158ee0 msgr2=0x7f62c406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:27.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.682+0000 7f62c9c00640 1 --2- 172.21.15.102:0/1853533549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158ee0 0x7f62c406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:27.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.682+0000 7f62c9c00640 1 -- 172.21.15.102:0/1853533549 >> v1:172.21.15.102:6789/0 conn(0x7f62c415b8e0 legacy=0x7f62c40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:27.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.682+0000 7f62c9c00640 1 -- 172.21.15.102:0/1853533549 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c415c7c0 con 0x7f62c41524c0 2024-03-18T16:27:27.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.685+0000 7f62c9c00640 1 --2- 172.21.15.102:0/1853533549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c41524c0 0x7f62c4072700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009870 tx=0x7f62b802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b81e115b1b955f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.689+0000 7f62c3e00640 1 -- 172.21.15.102:0/1853533549 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b8035020 con 0x7f62c41524c0 2024-03-18T16:27:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.689+0000 7f62c3e00640 1 -- 172.21.15.102:0/1853533549 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b8031640 con 0x7f62c41524c0 2024-03-18T16:27:27.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.689+0000 7f62c3e00640 1 -- 172.21.15.102:0/1853533549 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b802d040 con 0x7f62c41524c0 2024-03-18T16:27:27.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.690+0000 7f62cc600640 1 -- 172.21.15.102:0/1853533549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c41524c0 msgr2=0x7f62c4072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:27.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.690+0000 7f62cc600640 1 --2- 172.21.15.102:0/1853533549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c41524c0 0x7f62c4072700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f62b8009870 tx=0x7f62b802b8f0 comp rx=0 tx=0).stop 2024-03-18T16:27:27.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.691+0000 7f62cc600640 1 -- 172.21.15.102:0/1853533549 shutdown_connections 2024-03-18T16:27:27.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.691+0000 7f62cc600640 1 --2- 172.21.15.102:0/1853533549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c4158ee0 0x7f62c406e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:27.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.691+0000 7f62cc600640 1 --2- 172.21.15.102:0/1853533549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c41524c0 0x7f62c4072700 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:27.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.691+0000 7f62cc600640 1 -- 172.21.15.102:0/1853533549 >> 172.21.15.102:0/1853533549 conn(0x7f62c40700d0 msgr2=0x7f62c406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:27.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.691+0000 7f62cc600640 1 -- 172.21.15.102:0/1853533549 shutdown_connections 2024-03-18T16:27:27.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.691+0000 7f62cc600640 1 -- 172.21.15.102:0/1853533549 wait complete. 2024-03-18T16:27:27.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.693+0000 7f62cc600640 1 Processor -- start 2024-03-18T16:27:27.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.694+0000 7f62cc600640 1 -- start start 2024-03-18T16:27:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.695+0000 7f62cc600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41524c0 0x7f62c40c1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.695+0000 7f62cc600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c4158ee0 0x7f62c414a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.695+0000 7f62cc600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c415b8e0 0x7f62c414ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.695+0000 7f62cc600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f62c41589c0 con 0x7f62c41524c0 2024-03-18T16:27:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.695+0000 7f62cc600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62c41586e0 con 0x7f62c4158ee0 2024-03-18T16:27:27.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.695+0000 7f62cc600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f62c406e880 con 0x7f62c415b8e0 2024-03-18T16:27:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.696+0000 7f62c9200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c4158ee0 0x7f62c414a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.696+0000 7f62ca600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c415b8e0 0x7f62c414ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.696+0000 7f62c9200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c4158ee0 0x7f62c414a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35556/0 (socket says 172.21.15.102:35556) 2024-03-18T16:27:27.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.696+0000 7f62c9200640 1 -- 172.21.15.102:0/2581728805 learned_addr learned my addr 172.21.15.102:0/2581728805 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.699+0000 7f62c9200640 1 -- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c415b8e0 msgr2=0x7f62c414ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.699+0000 7f62c9200640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c415b8e0 0x7f62c414ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.699+0000 7f62c9200640 1 -- 172.21.15.102:0/2581728805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41524c0 msgr2=0x7f62c40c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:27.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.699+0000 7f62c9200640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41524c0 0x7f62c40c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.699+0000 7f62c9200640 1 -- 172.21.15.102:0/2581728805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c41efa00 con 0x7f62c4158ee0 2024-03-18T16:27:27.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.703+0000 7f62c9200640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c4158ee0 0x7f62c414a620 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f62b000b500 tx=0x7f62b000b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.709+0000 7f62c1600640 1 -- 172.21.15.102:0/2581728805 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0014070 con 0x7f62c4158ee0 2024-03-18T16:27:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.709+0000 7f62c1600640 1 -- 172.21.15.102:0/2581728805 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b0010870 con 0x7f62c4158ee0 2024-03-18T16:27:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.709+0000 7f62c1600640 1 -- 172.21.15.102:0/2581728805 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0010c60 con 0x7f62c4158ee0 2024-03-18T16:27:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.709+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c41efce0 con 0x7f62c4158ee0 2024-03-18T16:27:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.709+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c41f02a0 con 0x7f62c4158ee0 2024-03-18T16:27:27.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.711+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c4074fd0 con 0x7f62c4158ee0 2024-03-18T16:27:27.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.987+0000 7f62c1600640 1 -- 172.21.15.102:0/2581728805 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f62b0017070 con 0x7f62c4158ee0 2024-03-18T16:27:27.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.988+0000 7f62c1600640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62a406bc20 0x7f62a406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:27.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.988+0000 7f62c1600640 1 -- 172.21.15.102:0/2581728805 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f62b0057150 con 0x7f62c4158ee0 2024-03-18T16:27:27.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.989+0000 7f62c9c00640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62a406bc20 0x7f62a406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:27.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:27.990+0000 7f62c9c00640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62a406bc20 0x7f62a406e0e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f62b8036000 tx=0x7f62b8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:28.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.004+0000 7f62c1600640 1 -- 172.21.15.102:0/2581728805 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f62b00590b0 con 0x7f62c4158ee0 2024-03-18T16:27:28.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.257+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62c406c590 con 0x7f62c4158ee0 2024-03-18T16:27:28.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.269+0000 7f62c1600640 1 -- 172.21.15.102:0/2581728805 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f62b005d150 con 0x7f62c4158ee0 2024-03-18T16:27:28.271 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:28.271 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:28.272 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:28.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62a406bc20 msgr2=0x7f62a406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:28.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62a406bc20 0x7f62a406e0e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f62b8036000 tx=0x7f62b8030000 comp rx=0 tx=0).stop 2024-03-18T16:27:28.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c4158ee0 msgr2=0x7f62c414a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:28.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c4158ee0 0x7f62c414a620 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f62b000b500 tx=0x7f62b000b9c0 comp rx=0 tx=0).stop 2024-03-18T16:27:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 shutdown_connections 2024-03-18T16:27:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f62a406bc20 0x7f62a406e0e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f62c415b8e0 0x7f62c414ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62c4158ee0 0x7f62c414a620 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 --2- 172.21.15.102:0/2581728805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41524c0 0x7f62c40c1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 >> 172.21.15.102:0/2581728805 conn(0x7f62c40700d0 msgr2=0x7f62c40c51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.273+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 shutdown_connections 2024-03-18T16:27:28.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:28.274+0000 7f62cc600640 1 -- 172.21.15.102:0/2581728805 wait complete. 2024-03-18T16:27:28.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:28.902 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:29.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:30.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.077+0000 7fcd22200640 1 Processor -- start 2024-03-18T16:27:30.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.077+0000 7fcd22200640 1 -- start start 2024-03-18T16:27:30.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.078+0000 7fcd22200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcd1c154770 0x7fcd1c15ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:30.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.078+0000 7fcd22200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcd1c15e290 0x7fcd1c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:30.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.078+0000 7fcd22200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c0ce2f0 0x7fcd1c0ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:30.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.078+0000 7fcd22200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fcd1c15e7f0 con 0x7fcd1c15e290 2024-03-18T16:27:30.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.078+0000 7fcd22200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcd1c15a790 con 0x7fcd1c154770 2024-03-18T16:27:30.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.078+0000 7fcd22200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1c0cdf60 con 0x7fcd1c0ce2f0 2024-03-18T16:27:30.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.079+0000 7fcd1be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcd1c15e290 0x7fcd1c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:30.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.080+0000 7fcd1be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcd1c15e290 0x7fcd1c0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42222/0 (socket says 172.21.15.102:42222) 2024-03-18T16:27:30.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.080+0000 7fcd1be00640 1 -- 172.21.15.102:0/3172805445 learned_addr learned my addr 172.21.15.102:0/3172805445 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:30.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.080+0000 7fcd21800640 1 --2- 172.21.15.102:0/3172805445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c0ce2f0 0x7fcd1c0ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:30.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.081+0000 7fcd20e00640 1 --2- 172.21.15.102:0/3172805445 >> v2:172.21.15.154:3300/0 conn(0x7fcd1c154770 0x7fcd1c15ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.083+0000 7fcd1be00640 1 -- 172.21.15.102:0/3172805445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c0ce2f0 msgr2=0x7fcd1c0ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.083+0000 7fcd1be00640 1 --2- 172.21.15.102:0/3172805445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c0ce2f0 0x7fcd1c0ccaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.083+0000 7fcd1be00640 1 -- 172.21.15.102:0/3172805445 >> v2:172.21.15.154:3300/0 conn(0x7fcd1c154770 msgr2=0x7fcd1c15ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.083+0000 7fcd1be00640 1 --2- 172.21.15.102:0/3172805445 >> v2:172.21.15.154:3300/0 conn(0x7fcd1c154770 0x7fcd1c15ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.083+0000 7fcd1be00640 1 -- 172.21.15.102:0/3172805445 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1c0ca5c0 con 0x7fcd1c15e290 2024-03-18T16:27:30.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.086+0000 7fcd1be00640 1 --2- 172.21.15.102:0/3172805445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd1c15e290 0x7fcd1c0cc560 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c06bbd0 tx=0x7fcd0802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8b3480461d5bca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.094+0000 7fcd1b400640 1 -- 172.21.15.102:0/3172805445 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd080317c0 con 0x7fcd1c15e290 2024-03-18T16:27:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.094+0000 7fcd1b400640 1 -- 172.21.15.102:0/3172805445 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd08037070 con 0x7fcd1c15e290 2024-03-18T16:27:30.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.094+0000 7fcd1b400640 1 -- 172.21.15.102:0/3172805445 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd08031b80 con 0x7fcd1c15e290 2024-03-18T16:27:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.095+0000 7fcd22200640 1 -- 172.21.15.102:0/3172805445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd1c15e290 msgr2=0x7fcd1c0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.095+0000 7fcd22200640 1 --2- 172.21.15.102:0/3172805445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd1c15e290 0x7fcd1c0cc560 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c06bbd0 tx=0x7fcd0802b860 comp rx=0 tx=0).stop 2024-03-18T16:27:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.095+0000 7fcd22200640 1 -- 172.21.15.102:0/3172805445 shutdown_connections 2024-03-18T16:27:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.095+0000 7fcd22200640 1 --2- 172.21.15.102:0/3172805445 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c0ce2f0 0x7fcd1c0ccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.095+0000 7fcd22200640 1 --2- 172.21.15.102:0/3172805445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd1c15e290 0x7fcd1c0cc560 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.095+0000 7fcd22200640 1 --2- 172.21.15.102:0/3172805445 >> v2:172.21.15.154:3300/0 conn(0x7fcd1c154770 0x7fcd1c15ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.095+0000 7fcd22200640 1 -- 172.21.15.102:0/3172805445 >> 172.21.15.102:0/3172805445 conn(0x7fcd1c066f80 msgr2=0x7fcd1c067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:30.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.095+0000 7fcd22200640 1 -- 172.21.15.102:0/3172805445 shutdown_connections 2024-03-18T16:27:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.095+0000 7fcd22200640 1 -- 172.21.15.102:0/3172805445 wait complete. 2024-03-18T16:27:30.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.097+0000 7fcd22200640 1 Processor -- start 2024-03-18T16:27:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.097+0000 7fcd22200640 1 -- start start 2024-03-18T16:27:30.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.098+0000 7fcd22200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd1c0ce2f0 0x7fcd1c0b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.098+0000 7fcd22200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c154770 0x7fcd1c0b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.098+0000 7fcd22200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd1c15e290 0x7fcd1c1f0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.098+0000 7fcd22200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c15b160 con 0x7fcd1c0ce2f0 2024-03-18T16:27:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.098+0000 7fcd22200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c15a420 con 0x7fcd1c15e290 2024-03-18T16:27:30.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.098+0000 7fcd22200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1c0ced90 con 0x7fcd1c154770 2024-03-18T16:27:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.099+0000 7fcd1be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c154770 0x7fcd1c0b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.099+0000 7fcd20e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd1c0ce2f0 0x7fcd1c0b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.099+0000 7fcd21800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd1c15e290 0x7fcd1c1f0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.099+0000 7fcd21800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd1c15e290 0x7fcd1c1f0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:35570/0 (socket says 172.21.15.102:35570) 2024-03-18T16:27:30.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.099+0000 7fcd21800640 1 -- 172.21.15.102:0/3096017771 learned_addr learned my addr 172.21.15.102:0/3096017771 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:30.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.103+0000 7fcd21800640 1 -- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c154770 msgr2=0x7fcd1c0b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.103+0000 7fcd21800640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c154770 0x7fcd1c0b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.103+0000 7fcd21800640 1 -- 172.21.15.102:0/3096017771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd1c0ce2f0 msgr2=0x7fcd1c0b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.103+0000 7fcd21800640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd1c0ce2f0 0x7fcd1c0b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.103+0000 7fcd21800640 1 -- 172.21.15.102:0/3096017771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1c0b5240 con 0x7fcd1c15e290 2024-03-18T16:27:30.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.106+0000 7fcd21800640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd1c15e290 0x7fcd1c1f0490 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400ef00 tx=0x7fcd1400b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.110+0000 7fcd18c00640 1 -- 172.21.15.102:0/3096017771 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14018020 con 0x7fcd1c15e290 2024-03-18T16:27:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.110+0000 7fcd18c00640 1 -- 172.21.15.102:0/3096017771 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd1400bd20 con 0x7fcd1c15e290 2024-03-18T16:27:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.110+0000 7fcd18c00640 1 -- 172.21.15.102:0/3096017771 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14014640 con 0x7fcd1c15e290 2024-03-18T16:27:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.110+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd1c0b5520 con 0x7fcd1c15e290 2024-03-18T16:27:30.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.111+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd1c1f3e80 con 0x7fcd1c15e290 2024-03-18T16:27:30.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.112+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd1c15af50 con 0x7fcd1c15e290 2024-03-18T16:27:30.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.133+0000 7fcd18c00640 1 -- 172.21.15.102:0/3096017771 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcd14005d10 con 0x7fcd1c15e290 2024-03-18T16:27:30.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.134+0000 7fcd18c00640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fccfc06bc20 0x7fccfc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:30.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.135+0000 7fcd20e00640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fccfc06bc20 0x7fccfc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:30.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.135+0000 7fcd20e00640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fccfc06bc20 0x7fccfc06e0e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcd10009b40 tx=0x7fcd10009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:30.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.376+0000 7fcd18c00640 1 -- 172.21.15.102:0/3096017771 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fcd14057ca0 con 0x7fcd1c15e290 2024-03-18T16:27:30.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.387+0000 7fcd18c00640 1 -- 172.21.15.102:0/3096017771 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcd1c15af50 con 0x7fcd1c15e290 2024-03-18T16:27:30.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.628+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd1c14f170 con 0x7fcd1c15e290 2024-03-18T16:27:30.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.640+0000 7fcd18c00640 1 -- 172.21.15.102:0/3096017771 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fcd1c14f170 con 0x7fcd1c15e290 2024-03-18T16:27:30.642 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:30.643 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:30.643 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fccfc06bc20 msgr2=0x7fccfc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fccfc06bc20 0x7fccfc06e0e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcd10009b40 tx=0x7fcd10009190 comp rx=0 tx=0).stop 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd1c15e290 msgr2=0x7fcd1c1f0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd1c15e290 0x7fcd1c1f0490 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400ef00 tx=0x7fcd1400b480 comp rx=0 tx=0).stop 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 shutdown_connections 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fccfc06bc20 0x7fccfc06e0e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd1c15e290 0x7fcd1c1f0490 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcd1c154770 0x7fcd1c0b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 --2- 172.21.15.102:0/3096017771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd1c0ce2f0 0x7fcd1c0b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 >> 172.21.15.102:0/3096017771 conn(0x7fcd1c066f80 msgr2=0x7fcd1c152650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:30.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 shutdown_connections 2024-03-18T16:27:30.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:30.643+0000 7fcd22200640 1 -- 172.21.15.102:0/3096017771 wait complete. 2024-03-18T16:27:31.273 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:31.273 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:32.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:32.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.445+0000 7fe17b200640 1 Processor -- start 2024-03-18T16:27:32.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.445+0000 7fe17b200640 1 -- start start 2024-03-18T16:27:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.446+0000 7fe17b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe174158660 0x7fe174158a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.446+0000 7fe17b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17406c110 con 0x7fe1740c2b40 2024-03-18T16:27:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.446+0000 7fe17b200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe174158fd0 con 0x7fe17406b200 2024-03-18T16:27:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.446+0000 7fe17b200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe1741581d0 con 0x7fe174158660 2024-03-18T16:27:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.447+0000 7fe179400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe17406b200 0x7fe1740c5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:56138/0 (socket says 172.21.15.102:56138) 2024-03-18T16:27:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.447+0000 7fe179400640 1 -- 172.21.15.102:0/3764865888 learned_addr learned my addr 172.21.15.102:0/3764865888 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:32.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.447+0000 7fe17a800640 1 --2- 172.21.15.102:0/3764865888 >> v2:172.21.15.102:3300/0 conn(0x7fe174158660 0x7fe174158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.450+0000 7fe178a00640 1 -- 172.21.15.102:0/3764865888 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2313151305 0 0) 0x7fe174158fd0 con 0x7fe17406b200 2024-03-18T16:27:32.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.450+0000 7fe178a00640 1 -- 172.21.15.102:0/3764865888 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe15c003520 con 0x7fe17406b200 2024-03-18T16:27:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.452+0000 7fe17a800640 1 -- 172.21.15.102:0/3764865888 >> v1:172.21.15.154:6789/0 conn(0x7fe17406b200 legacy=0x7fe1740c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.452+0000 7fe17a800640 1 -- 172.21.15.102:0/3764865888 >> v1:172.21.15.102:6789/0 conn(0x7fe1740c2b40 legacy=0x7fe17406ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:32.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.452+0000 7fe17a800640 1 -- 172.21.15.102:0/3764865888 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1740c6cd0 con 0x7fe174158660 2024-03-18T16:27:32.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.458+0000 7fe17a800640 1 --2- 172.21.15.102:0/3764865888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe174158660 0x7fe174158a90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe170009d90 tx=0x7fe17002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86248c187a435db9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.464+0000 7fe178a00640 1 -- 172.21.15.102:0/3764865888 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170039070 con 0x7fe174158660 2024-03-18T16:27:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.464+0000 7fe178a00640 1 -- 172.21.15.102:0/3764865888 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe17002cbd0 con 0x7fe174158660 2024-03-18T16:27:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.464+0000 7fe178a00640 1 -- 172.21.15.102:0/3764865888 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170034440 con 0x7fe174158660 2024-03-18T16:27:32.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.465+0000 7fe17b200640 1 -- 172.21.15.102:0/3764865888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe174158660 msgr2=0x7fe174158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:32.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.465+0000 7fe17b200640 1 --2- 172.21.15.102:0/3764865888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe174158660 0x7fe174158a90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe170009d90 tx=0x7fe17002be20 comp rx=0 tx=0).stop 2024-03-18T16:27:32.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.465+0000 7fe17b200640 1 -- 172.21.15.102:0/3764865888 shutdown_connections 2024-03-18T16:27:32.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.465+0000 7fe17b200640 1 --2- 172.21.15.102:0/3764865888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe174158660 0x7fe174158a90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:32.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.465+0000 7fe17b200640 1 -- 172.21.15.102:0/3764865888 >> 172.21.15.102:0/3764865888 conn(0x7fe1740727e0 msgr2=0x7fe174072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:32.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.465+0000 7fe17b200640 1 -- 172.21.15.102:0/3764865888 shutdown_connections 2024-03-18T16:27:32.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.466+0000 7fe17b200640 1 -- 172.21.15.102:0/3764865888 wait complete. 2024-03-18T16:27:32.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.468+0000 7fe17b200640 1 Processor -- start 2024-03-18T16:27:32.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.468+0000 7fe17b200640 1 -- start start 2024-03-18T16:27:32.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.469+0000 7fe17b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17406b200 0x7fe174154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:32.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.469+0000 7fe17b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1740c2b40 0x7fe1740cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.469+0000 7fe17b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe174158660 0x7fe1740cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.469+0000 7fe17b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe1740c7990 con 0x7fe17406b200 2024-03-18T16:27:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.469+0000 7fe17b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe1740c76b0 con 0x7fe1740c2b40 2024-03-18T16:27:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.469+0000 7fe17b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe1740c7c70 con 0x7fe174158660 2024-03-18T16:27:32.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.469+0000 7fe179e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17406b200 0x7fe174154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:32.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.469+0000 7fe179400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1740c2b40 0x7fe1740cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:32.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.469+0000 7fe17a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe174158660 0x7fe1740cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:32.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.470+0000 7fe179400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1740c2b40 0x7fe1740cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38110/0 (socket says 172.21.15.102:38110) 2024-03-18T16:27:32.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.470+0000 7fe179400640 1 -- 172.21.15.102:0/2112837117 learned_addr learned my addr 172.21.15.102:0/2112837117 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:32.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.474+0000 7fe179400640 1 -- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe174158660 msgr2=0x7fe1740cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:32.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.474+0000 7fe179400640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe174158660 0x7fe1740cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:32.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.474+0000 7fe179400640 1 -- 172.21.15.102:0/2112837117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17406b200 msgr2=0x7fe174154870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:32.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.474+0000 7fe179400640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17406b200 0x7fe174154870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:32.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.474+0000 7fe179400640 1 -- 172.21.15.102:0/2112837117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17413a1e0 con 0x7fe1740c2b40 2024-03-18T16:27:32.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.474+0000 7fe179e00640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17406b200 0x7fe174154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:27:32.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.476+0000 7fe179400640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1740c2b40 0x7fe1740cb300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe1640010e0 tx=0x7fe16400c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:32.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.481+0000 7fe16e000640 1 -- 172.21.15.102:0/2112837117 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164017070 con 0x7fe1740c2b40 2024-03-18T16:27:32.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.481+0000 7fe16e000640 1 -- 172.21.15.102:0/2112837117 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe164005d00 con 0x7fe1740c2b40 2024-03-18T16:27:32.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.481+0000 7fe16e000640 1 -- 172.21.15.102:0/2112837117 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164007310 con 0x7fe1740c2b40 2024-03-18T16:27:32.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.482+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1740cd460 con 0x7fe1740c2b40 2024-03-18T16:27:32.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.482+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1740cd180 con 0x7fe1740c2b40 2024-03-18T16:27:32.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.483+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe174074fd0 con 0x7fe1740c2b40 2024-03-18T16:27:32.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.743+0000 7fe16e000640 1 -- 172.21.15.102:0/2112837117 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe16401a070 con 0x7fe1740c2b40 2024-03-18T16:27:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.746+0000 7fe16e000640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15806bbd0 0x7fe15806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:32.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.746+0000 7fe16e000640 1 -- 172.21.15.102:0/2112837117 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe16408a5f0 con 0x7fe1740c2b40 2024-03-18T16:27:32.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.746+0000 7fe179e00640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15806bbd0 0x7fe15806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:32.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.747+0000 7fe179e00640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15806bbd0 0x7fe15806e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe168001ad0 tx=0x7fe168009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:32.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:32.756+0000 7fe16e000640 1 -- 172.21.15.102:0/2112837117 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe164021020 con 0x7fe1740c2b40 2024-03-18T16:27:33.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.016+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe17406b7e0 con 0x7fe1740c2b40 2024-03-18T16:27:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.099+0000 7fe16e000640 1 -- 172.21.15.102:0/2112837117 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe164058680 con 0x7fe1740c2b40 2024-03-18T16:27:33.101 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:33.101 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.102+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15806bbd0 msgr2=0x7fe15806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.102+0000 7fe17b200640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15806bbd0 0x7fe15806e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe168001ad0 tx=0x7fe168009040 comp rx=0 tx=0).stop 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.102+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1740c2b40 msgr2=0x7fe1740cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.102+0000 7fe17b200640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1740c2b40 0x7fe1740cb300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe1640010e0 tx=0x7fe16400c890 comp rx=0 tx=0).stop 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.103+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 shutdown_connections 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.103+0000 7fe17b200640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15806bbd0 0x7fe15806e090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.103+0000 7fe17b200640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe174158660 0x7fe1740cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.103+0000 7fe17b200640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1740c2b40 0x7fe1740cb300 secure :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe1640010e0 tx=0x7fe16400c890 comp rx=0 tx=0).stop 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.103+0000 7fe17b200640 1 --2- 172.21.15.102:0/2112837117 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17406b200 0x7fe174154870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.103+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 >> 172.21.15.102:0/2112837117 conn(0x7fe1740727e0 msgr2=0x7fe17413c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.103+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 shutdown_connections 2024-03-18T16:27:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:33.103+0000 7fe17b200640 1 -- 172.21.15.102:0/2112837117 wait complete. 2024-03-18T16:27:33.734 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:33.734 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:34.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.907+0000 7f32e0a00640 1 Processor -- start 2024-03-18T16:27:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.908+0000 7f32e0a00640 1 -- start start 2024-03-18T16:27:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.909+0000 7f32e0a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f32dc15a310 0x7f32dc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:34.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.909+0000 7f32e0a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a6f0 0x7f32dc15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.909+0000 7f32e0a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32dc0ca500 con 0x7f32dc15baa0 2024-03-18T16:27:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.909+0000 7f32e0a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f32dc0cd410 con 0x7f32dc15a310 2024-03-18T16:27:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.909+0000 7f32e0a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f32dc15b050 con 0x7f32dc15a6f0 2024-03-18T16:27:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.910+0000 7f32db400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f32dc15a310 0x7f32dc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.910+0000 7f32daa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a6f0 0x7f32dc15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.911+0000 7f32db400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f32dc15a310 0x7f32dc06d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38118/0 (socket says 172.21.15.102:38118) 2024-03-18T16:27:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.911+0000 7f32db400640 1 -- 172.21.15.102:0/3260146054 learned_addr learned my addr 172.21.15.102:0/3260146054 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:34.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.914+0000 7f32daa00640 1 -- 172.21.15.102:0/3260146054 >> v2:172.21.15.154:3300/0 conn(0x7f32dc15a310 msgr2=0x7f32dc06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:34.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.914+0000 7f32daa00640 1 --2- 172.21.15.102:0/3260146054 >> v2:172.21.15.154:3300/0 conn(0x7f32dc15a310 0x7f32dc06d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:34.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.914+0000 7f32daa00640 1 -- 172.21.15.102:0/3260146054 >> v1:172.21.15.102:6789/0 conn(0x7f32dc15baa0 legacy=0x7f32dc15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:34.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.914+0000 7f32daa00640 1 -- 172.21.15.102:0/3260146054 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32dc0ca7a0 con 0x7f32dc15a6f0 2024-03-18T16:27:34.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.914+0000 7f32db400640 1 --2- 172.21.15.102:0/3260146054 >> v2:172.21.15.154:3300/0 conn(0x7f32dc15a310 0x7f32dc06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:27:34.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.917+0000 7f32daa00640 1 --2- 172.21.15.102:0/3260146054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a6f0 0x7f32dc15ab10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f32c80097a0 tx=0x7f32c802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=468e2caf25f7c68e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:34.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.924+0000 7f32da000640 1 -- 172.21.15.102:0/3260146054 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8031780 con 0x7f32dc15a6f0 2024-03-18T16:27:34.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.925+0000 7f32da000640 1 -- 172.21.15.102:0/3260146054 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c8037070 con 0x7f32dc15a6f0 2024-03-18T16:27:34.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.925+0000 7f32da000640 1 -- 172.21.15.102:0/3260146054 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8031b40 con 0x7f32dc15a6f0 2024-03-18T16:27:34.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.925+0000 7f32e0a00640 1 -- 172.21.15.102:0/3260146054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a6f0 msgr2=0x7f32dc15ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:34.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.925+0000 7f32e0a00640 1 --2- 172.21.15.102:0/3260146054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a6f0 0x7f32dc15ab10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f32c80097a0 tx=0x7f32c802b820 comp rx=0 tx=0).stop 2024-03-18T16:27:34.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.926+0000 7f32e0a00640 1 -- 172.21.15.102:0/3260146054 shutdown_connections 2024-03-18T16:27:34.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.926+0000 7f32e0a00640 1 --2- 172.21.15.102:0/3260146054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a6f0 0x7f32dc15ab10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:34.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.926+0000 7f32e0a00640 1 --2- 172.21.15.102:0/3260146054 >> v2:172.21.15.154:3300/0 conn(0x7f32dc15a310 0x7f32dc06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:34.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.926+0000 7f32e0a00640 1 -- 172.21.15.102:0/3260146054 >> 172.21.15.102:0/3260146054 conn(0x7f32dc08c1a0 msgr2=0x7f32dc066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:34.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.926+0000 7f32e0a00640 1 -- 172.21.15.102:0/3260146054 shutdown_connections 2024-03-18T16:27:34.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.926+0000 7f32e0a00640 1 -- 172.21.15.102:0/3260146054 wait complete. 2024-03-18T16:27:34.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.927+0000 7f32e0a00640 1 Processor -- start 2024-03-18T16:27:34.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.928+0000 7f32e0a00640 1 -- start start 2024-03-18T16:27:34.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.929+0000 7f32e0a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a310 0x7f32dc1f0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:34.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.929+0000 7f32e0a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32dc15a6f0 0x7f32dc1f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:34.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.929+0000 7f32e0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32dc15baa0 0x7f32dc157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:34.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.929+0000 7f32e0a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f32dc0cd130 con 0x7f32dc15baa0 2024-03-18T16:27:34.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.929+0000 7f32e0a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32dc0cce50 con 0x7f32dc15a6f0 2024-03-18T16:27:34.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.929+0000 7f32e0a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f32dc06d920 con 0x7f32dc15a310 2024-03-18T16:27:34.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.929+0000 7f32db400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a310 0x7f32dc1f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:34.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.929+0000 7f32daa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32dc15a6f0 0x7f32dc1f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:34.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.930+0000 7f32daa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32dc15a6f0 0x7f32dc1f1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38122/0 (socket says 172.21.15.102:38122) 2024-03-18T16:27:34.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.930+0000 7f32daa00640 1 -- 172.21.15.102:0/2724555944 learned_addr learned my addr 172.21.15.102:0/2724555944 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:34.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.934+0000 7f32daa00640 1 -- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a310 msgr2=0x7f32dc1f0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:34.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.934+0000 7f32daa00640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a310 0x7f32dc1f0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:34.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.934+0000 7f32daa00640 1 -- 172.21.15.102:0/2724555944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32dc15baa0 msgr2=0x7f32dc157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:34.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.934+0000 7f32daa00640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32dc15baa0 0x7f32dc157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:34.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.934+0000 7f32daa00640 1 -- 172.21.15.102:0/2724555944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32dc155ee0 con 0x7f32dc15a6f0 2024-03-18T16:27:34.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.938+0000 7f32daa00640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32dc15a6f0 0x7f32dc1f1c90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f32c80098c0 tx=0x7f32c80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:34.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.943+0000 7f32d3400640 1 -- 172.21.15.102:0/2724555944 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8032610 con 0x7f32dc15a6f0 2024-03-18T16:27:34.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.943+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32dc1561c0 con 0x7f32dc15a6f0 2024-03-18T16:27:34.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.943+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32dc156780 con 0x7f32dc15a6f0 2024-03-18T16:27:34.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:34.944+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32dc0cba30 con 0x7f32dc15a6f0 2024-03-18T16:27:35.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.154+0000 7f32d3400640 1 -- 172.21.15.102:0/2724555944 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c8037020 con 0x7f32dc15a6f0 2024-03-18T16:27:35.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.227+0000 7f32d3400640 1 -- 172.21.15.102:0/2724555944 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8005b90 con 0x7f32dc15a6f0 2024-03-18T16:27:35.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.229+0000 7f32d3400640 1 -- 172.21.15.102:0/2724555944 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f32c804c060 con 0x7f32dc15a6f0 2024-03-18T16:27:35.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.232+0000 7f32d3400640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f32bc06bbd0 0x7f32bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:35.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.233+0000 7f32db400640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f32bc06bbd0 0x7f32bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:35.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.233+0000 7f32db400640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f32bc06bbd0 0x7f32bc06e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f32cc006ca0 tx=0x7f32cc009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:35.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.235+0000 7f32d3400640 1 -- 172.21.15.102:0/2724555944 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f32c8078520 con 0x7f32dc15a6f0 2024-03-18T16:27:35.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.254+0000 7f32d3400640 1 -- 172.21.15.102:0/2724555944 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f32dc0cba30 con 0x7f32dc15a6f0 2024-03-18T16:27:35.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.493+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32dc1548c0 con 0x7f32dc15a6f0 2024-03-18T16:27:35.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.520+0000 7f32d3400640 1 -- 172.21.15.102:0/2724555944 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f32dc1548c0 con 0x7f32dc15a6f0 2024-03-18T16:27:35.523 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:35.523 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:35.523 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:35.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f32bc06bbd0 msgr2=0x7f32bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:35.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f32bc06bbd0 0x7f32bc06e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f32cc006ca0 tx=0x7f32cc009240 comp rx=0 tx=0).stop 2024-03-18T16:27:35.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32dc15a6f0 msgr2=0x7f32dc1f1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:35.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32dc15a6f0 0x7f32dc1f1c90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f32c80098c0 tx=0x7f32c80046c0 comp rx=0 tx=0).stop 2024-03-18T16:27:35.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 shutdown_connections 2024-03-18T16:27:35.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f32bc06bbd0 0x7f32bc06e090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:35.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32dc15baa0 0x7f32dc157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:35.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32dc15a6f0 0x7f32dc1f1c90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:35.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 --2- 172.21.15.102:0/2724555944 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f32dc15a310 0x7f32dc1f0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:35.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 >> 172.21.15.102:0/2724555944 conn(0x7f32dc08c1a0 msgr2=0x7f32dc14eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:35.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 shutdown_connections 2024-03-18T16:27:35.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:35.524+0000 7f32e0a00640 1 -- 172.21.15.102:0/2724555944 wait complete. 2024-03-18T16:27:36.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:36.154 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:37.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:37.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.341+0000 7fe3d2a00640 1 Processor -- start 2024-03-18T16:27:37.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.341+0000 7fe3d2a00640 1 -- start start 2024-03-18T16:27:37.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.342+0000 7fe3d2a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe3cc0cfe70 0x7fe3cc15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:37.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.342+0000 7fe3d2a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.342+0000 7fe3d2a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3cc0cabd0 con 0x7fe3cc0d0250 2024-03-18T16:27:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.342+0000 7fe3d2a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe3cc0caea0 con 0x7fe3cc0cfe70 2024-03-18T16:27:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.342+0000 7fe3d2a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe3cc0cb180 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.343+0000 7fe3d1600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe3cc0cfe70 0x7fe3cc15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.343+0000 7fe3d2000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.344+0000 7fe3d1600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe3cc0cfe70 0x7fe3cc15ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38134/0 (socket says 172.21.15.102:38134) 2024-03-18T16:27:37.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.344+0000 7fe3d1600640 1 -- 172.21.15.102:0/1924556560 learned_addr learned my addr 172.21.15.102:0/1924556560 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:37.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.346+0000 7fe3d1600640 1 -- 172.21.15.102:0/1924556560 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cc2f0 msgr2=0x7fe3cc0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:37.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.346+0000 7fe3d1600640 1 --2- 172.21.15.102:0/1924556560 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:37.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.346+0000 7fe3d1600640 1 -- 172.21.15.102:0/1924556560 >> v1:172.21.15.102:6789/0 conn(0x7fe3cc0d0250 legacy=0x7fe3cc0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:37.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.346+0000 7fe3d1600640 1 -- 172.21.15.102:0/1924556560 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3cc0cb460 con 0x7fe3cc0cfe70 2024-03-18T16:27:37.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.349+0000 7fe3d1600640 1 --2- 172.21.15.102:0/1924556560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cfe70 0x7fe3cc15ae40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009870 tx=0x7fe3c002ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd04f546acf61132 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:37.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.353+0000 7fe3cbe00640 1 -- 172.21.15.102:0/1924556560 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c002f840 con 0x7fe3cc0cfe70 2024-03-18T16:27:37.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.353+0000 7fe3cbe00640 1 -- 172.21.15.102:0/1924556560 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c002f9e0 con 0x7fe3cc0cfe70 2024-03-18T16:27:37.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.353+0000 7fe3cbe00640 1 -- 172.21.15.102:0/1924556560 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c002fdb0 con 0x7fe3cc0cfe70 2024-03-18T16:27:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.354+0000 7fe3d2a00640 1 -- 172.21.15.102:0/1924556560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cfe70 msgr2=0x7fe3cc15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.354+0000 7fe3d2a00640 1 --2- 172.21.15.102:0/1924556560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cfe70 0x7fe3cc15ae40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0009870 tx=0x7fe3c002ba20 comp rx=0 tx=0).stop 2024-03-18T16:27:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.354+0000 7fe3d2a00640 1 -- 172.21.15.102:0/1924556560 shutdown_connections 2024-03-18T16:27:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.354+0000 7fe3d2a00640 1 --2- 172.21.15.102:0/1924556560 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.354+0000 7fe3d2a00640 1 --2- 172.21.15.102:0/1924556560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cfe70 0x7fe3cc15ae40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.354+0000 7fe3d2a00640 1 -- 172.21.15.102:0/1924556560 >> 172.21.15.102:0/1924556560 conn(0x7fe3cc08c3e0 msgr2=0x7fe3cc06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.354+0000 7fe3d2a00640 1 -- 172.21.15.102:0/1924556560 shutdown_connections 2024-03-18T16:27:37.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.354+0000 7fe3d2a00640 1 -- 172.21.15.102:0/1924556560 wait complete. 2024-03-18T16:27:37.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.356+0000 7fe3d2a00640 1 Processor -- start 2024-03-18T16:27:37.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.356+0000 7fe3d2a00640 1 -- start start 2024-03-18T16:27:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.357+0000 7fe3d2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.357+0000 7fe3d2a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cfe70 0x7fe3cc1e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.357+0000 7fe3d2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3cc0d0250 0x7fe3cc1e08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.357+0000 7fe3d2a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe3cc0cbd00 con 0x7fe3cc0d0250 2024-03-18T16:27:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.357+0000 7fe3d2a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe3cc0cba20 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.357+0000 7fe3d2a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe3cc0ccc50 con 0x7fe3cc0cfe70 2024-03-18T16:27:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.358+0000 7fe3d1600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.358+0000 7fe3d0c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cfe70 0x7fe3cc1e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.358+0000 7fe3d2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3cc0d0250 0x7fe3cc1e08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.358+0000 7fe3d1600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0c7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38146/0 (socket says 172.21.15.102:38146) 2024-03-18T16:27:37.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.358+0000 7fe3d1600640 1 -- 172.21.15.102:0/2769623101 learned_addr learned my addr 172.21.15.102:0/2769623101 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:37.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.362+0000 7fe3d1600640 1 -- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cfe70 msgr2=0x7fe3cc1e03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.362+0000 7fe3d1600640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cfe70 0x7fe3cc1e03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.362+0000 7fe3d1600640 1 -- 172.21.15.102:0/2769623101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3cc0d0250 msgr2=0x7fe3cc1e08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.362+0000 7fe3d1600640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3cc0d0250 0x7fe3cc1e08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.362+0000 7fe3d1600640 1 -- 172.21.15.102:0/2769623101 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3cc1f3db0 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.365+0000 7fe3d1600640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0c7770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe3c002ba60 tx=0x7fe3c0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:37.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.367+0000 7fe3c9600640 1 -- 172.21.15.102:0/2769623101 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0047070 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.367+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3cc1f2f50 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.367+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3cc1f3510 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.369+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3cc074fd0 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.628+0000 7fe3c9600640 1 -- 172.21.15.102:0/2769623101 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c002f940 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.628+0000 7fe3c9600640 1 -- 172.21.15.102:0/2769623101 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0051060 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.630+0000 7fe3c9600640 1 -- 172.21.15.102:0/2769623101 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe3c004a070 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.631+0000 7fe3c9600640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3ac06bc00 0x7fe3ac06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:37.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.632+0000 7fe3d0c00640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3ac06bc00 0x7fe3ac06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:37.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.633+0000 7fe3d0c00640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3ac06bc00 0x7fe3ac06e0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0cf020 tx=0x7fe3b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:37.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.633+0000 7fe3c9600640 1 -- 172.21.15.102:0/2769623101 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe3c0079ae0 con 0x7fe3cc0cc2f0 2024-03-18T16:27:37.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:37.644+0000 7fe3c9600640 1 -- 172.21.15.102:0/2769623101 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe3c0033030 con 0x7fe3cc0cc2f0 2024-03-18T16:27:38.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.018+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3cc0cdcc0 con 0x7fe3cc0cc2f0 2024-03-18T16:27:38.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.031+0000 7fe3c9600640 1 -- 172.21.15.102:0/2769623101 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe3c007ba40 con 0x7fe3cc0cc2f0 2024-03-18T16:27:38.033 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:38.033 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:38.033 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:38.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3ac06bc00 msgr2=0x7fe3ac06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:38.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3ac06bc00 0x7fe3ac06e0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0cf020 tx=0x7fe3b8009040 comp rx=0 tx=0).stop 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cc2f0 msgr2=0x7fe3cc0c7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0c7770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe3c002ba60 tx=0x7fe3c0004730 comp rx=0 tx=0).stop 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 shutdown_connections 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3ac06bc00 0x7fe3ac06e0c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3cc0d0250 0x7fe3cc1e08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3cc0cfe70 0x7fe3cc1e03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 --2- 172.21.15.102:0/2769623101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3cc0cc2f0 0x7fe3cc0c7770 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 >> 172.21.15.102:0/2769623101 conn(0x7fe3cc08c3e0 msgr2=0x7fe3cc0ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.033+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 shutdown_connections 2024-03-18T16:27:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:38.034+0000 7fe3d2a00640 1 -- 172.21.15.102:0/2769623101 wait complete. 2024-03-18T16:27:38.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:38.657 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:39.657 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:39.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.839+0000 7f65bda00640 1 Processor -- start 2024-03-18T16:27:39.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.839+0000 7f65bda00640 1 -- start start 2024-03-18T16:27:39.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.841+0000 7f65bda00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f65b80d0250 0x7f65b815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:39.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.841+0000 7f65bda00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65b80cabd0 con 0x7f65b80cc2f0 2024-03-18T16:27:39.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.841+0000 7f65bda00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65b80caea0 con 0x7f65b80cfe70 2024-03-18T16:27:39.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.841+0000 7f65bda00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f65b80cb180 con 0x7f65b80d0250 2024-03-18T16:27:39.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.863+0000 7f65b7400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f65b80d0250 0x7f65b815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:39.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.864+0000 7f65b7400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f65b80d0250 0x7f65b815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38156/0 (socket says 172.21.15.102:38156) 2024-03-18T16:27:39.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.864+0000 7f65b7400640 1 -- 172.21.15.102:0/2189637126 learned_addr learned my addr 172.21.15.102:0/2189637126 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:39.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.867+0000 7f65b7400640 1 -- 172.21.15.102:0/2189637126 >> v1:172.21.15.154:6789/0 conn(0x7f65b80cfe70 legacy=0x7f65b80c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:39.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.867+0000 7f65b7400640 1 -- 172.21.15.102:0/2189637126 >> v1:172.21.15.102:6789/0 conn(0x7f65b80cc2f0 legacy=0x7f65b80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:39.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.867+0000 7f65b7400640 1 -- 172.21.15.102:0/2189637126 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b80cb460 con 0x7f65b80d0250 2024-03-18T16:27:39.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.871+0000 7f65b7400640 1 --2- 172.21.15.102:0/2189637126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80d0250 0x7f65b815ae40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009870 tx=0x7f65a402ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2358d27cab8b77e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:39.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.875+0000 7f65b6a00640 1 -- 172.21.15.102:0/2189637126 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a402f840 con 0x7f65b80d0250 2024-03-18T16:27:39.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.875+0000 7f65b6a00640 1 -- 172.21.15.102:0/2189637126 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a402f9e0 con 0x7f65b80d0250 2024-03-18T16:27:39.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.875+0000 7f65b6a00640 1 -- 172.21.15.102:0/2189637126 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a402fdb0 con 0x7f65b80d0250 2024-03-18T16:27:39.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.876+0000 7f65bda00640 1 -- 172.21.15.102:0/2189637126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80d0250 msgr2=0x7f65b815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:39.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.876+0000 7f65bda00640 1 --2- 172.21.15.102:0/2189637126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80d0250 0x7f65b815ae40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f65a4009870 tx=0x7f65a402ba20 comp rx=0 tx=0).stop 2024-03-18T16:27:39.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.876+0000 7f65bda00640 1 -- 172.21.15.102:0/2189637126 shutdown_connections 2024-03-18T16:27:39.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.876+0000 7f65bda00640 1 --2- 172.21.15.102:0/2189637126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80d0250 0x7f65b815ae40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:39.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.876+0000 7f65bda00640 1 -- 172.21.15.102:0/2189637126 >> 172.21.15.102:0/2189637126 conn(0x7f65b808c3e0 msgr2=0x7f65b806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:39.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.877+0000 7f65bda00640 1 -- 172.21.15.102:0/2189637126 shutdown_connections 2024-03-18T16:27:39.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.877+0000 7f65bda00640 1 -- 172.21.15.102:0/2189637126 wait complete. 2024-03-18T16:27:39.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.879+0000 7f65bda00640 1 Processor -- start 2024-03-18T16:27:39.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.879+0000 7f65bda00640 1 -- start start 2024-03-18T16:27:39.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.881+0000 7f65bda00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80cc2f0 0x7f65b8159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:39.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.881+0000 7f65bda00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80cfe70 0x7f65b81f0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:39.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.881+0000 7f65bda00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65b80d0250 0x7f65b8159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:39.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.881+0000 7f65bda00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f65b80cbd00 con 0x7f65b80cfe70 2024-03-18T16:27:39.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.881+0000 7f65bda00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f65b80cba20 con 0x7f65b80cc2f0 2024-03-18T16:27:39.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.881+0000 7f65bda00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f65b80c8330 con 0x7f65b80d0250 2024-03-18T16:27:39.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.881+0000 7f65b7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80cc2f0 0x7f65b8159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:39.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.882+0000 7f65b7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80cc2f0 0x7f65b8159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38170/0 (socket says 172.21.15.102:38170) 2024-03-18T16:27:39.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.882+0000 7f65b7e00640 1 -- 172.21.15.102:0/1208245875 learned_addr learned my addr 172.21.15.102:0/1208245875 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:39.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.882+0000 7f65bd000640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65b80d0250 0x7f65b8159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:39.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.885+0000 7f65b7e00640 1 -- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65b80d0250 msgr2=0x7f65b8159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:39.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.885+0000 7f65b7e00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65b80d0250 0x7f65b8159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:39.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.885+0000 7f65b7e00640 1 -- 172.21.15.102:0/1208245875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80cfe70 msgr2=0x7f65b81f0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:39.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.885+0000 7f65b7e00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80cfe70 0x7f65b81f0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:39.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.885+0000 7f65b7e00640 1 -- 172.21.15.102:0/1208245875 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b815a340 con 0x7f65b80cc2f0 2024-03-18T16:27:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.888+0000 7f65b7e00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80cc2f0 0x7f65b8159d00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f65a800c450 tx=0x7f65a800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:39.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.892+0000 7f65afe00640 1 -- 172.21.15.102:0/1208245875 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a80052d0 con 0x7f65b80cc2f0 2024-03-18T16:27:39.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.892+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b8155e90 con 0x7f65b80cc2f0 2024-03-18T16:27:39.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.892+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b81584b0 con 0x7f65b80cc2f0 2024-03-18T16:27:39.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.894+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b8074fd0 con 0x7f65b80cc2f0 2024-03-18T16:27:39.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.895+0000 7f65afe00640 1 -- 172.21.15.102:0/1208245875 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a8005470 con 0x7f65b80cc2f0 2024-03-18T16:27:39.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:39.895+0000 7f65afe00640 1 -- 172.21.15.102:0/1208245875 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a8007250 con 0x7f65b80cc2f0 2024-03-18T16:27:40.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.163+0000 7f65afe00640 1 -- 172.21.15.102:0/1208245875 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f65a80073f0 con 0x7f65b80cc2f0 2024-03-18T16:27:40.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.165+0000 7f65afe00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f659806bdd0 0x7f659806e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:40.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.165+0000 7f65afe00640 1 -- 172.21.15.102:0/1208245875 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f65a8007030 con 0x7f65b80cc2f0 2024-03-18T16:27:40.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.165+0000 7f65b7400640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f659806bdd0 0x7f659806e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:40.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.166+0000 7f65b7400640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f659806bdd0 0x7f659806e290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f65a4036000 tx=0x7f65a4039040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:40.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.176+0000 7f65afe00640 1 -- 172.21.15.102:0/1208245875 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f65a80599e0 con 0x7f65b80cc2f0 2024-03-18T16:27:40.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.442+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65b80c6ab0 con 0x7f65b80cc2f0 2024-03-18T16:27:40.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.454+0000 7f65afe00640 1 -- 172.21.15.102:0/1208245875 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f65a80592e0 con 0x7f65b80cc2f0 2024-03-18T16:27:40.456 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:40.456 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:40.457 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f659806bdd0 msgr2=0x7f659806e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f659806bdd0 0x7f659806e290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f65a4036000 tx=0x7f65a4039040 comp rx=0 tx=0).stop 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80cc2f0 msgr2=0x7f65b8159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80cc2f0 0x7f65b8159d00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f65a800c450 tx=0x7f65a800c910 comp rx=0 tx=0).stop 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 shutdown_connections 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f659806bdd0 0x7f659806e290 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f65b80d0250 0x7f65b8159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65b80cfe70 0x7f65b81f0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 --2- 172.21.15.102:0/1208245875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65b80cc2f0 0x7f65b8159d00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 >> 172.21.15.102:0/1208245875 conn(0x7f65b808c3e0 msgr2=0x7f65b806c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 shutdown_connections 2024-03-18T16:27:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:40.457+0000 7f65bda00640 1 -- 172.21.15.102:0/1208245875 wait complete. 2024-03-18T16:27:41.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:41.079 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:42.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.249+0000 7f8b2d800640 1 Processor -- start 2024-03-18T16:27:42.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.250+0000 7f8b2d800640 1 -- start start 2024-03-18T16:27:42.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.251+0000 7f8b2d800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8b280c3890 0x7f8b280c8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.251+0000 7f8b2d800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b280c3cd0 0x7f8b280c9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.251+0000 7f8b2d800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2815a170 con 0x7f8b28154520 2024-03-18T16:27:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.251+0000 7f8b2d800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8b28154900 con 0x7f8b280c3890 2024-03-18T16:27:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.251+0000 7f8b2d800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8b280c4160 con 0x7f8b280c3cd0 2024-03-18T16:27:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.252+0000 7f8b2ce00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b280c3cd0 0x7f8b280c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:42.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.252+0000 7f8b27400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8b280c3890 0x7f8b280c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:42.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.252+0000 7f8b2ce00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b280c3cd0 0x7f8b280c9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56964/0 (socket says 172.21.15.102:56964) 2024-03-18T16:27:42.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.252+0000 7f8b2ce00640 1 -- 172.21.15.102:0/2155970194 learned_addr learned my addr 172.21.15.102:0/2155970194 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:42.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.256+0000 7f8b2ce00640 1 -- 172.21.15.102:0/2155970194 >> v2:172.21.15.102:3300/0 conn(0x7f8b280c3890 msgr2=0x7f8b280c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:42.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.256+0000 7f8b2ce00640 1 --2- 172.21.15.102:0/2155970194 >> v2:172.21.15.102:3300/0 conn(0x7f8b280c3890 0x7f8b280c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.256+0000 7f8b2ce00640 1 -- 172.21.15.102:0/2155970194 >> v1:172.21.15.102:6789/0 conn(0x7f8b28154520 legacy=0x7f8b280c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:42.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.256+0000 7f8b2ce00640 1 -- 172.21.15.102:0/2155970194 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b280c4440 con 0x7f8b280c3cd0 2024-03-18T16:27:42.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.259+0000 7f8b2ce00640 1 --2- 172.21.15.102:0/2155970194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b280c3cd0 0x7f8b280c9370 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8b280c37b0 tx=0x7f8b2002bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e8e563ccb46d7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.264+0000 7f8b26a00640 1 -- 172.21.15.102:0/2155970194 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2002cb10 con 0x7f8b280c3cd0 2024-03-18T16:27:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.264+0000 7f8b26a00640 1 -- 172.21.15.102:0/2155970194 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b2002ccb0 con 0x7f8b280c3cd0 2024-03-18T16:27:42.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.264+0000 7f8b26a00640 1 -- 172.21.15.102:0/2155970194 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2002ce60 con 0x7f8b280c3cd0 2024-03-18T16:27:42.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.265+0000 7f8b2d800640 1 -- 172.21.15.102:0/2155970194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b280c3cd0 msgr2=0x7f8b280c9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:42.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.265+0000 7f8b2d800640 1 --2- 172.21.15.102:0/2155970194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b280c3cd0 0x7f8b280c9370 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8b280c37b0 tx=0x7f8b2002bf70 comp rx=0 tx=0).stop 2024-03-18T16:27:42.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.265+0000 7f8b2d800640 1 -- 172.21.15.102:0/2155970194 shutdown_connections 2024-03-18T16:27:42.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.265+0000 7f8b2d800640 1 --2- 172.21.15.102:0/2155970194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b280c3cd0 0x7f8b280c9370 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.265+0000 7f8b2d800640 1 --2- 172.21.15.102:0/2155970194 >> v2:172.21.15.102:3300/0 conn(0x7f8b280c3890 0x7f8b280c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.265+0000 7f8b2d800640 1 -- 172.21.15.102:0/2155970194 >> 172.21.15.102:0/2155970194 conn(0x7f8b28066f80 msgr2=0x7f8b28067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:42.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.265+0000 7f8b2d800640 1 -- 172.21.15.102:0/2155970194 shutdown_connections 2024-03-18T16:27:42.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.265+0000 7f8b2d800640 1 -- 172.21.15.102:0/2155970194 wait complete. 2024-03-18T16:27:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.268+0000 7f8b2d800640 1 Processor -- start 2024-03-18T16:27:42.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.268+0000 7f8b2d800640 1 -- start start 2024-03-18T16:27:42.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.269+0000 7f8b2d800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b280c3890 0x7f8b281f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:42.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.269+0000 7f8b2d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b28154520 0x7f8b281426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:42.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.269+0000 7f8b2d800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b281539d0 0x7f8b28142c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:42.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.269+0000 7f8b2d800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2815a9f0 con 0x7f8b28154520 2024-03-18T16:27:42.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.269+0000 7f8b2d800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2815a710 con 0x7f8b281539d0 2024-03-18T16:27:42.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.269+0000 7f8b2d800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8b280c8e60 con 0x7f8b280c3890 2024-03-18T16:27:42.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.270+0000 7f8b27e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b280c3890 0x7f8b281f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:42.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.270+0000 7f8b2ce00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b281539d0 0x7f8b28142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:42.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.271+0000 7f8b27e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b280c3890 0x7f8b281f17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:35756/0 (socket says 172.21.15.102:35756) 2024-03-18T16:27:42.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.271+0000 7f8b27e00640 1 -- 172.21.15.102:0/3369504174 learned_addr learned my addr 172.21.15.102:0/3369504174 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:42.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.271+0000 7f8b27400640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b28154520 0x7f8b281426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:42.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.274+0000 7f8b27e00640 1 -- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b281539d0 msgr2=0x7f8b28142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:42.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.274+0000 7f8b27e00640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b281539d0 0x7f8b28142c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.274+0000 7f8b27e00640 1 -- 172.21.15.102:0/3369504174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b28154520 msgr2=0x7f8b281426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:42.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.274+0000 7f8b27e00640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b28154520 0x7f8b281426d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.274+0000 7f8b27e00640 1 -- 172.21.15.102:0/3369504174 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b280c5f10 con 0x7f8b280c3890 2024-03-18T16:27:42.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.277+0000 7f8b27e00640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b280c3890 0x7f8b281f17d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800c3f0 tx=0x7f8b1800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.280+0000 7f8b1fe00640 1 -- 172.21.15.102:0/3369504174 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b180052d0 con 0x7f8b280c3890 2024-03-18T16:27:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.280+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b280c66e0 con 0x7f8b280c3890 2024-03-18T16:27:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.280+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b280c6ca0 con 0x7f8b280c3890 2024-03-18T16:27:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.280+0000 7f8b1fe00640 1 -- 172.21.15.102:0/3369504174 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b18005470 con 0x7f8b280c3890 2024-03-18T16:27:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.280+0000 7f8b1fe00640 1 -- 172.21.15.102:0/3369504174 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18007220 con 0x7f8b280c3890 2024-03-18T16:27:42.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.281+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b00005290 con 0x7f8b280c3890 2024-03-18T16:27:42.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.538+0000 7f8b1fe00640 1 -- 172.21.15.102:0/3369504174 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8b180073c0 con 0x7f8b280c3890 2024-03-18T16:27:42.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.540+0000 7f8b1fe00640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b0806bbd0 0x7f8b0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:42.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.540+0000 7f8b27400640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b0806bbd0 0x7f8b0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:42.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.541+0000 7f8b1fe00640 1 -- 172.21.15.102:0/3369504174 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8b18056b20 con 0x7f8b280c3890 2024-03-18T16:27:42.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.541+0000 7f8b27400640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b0806bbd0 0x7f8b0806e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8b281eaca0 tx=0x7f8b14009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:42.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.551+0000 7f8b1fe00640 1 -- 172.21.15.102:0/3369504174 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8b18020070 con 0x7f8b280c3890 2024-03-18T16:27:42.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.783+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b00005580 con 0x7f8b280c3890 2024-03-18T16:27:42.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.796+0000 7f8b1fe00640 1 -- 172.21.15.102:0/3369504174 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8b280c5f10 con 0x7f8b280c3890 2024-03-18T16:27:42.799 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:42.800 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:42.800 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b0806bbd0 msgr2=0x7f8b0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b0806bbd0 0x7f8b0806e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8b281eaca0 tx=0x7f8b14009190 comp rx=0 tx=0).stop 2024-03-18T16:27:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b280c3890 msgr2=0x7f8b281f17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b280c3890 0x7f8b281f17d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800c3f0 tx=0x7f8b1800c8b0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 shutdown_connections 2024-03-18T16:27:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b0806bbd0 0x7f8b0806e090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b281539d0 0x7f8b28142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b28154520 0x7f8b281426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 --2- 172.21.15.102:0/3369504174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b280c3890 0x7f8b281f17d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 >> 172.21.15.102:0/3369504174 conn(0x7f8b28066f80 msgr2=0x7f8b28149390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 shutdown_connections 2024-03-18T16:27:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:42.799+0000 7f8b2d800640 1 -- 172.21.15.102:0/3369504174 wait complete. 2024-03-18T16:27:43.421 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:43.421 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:44.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:44.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.591+0000 7f39bf200640 1 Processor -- start 2024-03-18T16:27:44.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.591+0000 7f39bf200640 1 -- start start 2024-03-18T16:27:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.592+0000 7f39bf200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c00ca960 0x7f39c006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.593+0000 7f39bf200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f39c015a1a0 0x7f39c00705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.593+0000 7f39bf200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39c006c9c0 con 0x7f39c00cad40 2024-03-18T16:27:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.593+0000 7f39bf200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f39c00c8680 con 0x7f39c015a1a0 2024-03-18T16:27:44.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.593+0000 7f39bf200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f39c006cca0 con 0x7f39c00ca960 2024-03-18T16:27:44.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.594+0000 7f39bde00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c00ca960 0x7f39c006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:44.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.594+0000 7f39be800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f39c015a1a0 0x7f39c00705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:44.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.594+0000 7f39bd400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f39c00cad40 0x7f39c015e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:53014/0 (socket says 172.21.15.102:53014) 2024-03-18T16:27:44.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.594+0000 7f39bd400640 1 -- 172.21.15.102:0/3688030550 learned_addr learned my addr 172.21.15.102:0/3688030550 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:44.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.596+0000 7f39bde00640 1 -- 172.21.15.102:0/3688030550 >> v2:172.21.15.154:3300/0 conn(0x7f39c015a1a0 msgr2=0x7f39c00705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:44.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.596+0000 7f39bde00640 1 --2- 172.21.15.102:0/3688030550 >> v2:172.21.15.154:3300/0 conn(0x7f39c015a1a0 0x7f39c00705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:44.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.597+0000 7f39bde00640 1 -- 172.21.15.102:0/3688030550 >> v1:172.21.15.154:6789/0 conn(0x7f39c00cad40 legacy=0x7f39c015e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:44.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.597+0000 7f39bde00640 1 -- 172.21.15.102:0/3688030550 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c006cf80 con 0x7f39c00ca960 2024-03-18T16:27:44.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.599+0000 7f39bde00640 1 --2- 172.21.15.102:0/3688030550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c00ca960 0x7f39c006c310 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f39b00097a0 tx=0x7f39b002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccc0f2206bb00169 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:44.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.603+0000 7f39bca00640 1 -- 172.21.15.102:0/3688030550 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0034070 con 0x7f39c00ca960 2024-03-18T16:27:44.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.603+0000 7f39bca00640 1 -- 172.21.15.102:0/3688030550 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b0030610 con 0x7f39c00ca960 2024-03-18T16:27:44.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.603+0000 7f39bca00640 1 -- 172.21.15.102:0/3688030550 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b00309e0 con 0x7f39c00ca960 2024-03-18T16:27:44.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.604+0000 7f39bf200640 1 -- 172.21.15.102:0/3688030550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c00ca960 msgr2=0x7f39c006c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:44.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.604+0000 7f39bf200640 1 --2- 172.21.15.102:0/3688030550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c00ca960 0x7f39c006c310 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f39b00097a0 tx=0x7f39b002b960 comp rx=0 tx=0).stop 2024-03-18T16:27:44.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.604+0000 7f39bf200640 1 -- 172.21.15.102:0/3688030550 shutdown_connections 2024-03-18T16:27:44.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.604+0000 7f39bf200640 1 --2- 172.21.15.102:0/3688030550 >> v2:172.21.15.154:3300/0 conn(0x7f39c015a1a0 0x7f39c00705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:44.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.604+0000 7f39bf200640 1 --2- 172.21.15.102:0/3688030550 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c00ca960 0x7f39c006c310 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:44.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.604+0000 7f39bf200640 1 -- 172.21.15.102:0/3688030550 >> 172.21.15.102:0/3688030550 conn(0x7f39c0067190 msgr2=0x7f39c00675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:44.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.604+0000 7f39bf200640 1 -- 172.21.15.102:0/3688030550 shutdown_connections 2024-03-18T16:27:44.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.605+0000 7f39bf200640 1 -- 172.21.15.102:0/3688030550 wait complete. 2024-03-18T16:27:44.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.606+0000 7f39bf200640 1 Processor -- start 2024-03-18T16:27:44.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.606+0000 7f39bf200640 1 -- start start 2024-03-18T16:27:44.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.607+0000 7f39bf200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c00ca960 0x7f39c00c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:44.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.607+0000 7f39bf200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c00cad40 0x7f39c0141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:44.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.607+0000 7f39bf200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c015a1a0 0x7f39c0156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:44.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.607+0000 7f39bf200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f39c0070ae0 con 0x7f39c00cad40 2024-03-18T16:27:44.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.607+0000 7f39bf200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39c015dc10 con 0x7f39c00ca960 2024-03-18T16:27:44.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.607+0000 7f39bf200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f39c0070c60 con 0x7f39c015a1a0 2024-03-18T16:27:44.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.608+0000 7f39bde00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c00ca960 0x7f39c00c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:44.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.608+0000 7f39be800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c015a1a0 0x7f39c0156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:44.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.608+0000 7f39bd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c00cad40 0x7f39c0141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:44.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.608+0000 7f39bde00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c00ca960 0x7f39c00c2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56994/0 (socket says 172.21.15.102:56994) 2024-03-18T16:27:44.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.608+0000 7f39bde00640 1 -- 172.21.15.102:0/2183736005 learned_addr learned my addr 172.21.15.102:0/2183736005 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.612+0000 7f39be800640 1 -- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c00ca960 msgr2=0x7f39c00c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.612+0000 7f39be800640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c00ca960 0x7f39c00c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.612+0000 7f39be800640 1 -- 172.21.15.102:0/2183736005 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c00cad40 msgr2=0x7f39c0141ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.612+0000 7f39be800640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c00cad40 0x7f39c0141ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:44.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.612+0000 7f39be800640 1 -- 172.21.15.102:0/2183736005 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c01560f0 con 0x7f39c015a1a0 2024-03-18T16:27:44.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.615+0000 7f39be800640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c015a1a0 0x7f39c0156e70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f39b800eba0 tx=0x7f39b800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:44.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.619+0000 7f39b6000640 1 -- 172.21.15.102:0/2183736005 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8018020 con 0x7f39c015a1a0 2024-03-18T16:27:44.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.619+0000 7f39b6000640 1 -- 172.21.15.102:0/2183736005 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b800cd20 con 0x7f39c015a1a0 2024-03-18T16:27:44.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.619+0000 7f39b6000640 1 -- 172.21.15.102:0/2183736005 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b80146c0 con 0x7f39c015a1a0 2024-03-18T16:27:44.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.619+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c01f0580 con 0x7f39c015a1a0 2024-03-18T16:27:44.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.619+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c0156670 con 0x7f39c015a1a0 2024-03-18T16:27:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.621+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c0074fd0 con 0x7f39c015a1a0 2024-03-18T16:27:44.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.640+0000 7f39b6000640 1 -- 172.21.15.102:0/2183736005 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f39b8005d10 con 0x7f39c015a1a0 2024-03-18T16:27:44.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.641+0000 7f39b6000640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39a006bc50 0x7f39a006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:44.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.642+0000 7f39bde00640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39a006bc50 0x7f39a006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:44.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.643+0000 7f39bde00640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39a006bc50 0x7f39a006e110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009560 tx=0x7f39b003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:44.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.644+0000 7f39b6000640 1 -- 172.21.15.102:0/2183736005 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f39b8058050 con 0x7f39c015a1a0 2024-03-18T16:27:44.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:44.897+0000 7f39b6000640 1 -- 172.21.15.102:0/2183736005 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f39b8022030 con 0x7f39c015a1a0 2024-03-18T16:27:45.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.210+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39c0067940 con 0x7f39c015a1a0 2024-03-18T16:27:45.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.251+0000 7f39b6000640 1 -- 172.21.15.102:0/2183736005 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f39b8010040 con 0x7f39c015a1a0 2024-03-18T16:27:45.255 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:45.255 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:45.255 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39a006bc50 msgr2=0x7f39a006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39a006bc50 0x7f39a006e110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009560 tx=0x7f39b003b040 comp rx=0 tx=0).stop 2024-03-18T16:27:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c015a1a0 msgr2=0x7f39c0156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c015a1a0 0x7f39c0156e70 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f39b800eba0 tx=0x7f39b800c490 comp rx=0 tx=0).stop 2024-03-18T16:27:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 shutdown_connections 2024-03-18T16:27:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f39a006bc50 0x7f39a006e110 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f39c015a1a0 0x7f39c0156e70 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39c00cad40 0x7f39c0141ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 --2- 172.21.15.102:0/2183736005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39c00ca960 0x7f39c00c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 >> 172.21.15.102:0/2183736005 conn(0x7f39c0067190 msgr2=0x7f39c015d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 shutdown_connections 2024-03-18T16:27:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:45.255+0000 7f39bf200640 1 -- 172.21.15.102:0/2183736005 wait complete. 2024-03-18T16:27:45.878 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:45.878 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:46.879 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:47.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.056+0000 7f2dad000640 1 Processor -- start 2024-03-18T16:27:47.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.056+0000 7f2dad000640 1 -- start start 2024-03-18T16:27:47.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.058+0000 7f2dad000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2da815a6f0 0x7f2da815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:47.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.058+0000 7f2dad000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 0x7f2da815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:47.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.058+0000 7f2dad000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da815c470 con 0x7f2da815a310 2024-03-18T16:27:47.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.058+0000 7f2dad000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2da80cd410 con 0x7f2da815a6f0 2024-03-18T16:27:47.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.058+0000 7f2dad000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2da80ca500 con 0x7f2da815baa0 2024-03-18T16:27:47.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.058+0000 7f2da7400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f2da815a310 0x7f2da806d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:53024/0 (socket says 172.21.15.102:53024) 2024-03-18T16:27:47.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.058+0000 7f2da7400640 1 -- 172.21.15.102:0/3211449635 learned_addr learned my addr 172.21.15.102:0/3211449635 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:47.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.058+0000 7f2da7e00640 1 --2- 172.21.15.102:0/3211449635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 0x7f2da815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:47.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.058+0000 7f2da6a00640 1 --2- 172.21.15.102:0/3211449635 >> v2:172.21.15.102:3300/0 conn(0x7f2da815a6f0 0x7f2da815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:47.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.061+0000 7f2da6000640 1 -- 172.21.15.102:0/3211449635 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3772542648 0 0) 0x7f2da815c470 con 0x7f2da815a310 2024-03-18T16:27:47.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.061+0000 7f2da6000640 1 -- 172.21.15.102:0/3211449635 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d8c003520 con 0x7f2da815a310 2024-03-18T16:27:47.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.061+0000 7f2da7e00640 1 -- 172.21.15.102:0/3211449635 >> v2:172.21.15.102:3300/0 conn(0x7f2da815a6f0 msgr2=0x7f2da815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:47.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.061+0000 7f2da7e00640 1 --2- 172.21.15.102:0/3211449635 >> v2:172.21.15.102:3300/0 conn(0x7f2da815a6f0 0x7f2da815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.061+0000 7f2da7e00640 1 -- 172.21.15.102:0/3211449635 >> v1:172.21.15.154:6789/0 conn(0x7f2da815a310 legacy=0x7f2da806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:47.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.062+0000 7f2da7e00640 1 -- 172.21.15.102:0/3211449635 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da815b080 con 0x7f2da815baa0 2024-03-18T16:27:47.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.064+0000 7f2da7e00640 1 --2- 172.21.15.102:0/3211449635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 0x7f2da815bf30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009d90 tx=0x7f2da002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=662f08cf1b1c29e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:47.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.068+0000 7f2da6000640 1 -- 172.21.15.102:0/3211449635 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0039070 con 0x7f2da815baa0 2024-03-18T16:27:47.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.068+0000 7f2da6000640 1 -- 172.21.15.102:0/3211449635 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da002cc60 con 0x7f2da815baa0 2024-03-18T16:27:47.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.068+0000 7f2da6000640 1 -- 172.21.15.102:0/3211449635 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0034660 con 0x7f2da815baa0 2024-03-18T16:27:47.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.069+0000 7f2dad000640 1 -- 172.21.15.102:0/3211449635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 msgr2=0x7f2da815bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:47.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.069+0000 7f2dad000640 1 --2- 172.21.15.102:0/3211449635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 0x7f2da815bf30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009d90 tx=0x7f2da002beb0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.070+0000 7f2dad000640 1 -- 172.21.15.102:0/3211449635 shutdown_connections 2024-03-18T16:27:47.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.070+0000 7f2dad000640 1 --2- 172.21.15.102:0/3211449635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 0x7f2da815bf30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.070+0000 7f2dad000640 1 --2- 172.21.15.102:0/3211449635 >> v2:172.21.15.102:3300/0 conn(0x7f2da815a6f0 0x7f2da815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.070+0000 7f2dad000640 1 -- 172.21.15.102:0/3211449635 >> 172.21.15.102:0/3211449635 conn(0x7f2da808c1a0 msgr2=0x7f2da8066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:47.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.070+0000 7f2dad000640 1 -- 172.21.15.102:0/3211449635 shutdown_connections 2024-03-18T16:27:47.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.070+0000 7f2dad000640 1 -- 172.21.15.102:0/3211449635 wait complete. 2024-03-18T16:27:47.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.071+0000 7f2dad000640 1 Processor -- start 2024-03-18T16:27:47.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.072+0000 7f2dad000640 1 -- start start 2024-03-18T16:27:47.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.073+0000 7f2dad000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da815a310 0x7f2da81f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:47.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.073+0000 7f2dad000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da815a6f0 0x7f2da815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:47.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.073+0000 7f2dad000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 0x7f2da815c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:47.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.073+0000 7f2dad000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2da80cce50 con 0x7f2da815a6f0 2024-03-18T16:27:47.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.073+0000 7f2dad000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2da80caa80 con 0x7f2da815a310 2024-03-18T16:27:47.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.073+0000 7f2dad000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2da80cd130 con 0x7f2da815baa0 2024-03-18T16:27:47.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.074+0000 7f2da7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da815a310 0x7f2da81f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:47.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.074+0000 7f2da7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 0x7f2da815c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:47.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.074+0000 7f2da6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da815a6f0 0x7f2da815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:47.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.074+0000 7f2da7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da815a310 0x7f2da81f0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57002/0 (socket says 172.21.15.102:57002) 2024-03-18T16:27:47.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.074+0000 7f2da7400640 1 -- 172.21.15.102:0/4033102284 learned_addr learned my addr 172.21.15.102:0/4033102284 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:47.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.078+0000 7f2da7400640 1 -- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 msgr2=0x7f2da815c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:47.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.078+0000 7f2da7400640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 0x7f2da815c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.078+0000 7f2da7400640 1 -- 172.21.15.102:0/4033102284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da815a6f0 msgr2=0x7f2da815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:47.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.078+0000 7f2da7400640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da815a6f0 0x7f2da815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.078+0000 7f2da7400640 1 -- 172.21.15.102:0/4033102284 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da8150030 con 0x7f2da815a310 2024-03-18T16:27:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.080+0000 7f2da7400640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da815a310 0x7f2da81f0580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800c3c0 tx=0x7f2d9800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:47.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.082+0000 7f2d9f400640 1 -- 172.21.15.102:0/4033102284 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98017070 con 0x7f2da815a310 2024-03-18T16:27:47.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.082+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da81f0bc0 con 0x7f2da815a310 2024-03-18T16:27:47.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.082+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da81f1180 con 0x7f2da815a310 2024-03-18T16:27:47.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.083+0000 7f2d9f400640 1 -- 172.21.15.102:0/4033102284 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d98005ce0 con 0x7f2da815a310 2024-03-18T16:27:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.084+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d80005290 con 0x7f2da815a310 2024-03-18T16:27:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.084+0000 7f2d9f400640 1 -- 172.21.15.102:0/4033102284 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98021060 con 0x7f2da815a310 2024-03-18T16:27:47.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.109+0000 7f2d9f400640 1 -- 172.21.15.102:0/4033102284 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2d9801a070 con 0x7f2da815a310 2024-03-18T16:27:47.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.110+0000 7f2d9f400640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2d8806bc00 0x7f2d8806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:47.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.110+0000 7f2da6a00640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2d8806bc00 0x7f2d8806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:47.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.111+0000 7f2da6a00640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2d8806bc00 0x7f2d8806e0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2da815ce40 tx=0x7f2d94009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:47.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.112+0000 7f2d9f400640 1 -- 172.21.15.102:0/4033102284 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f2d98056a80 con 0x7f2da815a310 2024-03-18T16:27:47.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.407+0000 7f2d9f400640 1 -- 172.21.15.102:0/4033102284 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2d980589e0 con 0x7f2da815a310 2024-03-18T16:27:47.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.643+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d80005580 con 0x7f2da815a310 2024-03-18T16:27:47.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.700+0000 7f2d9f400640 1 -- 172.21.15.102:0/4033102284 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f2d9805ca80 con 0x7f2da815a310 2024-03-18T16:27:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:47.703 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:47.703 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:47.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2d8806bc00 msgr2=0x7f2d8806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:47.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2d8806bc00 0x7f2d8806e0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2da815ce40 tx=0x7f2d94009190 comp rx=0 tx=0).stop 2024-03-18T16:27:47.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da815a310 msgr2=0x7f2da81f0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:47.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da815a310 0x7f2da81f0580 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800c3c0 tx=0x7f2d9800c880 comp rx=0 tx=0).stop 2024-03-18T16:27:47.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 shutdown_connections 2024-03-18T16:27:47.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2d8806bc00 0x7f2d8806e0c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2da815baa0 0x7f2da815c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2da815a6f0 0x7f2da815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 --2- 172.21.15.102:0/4033102284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2da815a310 0x7f2da81f0580 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:47.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 >> 172.21.15.102:0/4033102284 conn(0x7f2da808c1a0 msgr2=0x7f2da814ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:47.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 shutdown_connections 2024-03-18T16:27:47.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:47.703+0000 7f2dad000640 1 -- 172.21.15.102:0/4033102284 wait complete. 2024-03-18T16:27:48.333 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:48.333 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:49.334 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:49.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.506+0000 7f854f200640 1 Processor -- start 2024-03-18T16:27:49.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.506+0000 7f854f200640 1 -- start start 2024-03-18T16:27:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.507+0000 7f854f200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8550158420 0x7f855015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.507+0000 7f854f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85500c4fa0 0x7f85500c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.507+0000 7f854f200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85500c72d0 con 0x7f8550141650 2024-03-18T16:27:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.507+0000 7f854f200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f855015c8e0 con 0x7f8550158420 2024-03-18T16:27:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.507+0000 7f854f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8550159370 con 0x7f85500c4fa0 2024-03-18T16:27:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.508+0000 7f854d400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8550158420 0x7f855015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.508+0000 7f854de00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8550141650 0x7f85500c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:53032/0 (socket says 172.21.15.102:53032) 2024-03-18T16:27:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.508+0000 7f854de00640 1 -- 172.21.15.102:0/535016430 learned_addr learned my addr 172.21.15.102:0/535016430 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.509+0000 7f854e800640 1 --2- 172.21.15.102:0/535016430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85500c4fa0 0x7f85500c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:49.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.511+0000 7f854e800640 1 -- 172.21.15.102:0/535016430 >> v2:172.21.15.154:3300/0 conn(0x7f8550158420 msgr2=0x7f855015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:49.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.511+0000 7f854e800640 1 --2- 172.21.15.102:0/535016430 >> v2:172.21.15.154:3300/0 conn(0x7f8550158420 0x7f855015c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:49.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.511+0000 7f854e800640 1 -- 172.21.15.102:0/535016430 >> v1:172.21.15.154:6789/0 conn(0x7f8550141650 legacy=0x7f85500c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:49.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.511+0000 7f854e800640 1 -- 172.21.15.102:0/535016430 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8550158840 con 0x7f85500c4fa0 2024-03-18T16:27:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.513+0000 7f854e800640 1 --2- 172.21.15.102:0/535016430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85500c4fa0 0x7f85500c5430 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8548009d90 tx=0x7f854802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=519f9ea1c6521b27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.517+0000 7f854ca00640 1 -- 172.21.15.102:0/535016430 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548039070 con 0x7f85500c4fa0 2024-03-18T16:27:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.517+0000 7f854ca00640 1 -- 172.21.15.102:0/535016430 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f854802cc50 con 0x7f85500c4fa0 2024-03-18T16:27:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.517+0000 7f854ca00640 1 -- 172.21.15.102:0/535016430 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548034440 con 0x7f85500c4fa0 2024-03-18T16:27:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.518+0000 7f854f200640 1 -- 172.21.15.102:0/535016430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85500c4fa0 msgr2=0x7f85500c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.518+0000 7f854f200640 1 --2- 172.21.15.102:0/535016430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85500c4fa0 0x7f85500c5430 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8548009d90 tx=0x7f854802bea0 comp rx=0 tx=0).stop 2024-03-18T16:27:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.518+0000 7f854f200640 1 -- 172.21.15.102:0/535016430 shutdown_connections 2024-03-18T16:27:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.518+0000 7f854f200640 1 --2- 172.21.15.102:0/535016430 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f85500c4fa0 0x7f85500c5430 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.518+0000 7f854f200640 1 --2- 172.21.15.102:0/535016430 >> v2:172.21.15.154:3300/0 conn(0x7f8550158420 0x7f855015c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.518+0000 7f854f200640 1 -- 172.21.15.102:0/535016430 >> 172.21.15.102:0/535016430 conn(0x7f85500cb890 msgr2=0x7f85500cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.518+0000 7f854f200640 1 -- 172.21.15.102:0/535016430 shutdown_connections 2024-03-18T16:27:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.518+0000 7f854f200640 1 -- 172.21.15.102:0/535016430 wait complete. 2024-03-18T16:27:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.521+0000 7f854f200640 1 Processor -- start 2024-03-18T16:27:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.521+0000 7f854f200640 1 -- start start 2024-03-18T16:27:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.522+0000 7f854f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85500c4fa0 0x7f8550157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.522+0000 7f854f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8550141650 0x7f855013cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.522+0000 7f854f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8550158420 0x7f85501f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.522+0000 7f854f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8550159090 con 0x7f85500c4fa0 2024-03-18T16:27:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.522+0000 7f854f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8550158db0 con 0x7f8550141650 2024-03-18T16:27:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.522+0000 7f854f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85500c63c0 con 0x7f8550158420 2024-03-18T16:27:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.523+0000 7f854d400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8550141650 0x7f855013cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.523+0000 7f854e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8550158420 0x7f85501f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.523+0000 7f854d400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8550141650 0x7f855013cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57026/0 (socket says 172.21.15.102:57026) 2024-03-18T16:27:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.523+0000 7f854d400640 1 -- 172.21.15.102:0/3084118327 learned_addr learned my addr 172.21.15.102:0/3084118327 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.523+0000 7f854de00640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85500c4fa0 0x7f8550157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.527+0000 7f854d400640 1 -- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8550158420 msgr2=0x7f85501f3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.527+0000 7f854d400640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8550158420 0x7f85501f3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.527+0000 7f854d400640 1 -- 172.21.15.102:0/3084118327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85500c4fa0 msgr2=0x7f8550157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.527+0000 7f854d400640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85500c4fa0 0x7f8550157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.527+0000 7f854d400640 1 -- 172.21.15.102:0/3084118327 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855013c670 con 0x7f8550141650 2024-03-18T16:27:49.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.530+0000 7f854d400640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8550141650 0x7f855013cca0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f853c00e870 tx=0x7f853c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:49.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.535+0000 7f8546000640 1 -- 172.21.15.102:0/3084118327 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c017070 con 0x7f8550141650 2024-03-18T16:27:49.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.535+0000 7f8546000640 1 -- 172.21.15.102:0/3084118327 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853c00c940 con 0x7f8550141650 2024-03-18T16:27:49.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.535+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8550136420 con 0x7f8550141650 2024-03-18T16:27:49.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.535+0000 7f8546000640 1 -- 172.21.15.102:0/3084118327 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c00cd30 con 0x7f8550141650 2024-03-18T16:27:49.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.535+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f855015a720 con 0x7f8550141650 2024-03-18T16:27:49.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.536+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85500c74a0 con 0x7f8550141650 2024-03-18T16:27:49.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.571+0000 7f8546000640 1 -- 172.21.15.102:0/3084118327 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f853c01a070 con 0x7f8550141650 2024-03-18T16:27:49.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.573+0000 7f8546000640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f853006bc50 0x7f853006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:49.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.574+0000 7f854de00640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f853006bc50 0x7f853006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:49.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.574+0000 7f8546000640 1 -- 172.21.15.102:0/3084118327 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f853c057550 con 0x7f8550141650 2024-03-18T16:27:49.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.574+0000 7f854de00640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f853006bc50 0x7f853006e110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8540007fd0 tx=0x7f8540007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:49.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:49.824+0000 7f8546000640 1 -- 172.21.15.102:0/3084118327 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f85500c74a0 con 0x7f8550141650 2024-03-18T16:27:50.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.069+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8550151f70 con 0x7f8550141650 2024-03-18T16:27:50.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.097+0000 7f8546000640 1 -- 172.21.15.102:0/3084118327 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8550151f70 con 0x7f8550141650 2024-03-18T16:27:50.100 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:50.100 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:50.100 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:50.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f853006bc50 msgr2=0x7f853006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:50.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f853006bc50 0x7f853006e110 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8540007fd0 tx=0x7f8540007e80 comp rx=0 tx=0).stop 2024-03-18T16:27:50.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8550141650 msgr2=0x7f855013cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:50.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8550141650 0x7f855013cca0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f853c00e870 tx=0x7f853c00ed30 comp rx=0 tx=0).stop 2024-03-18T16:27:50.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 shutdown_connections 2024-03-18T16:27:50.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f853006bc50 0x7f853006e110 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:50.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8550158420 0x7f85501f3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:50.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8550141650 0x7f855013cca0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:50.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 --2- 172.21.15.102:0/3084118327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85500c4fa0 0x7f8550157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:50.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 >> 172.21.15.102:0/3084118327 conn(0x7f85500cb890 msgr2=0x7f855013edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:50.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 shutdown_connections 2024-03-18T16:27:50.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:50.100+0000 7f854f200640 1 -- 172.21.15.102:0/3084118327 wait complete. 2024-03-18T16:27:50.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:50.724 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:51.725 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:51.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.903+0000 7fd2d7e00640 1 Processor -- start 2024-03-18T16:27:51.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.903+0000 7fd2d7e00640 1 -- start start 2024-03-18T16:27:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.904+0000 7fd2d7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a310 0x7fd2d806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.904+0000 7fd2d7e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd2d815a6f0 0x7fd2d815ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.904+0000 7fd2d7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd2d815baa0 0x7fd2d815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.904+0000 7fd2d7e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d80cd410 con 0x7fd2d815baa0 2024-03-18T16:27:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.904+0000 7fd2d7e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d815c470 con 0x7fd2d815a6f0 2024-03-18T16:27:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.904+0000 7fd2d7e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d80ca500 con 0x7fd2d815a310 2024-03-18T16:27:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.905+0000 7fd2d6000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd2d815a6f0 0x7fd2d815ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.905+0000 7fd2d6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a310 0x7fd2d806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.905+0000 7fd2d7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd2d815baa0 0x7fd2d815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.906+0000 7fd2d6000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd2d815a6f0 0x7fd2d815ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53364/0 (socket says 172.21.15.102:53364) 2024-03-18T16:27:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.906+0000 7fd2d6000640 1 -- 172.21.15.102:0/1568874807 learned_addr learned my addr 172.21.15.102:0/1568874807 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:51.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.909+0000 7fd2d6000640 1 -- 172.21.15.102:0/1568874807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a310 msgr2=0x7fd2d806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:51.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.909+0000 7fd2d6000640 1 --2- 172.21.15.102:0/1568874807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a310 0x7fd2d806d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:51.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.909+0000 7fd2d6000640 1 -- 172.21.15.102:0/1568874807 >> v2:172.21.15.102:3300/0 conn(0x7fd2d815baa0 msgr2=0x7fd2d815bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:51.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.909+0000 7fd2d6000640 1 --2- 172.21.15.102:0/1568874807 >> v2:172.21.15.102:3300/0 conn(0x7fd2d815baa0 0x7fd2d815bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:51.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.909+0000 7fd2d6000640 1 -- 172.21.15.102:0/1568874807 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d815b050 con 0x7fd2d815a6f0 2024-03-18T16:27:51.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.911+0000 7fd2d6000640 1 --2- 172.21.15.102:0/1568874807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815a6f0 0x7fd2d815ab10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2d808cae0 tx=0x7fd2c402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=710aaf5d24d50414 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:51.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.916+0000 7fd2d5600640 1 -- 172.21.15.102:0/1568874807 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4035020 con 0x7fd2d815a6f0 2024-03-18T16:27:51.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.916+0000 7fd2d5600640 1 -- 172.21.15.102:0/1568874807 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2c4031870 con 0x7fd2d815a6f0 2024-03-18T16:27:51.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.916+0000 7fd2d5600640 1 -- 172.21.15.102:0/1568874807 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4031c40 con 0x7fd2d815a6f0 2024-03-18T16:27:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.917+0000 7fd2d7e00640 1 -- 172.21.15.102:0/1568874807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815a6f0 msgr2=0x7fd2d815ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.917+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/1568874807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815a6f0 0x7fd2d815ab10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd2d808cae0 tx=0x7fd2c402b910 comp rx=0 tx=0).stop 2024-03-18T16:27:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.917+0000 7fd2d7e00640 1 -- 172.21.15.102:0/1568874807 shutdown_connections 2024-03-18T16:27:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.917+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/1568874807 >> v2:172.21.15.102:3300/0 conn(0x7fd2d815baa0 0x7fd2d815bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.917+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/1568874807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815a6f0 0x7fd2d815ab10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.917+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/1568874807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a310 0x7fd2d806d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.917+0000 7fd2d7e00640 1 -- 172.21.15.102:0/1568874807 >> 172.21.15.102:0/1568874807 conn(0x7fd2d808c1a0 msgr2=0x7fd2d8066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.917+0000 7fd2d7e00640 1 -- 172.21.15.102:0/1568874807 shutdown_connections 2024-03-18T16:27:51.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.917+0000 7fd2d7e00640 1 -- 172.21.15.102:0/1568874807 wait complete. 2024-03-18T16:27:51.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.920+0000 7fd2d7e00640 1 Processor -- start 2024-03-18T16:27:51.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.920+0000 7fd2d7e00640 1 -- start start 2024-03-18T16:27:51.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.921+0000 7fd2d7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2d815a310 0x7fd2d81f04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:51.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.921+0000 7fd2d7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a6f0 0x7fd2d814de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:51.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.921+0000 7fd2d7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815baa0 0x7fd2d814e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:51.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.921+0000 7fd2d7e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d80cce50 con 0x7fd2d815a310 2024-03-18T16:27:51.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.921+0000 7fd2d7e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d80caa80 con 0x7fd2d815baa0 2024-03-18T16:27:51.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.921+0000 7fd2d7e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d80cd130 con 0x7fd2d815a6f0 2024-03-18T16:27:51.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.922+0000 7fd2d7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815baa0 0x7fd2d814e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:51.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.922+0000 7fd2d6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a6f0 0x7fd2d814de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:51.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.922+0000 7fd2d6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2d815a310 0x7fd2d81f04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:51.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.922+0000 7fd2d7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815baa0 0x7fd2d814e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53374/0 (socket says 172.21.15.102:53374) 2024-03-18T16:27:51.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.922+0000 7fd2d7400640 1 -- 172.21.15.102:0/3779763657 learned_addr learned my addr 172.21.15.102:0/3779763657 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.927+0000 7fd2d7400640 1 -- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a6f0 msgr2=0x7fd2d814de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.927+0000 7fd2d7400640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a6f0 0x7fd2d814de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.927+0000 7fd2d7400640 1 -- 172.21.15.102:0/3779763657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2d815a310 msgr2=0x7fd2d81f04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.927+0000 7fd2d7400640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2d815a310 0x7fd2d81f04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:51.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.927+0000 7fd2d7400640 1 -- 172.21.15.102:0/3779763657 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d81f4930 con 0x7fd2d815baa0 2024-03-18T16:27:51.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.930+0000 7fd2d7400640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815baa0 0x7fd2d814e340 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000efc0 tx=0x7fd2d000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:51.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.934+0000 7fd2cea00640 1 -- 172.21.15.102:0/3779763657 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0018020 con 0x7fd2d815baa0 2024-03-18T16:27:51.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.934+0000 7fd2cea00640 1 -- 172.21.15.102:0/3779763657 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d0014480 con 0x7fd2d815baa0 2024-03-18T16:27:51.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.934+0000 7fd2cea00640 1 -- 172.21.15.102:0/3779763657 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0014870 con 0x7fd2d815baa0 2024-03-18T16:27:51.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.934+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d81e4190 con 0x7fd2d815baa0 2024-03-18T16:27:51.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.934+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d814f370 con 0x7fd2d815baa0 2024-03-18T16:27:51.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.935+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d8074fd0 con 0x7fd2d815baa0 2024-03-18T16:27:51.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.955+0000 7fd2cea00640 1 -- 172.21.15.102:0/3779763657 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd2d0005d10 con 0x7fd2d815baa0 2024-03-18T16:27:51.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.956+0000 7fd2cea00640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd2b806bc20 0x7fd2b806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.956+0000 7fd2d6a00640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd2b806bc20 0x7fd2b806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.957+0000 7fd2d6a00640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd2b806bc20 0x7fd2b806e0e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009a20 tx=0x7fd2c8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:51.959+0000 7fd2cea00640 1 -- 172.21.15.102:0/3779763657 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd2d0058240 con 0x7fd2d815baa0 2024-03-18T16:27:52.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:27:52.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:27:52.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.219+0000 7fd2cea00640 1 -- 172.21.15.102:0/3779763657 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd2d0022030 con 0x7fd2d815baa0 2024-03-18T16:27:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.462+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2d80cee30 con 0x7fd2d815baa0 2024-03-18T16:27:52.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.475+0000 7fd2cea00640 1 -- 172.21.15.102:0/3779763657 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd2d0010040 con 0x7fd2d815baa0 2024-03-18T16:27:52.477 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:52.477 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:52.477 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:52.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd2b806bc20 msgr2=0x7fd2b806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:52.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd2b806bc20 0x7fd2b806e0e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009a20 tx=0x7fd2c8009190 comp rx=0 tx=0).stop 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815baa0 msgr2=0x7fd2d814e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815baa0 0x7fd2d814e340 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000efc0 tx=0x7fd2d000c490 comp rx=0 tx=0).stop 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 shutdown_connections 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd2b806bc20 0x7fd2b806e0e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2d815baa0 0x7fd2d814e340 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2d815a6f0 0x7fd2d814de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 --2- 172.21.15.102:0/3779763657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2d815a310 0x7fd2d81f04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 >> 172.21.15.102:0/3779763657 conn(0x7fd2d808c1a0 msgr2=0x7fd2d8066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.477+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 shutdown_connections 2024-03-18T16:27:52.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:52.478+0000 7fd2d7e00640 1 -- 172.21.15.102:0/3779763657 wait complete. 2024-03-18T16:27:53.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:53.108 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:54.109 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.282+0000 7fea46e00640 1 Processor -- start 2024-03-18T16:27:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.282+0000 7fea46e00640 1 -- start start 2024-03-18T16:27:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.283+0000 7fea46e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fea40154630 0x7fea400cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.283+0000 7fea46e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea400cf5c0 con 0x7fea400ca960 2024-03-18T16:27:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.284+0000 7fea46e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea4006d640 con 0x7fea400ca510 2024-03-18T16:27:54.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.284+0000 7fea46e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fea4006d920 con 0x7fea40154630 2024-03-18T16:27:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.285+0000 7fea46400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fea400ca960 0x7fea4015a520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47798/0 (socket says 172.21.15.102:47798) 2024-03-18T16:27:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.285+0000 7fea46400640 1 -- 172.21.15.102:0/533520099 learned_addr learned my addr 172.21.15.102:0/533520099 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:27:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.285+0000 7fea45a00640 1 --2- 172.21.15.102:0/533520099 >> v2:172.21.15.154:3300/0 conn(0x7fea40154630 0x7fea400cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.288+0000 7fea3fe00640 1 -- 172.21.15.102:0/533520099 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1186435007 0 0) 0x7fea400cf5c0 con 0x7fea400ca960 2024-03-18T16:27:54.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.288+0000 7fea3fe00640 1 -- 172.21.15.102:0/533520099 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea28003520 con 0x7fea400ca960 2024-03-18T16:27:54.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.290+0000 7fea45a00640 1 -- 172.21.15.102:0/533520099 >> v1:172.21.15.154:6789/0 conn(0x7fea400ca510 legacy=0x7fea400cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.290+0000 7fea45a00640 1 -- 172.21.15.102:0/533520099 >> v1:172.21.15.102:6789/0 conn(0x7fea400ca960 legacy=0x7fea4015a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:54.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.290+0000 7fea45a00640 1 -- 172.21.15.102:0/533520099 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4006dc00 con 0x7fea40154630 2024-03-18T16:27:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.293+0000 7fea45a00640 1 --2- 172.21.15.102:0/533520099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 0x7fea400cee80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fea340097a0 tx=0x7fea3402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3485da9bad8f7c9f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.297+0000 7fea3fe00640 1 -- 172.21.15.102:0/533520099 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3402f810 con 0x7fea40154630 2024-03-18T16:27:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.297+0000 7fea3fe00640 1 -- 172.21.15.102:0/533520099 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea3402f9b0 con 0x7fea40154630 2024-03-18T16:27:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.297+0000 7fea3fe00640 1 -- 172.21.15.102:0/533520099 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3402fd80 con 0x7fea40154630 2024-03-18T16:27:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.298+0000 7fea46e00640 1 -- 172.21.15.102:0/533520099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 msgr2=0x7fea400cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.298+0000 7fea46e00640 1 --2- 172.21.15.102:0/533520099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 0x7fea400cee80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fea340097a0 tx=0x7fea3402b9f0 comp rx=0 tx=0).stop 2024-03-18T16:27:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.298+0000 7fea46e00640 1 -- 172.21.15.102:0/533520099 shutdown_connections 2024-03-18T16:27:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.298+0000 7fea46e00640 1 --2- 172.21.15.102:0/533520099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 0x7fea400cee80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.298+0000 7fea46e00640 1 -- 172.21.15.102:0/533520099 >> 172.21.15.102:0/533520099 conn(0x7fea40067160 msgr2=0x7fea40067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.298+0000 7fea46e00640 1 -- 172.21.15.102:0/533520099 shutdown_connections 2024-03-18T16:27:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.298+0000 7fea46e00640 1 -- 172.21.15.102:0/533520099 wait complete. 2024-03-18T16:27:54.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.300+0000 7fea46e00640 1 Processor -- start 2024-03-18T16:27:54.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.300+0000 7fea46e00640 1 -- start start 2024-03-18T16:27:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.301+0000 7fea46e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea400ca510 0x7fea40159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.301+0000 7fea46e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea400ca960 0x7fea4014ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.301+0000 7fea46e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 0x7fea4014d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.301+0000 7fea46e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fea40154b90 con 0x7fea400ca960 2024-03-18T16:27:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.301+0000 7fea46e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea40154a10 con 0x7fea40154630 2024-03-18T16:27:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.301+0000 7fea46e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fea400d0540 con 0x7fea400ca510 2024-03-18T16:27:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.302+0000 7fea45a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea400ca510 0x7fea40159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.302+0000 7fea46400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 0x7fea4014d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.302+0000 7fea45000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea400ca960 0x7fea4014ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.302+0000 7fea46400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 0x7fea4014d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53398/0 (socket says 172.21.15.102:53398) 2024-03-18T16:27:54.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.302+0000 7fea46400640 1 -- 172.21.15.102:0/3750296304 learned_addr learned my addr 172.21.15.102:0/3750296304 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.306+0000 7fea45000640 1 -- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea400ca510 msgr2=0x7fea40159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.306+0000 7fea45000640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea400ca510 0x7fea40159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.306+0000 7fea45000640 1 -- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 msgr2=0x7fea4014d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.306+0000 7fea45000640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 0x7fea4014d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:54.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.306+0000 7fea45000640 1 -- 172.21.15.102:0/3750296304 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea40155700 con 0x7fea400ca960 2024-03-18T16:27:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.310+0000 7fea45000640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea400ca960 0x7fea4014ce00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00c3c0 tx=0x7fea2c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.314+0000 7fea3d600640 1 -- 172.21.15.102:0/3750296304 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c017070 con 0x7fea400ca960 2024-03-18T16:27:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.314+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea401559e0 con 0x7fea400ca960 2024-03-18T16:27:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.314+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea40155fa0 con 0x7fea400ca960 2024-03-18T16:27:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.315+0000 7fea3d600640 1 -- 172.21.15.102:0/3750296304 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea2c0052d0 con 0x7fea400ca960 2024-03-18T16:27:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.315+0000 7fea3d600640 1 -- 172.21.15.102:0/3750296304 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c021060 con 0x7fea400ca960 2024-03-18T16:27:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.316+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea1c005290 con 0x7fea400ca960 2024-03-18T16:27:54.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.587+0000 7fea3d600640 1 -- 172.21.15.102:0/3750296304 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fea2c01a070 con 0x7fea400ca960 2024-03-18T16:27:54.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.588+0000 7fea3d600640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea2006bbd0 0x7fea2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:54.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.589+0000 7fea45a00640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea2006bbd0 0x7fea2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:54.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.590+0000 7fea45a00640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea2006bbd0 0x7fea2006e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fea34036000 tx=0x7fea34008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:54.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.615+0000 7fea3d600640 1 -- 172.21.15.102:0/3750296304 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fea2c056a30 con 0x7fea400ca960 2024-03-18T16:27:54.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.615+0000 7fea3d600640 1 -- 172.21.15.102:0/3750296304 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fea2c058990 con 0x7fea400ca960 2024-03-18T16:27:54.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.885+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea1c005580 con 0x7fea400ca960 2024-03-18T16:27:54.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.906+0000 7fea3d600640 1 -- 172.21.15.102:0/3750296304 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fea2c05ca30 con 0x7fea400ca960 2024-03-18T16:27:54.909 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:54.909 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:54.909 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:54.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea2006bbd0 msgr2=0x7fea2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:54.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea2006bbd0 0x7fea2006e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fea34036000 tx=0x7fea34008030 comp rx=0 tx=0).stop 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea400ca960 msgr2=0x7fea4014ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea400ca960 0x7fea4014ce00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00c3c0 tx=0x7fea2c00c880 comp rx=0 tx=0).stop 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 shutdown_connections 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea2006bbd0 0x7fea2006e090 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea40154630 0x7fea4014d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea400ca960 0x7fea4014ce00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 --2- 172.21.15.102:0/3750296304 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea400ca510 0x7fea40159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 >> 172.21.15.102:0/3750296304 conn(0x7fea40067160 msgr2=0x7fea401526e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.909+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 shutdown_connections 2024-03-18T16:27:54.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:54.910+0000 7fea46e00640 1 -- 172.21.15.102:0/3750296304 wait complete. 2024-03-18T16:27:55.531 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:55.531 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:56.532 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:56.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.708+0000 7f6fa1400640 1 Processor -- start 2024-03-18T16:27:56.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.708+0000 7f6fa1400640 1 -- start start 2024-03-18T16:27:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.709+0000 7f6fa1400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6f9c0cad40 0x7f6f9c06c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.709+0000 7f6fa1400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6f9c15a1a0 0x7f6f9c0705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.709+0000 7f6fa1400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f9c06c9c0 con 0x7f6f9c0ca960 2024-03-18T16:27:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.709+0000 7f6fa1400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6f9c0c8680 con 0x7f6f9c15a1a0 2024-03-18T16:27:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.709+0000 7f6fa1400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6f9c06cca0 con 0x7f6f9c0cad40 2024-03-18T16:27:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.710+0000 7f6fa0a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6f9c15a1a0 0x7f6f9c0705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:56.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.711+0000 7f6fa0a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6f9c15a1a0 0x7f6f9c0705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60396/0 (socket says 172.21.15.102:60396) 2024-03-18T16:27:56.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.711+0000 7f6fa0a00640 1 -- 172.21.15.102:0/834471501 learned_addr learned my addr 172.21.15.102:0/834471501 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:56.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.714+0000 7f6fa0a00640 1 -- 172.21.15.102:0/834471501 >> v2:172.21.15.154:3300/0 conn(0x7f6f9c0cad40 msgr2=0x7f6f9c06c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:56.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.714+0000 7f6fa0a00640 1 --2- 172.21.15.102:0/834471501 >> v2:172.21.15.154:3300/0 conn(0x7f6f9c0cad40 0x7f6f9c06c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:56.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.714+0000 7f6fa0a00640 1 -- 172.21.15.102:0/834471501 >> v1:172.21.15.154:6789/0 conn(0x7f6f9c0ca960 legacy=0x7f6f9c15e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:56.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.714+0000 7f6fa0a00640 1 -- 172.21.15.102:0/834471501 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9c06cf80 con 0x7f6f9c15a1a0 2024-03-18T16:27:56.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.717+0000 7f6fa0a00640 1 --2- 172.21.15.102:0/834471501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c15a1a0 0x7f6f9c0705a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0cb1f0 tx=0x7f6f9402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac7ef4fad449e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:56.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.722+0000 7f6f9aa00640 1 -- 172.21.15.102:0/834471501 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9402cb20 con 0x7f6f9c15a1a0 2024-03-18T16:27:56.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.723+0000 7f6f9aa00640 1 -- 172.21.15.102:0/834471501 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f9402ccc0 con 0x7f6f9c15a1a0 2024-03-18T16:27:56.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.723+0000 7f6f9aa00640 1 -- 172.21.15.102:0/834471501 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94034610 con 0x7f6f9c15a1a0 2024-03-18T16:27:56.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.724+0000 7f6fa1400640 1 -- 172.21.15.102:0/834471501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c15a1a0 msgr2=0x7f6f9c0705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:56.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.724+0000 7f6fa1400640 1 --2- 172.21.15.102:0/834471501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c15a1a0 0x7f6f9c0705a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c0cb1f0 tx=0x7f6f9402bf80 comp rx=0 tx=0).stop 2024-03-18T16:27:56.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.724+0000 7f6fa1400640 1 -- 172.21.15.102:0/834471501 shutdown_connections 2024-03-18T16:27:56.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.724+0000 7f6fa1400640 1 --2- 172.21.15.102:0/834471501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c15a1a0 0x7f6f9c0705a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:56.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.724+0000 7f6fa1400640 1 --2- 172.21.15.102:0/834471501 >> v2:172.21.15.154:3300/0 conn(0x7f6f9c0cad40 0x7f6f9c06c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:56.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.724+0000 7f6fa1400640 1 -- 172.21.15.102:0/834471501 >> 172.21.15.102:0/834471501 conn(0x7f6f9c067190 msgr2=0x7f6f9c0675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:56.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.724+0000 7f6fa1400640 1 -- 172.21.15.102:0/834471501 shutdown_connections 2024-03-18T16:27:56.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.724+0000 7f6fa1400640 1 -- 172.21.15.102:0/834471501 wait complete. 2024-03-18T16:27:56.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.727+0000 7f6fa1400640 1 Processor -- start 2024-03-18T16:27:56.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.727+0000 7f6fa1400640 1 -- start start 2024-03-18T16:27:56.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.728+0000 7f6fa1400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f9c0ca960 0x7f6f9c1f3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:56.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.728+0000 7f6fa1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c0cad40 0x7f6f9c1f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:56.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.729+0000 7f6fa1400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f9c15a1a0 0x7f6f9c1e3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:56.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.729+0000 7f6fa1400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c070ae0 con 0x7f6f9c0cad40 2024-03-18T16:27:56.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.729+0000 7f6fa1400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9c15dc10 con 0x7f6f9c0ca960 2024-03-18T16:27:56.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.729+0000 7f6fa1400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6f9c070c60 con 0x7f6f9c15a1a0 2024-03-18T16:27:56.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.729+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f9c0ca960 0x7f6f9c1f3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:56.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.729+0000 7f6fa0a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f9c15a1a0 0x7f6f9c1e3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:56.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.729+0000 7f6f9be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f9c0ca960 0x7f6f9c1f3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53418/0 (socket says 172.21.15.102:53418) 2024-03-18T16:27:56.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.729+0000 7f6f9be00640 1 -- 172.21.15.102:0/1056671632 learned_addr learned my addr 172.21.15.102:0/1056671632 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:56.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.733+0000 7f6f9be00640 1 -- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f9c15a1a0 msgr2=0x7f6f9c1e3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:56.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.733+0000 7f6f9be00640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f9c15a1a0 0x7f6f9c1e3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:56.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.733+0000 7f6f9be00640 1 -- 172.21.15.102:0/1056671632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c0cad40 msgr2=0x7f6f9c1f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:56.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.733+0000 7f6f9be00640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c0cad40 0x7f6f9c1f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:56.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.733+0000 7f6f9be00640 1 -- 172.21.15.102:0/1056671632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f9c153a60 con 0x7f6f9c0ca960 2024-03-18T16:27:56.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.736+0000 7f6f9be00640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f9c0ca960 0x7f6f9c1f3110 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00c410 tx=0x7f6f8c00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:56.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.740+0000 7f6f93e00640 1 -- 172.21.15.102:0/1056671632 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c017070 con 0x7f6f9c0ca960 2024-03-18T16:27:56.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.740+0000 7f6f93e00640 1 -- 172.21.15.102:0/1056671632 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f8c0035f0 con 0x7f6f9c0ca960 2024-03-18T16:27:56.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.740+0000 7f6f93e00640 1 -- 172.21.15.102:0/1056671632 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c003a20 con 0x7f6f9c0ca960 2024-03-18T16:27:56.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.740+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9c1f3d50 con 0x7f6f9c0ca960 2024-03-18T16:27:56.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.740+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f9c13cf10 con 0x7f6f9c0ca960 2024-03-18T16:27:56.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.742+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f9c074fd0 con 0x7f6f9c0ca960 2024-03-18T16:27:56.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.766+0000 7f6f93e00640 1 -- 172.21.15.102:0/1056671632 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6f8c01a070 con 0x7f6f9c0ca960 2024-03-18T16:27:56.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.768+0000 7f6f93e00640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6f7c06bc20 0x7f6f7c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:56.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.768+0000 7f6f9b400640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6f7c06bc20 0x7f6f7c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:56.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:56.769+0000 7f6f9b400640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6f7c06bc20 0x7f6f7c06e0e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c1ed560 tx=0x7f6f88006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:57.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.011+0000 7f6f93e00640 1 -- 172.21.15.102:0/1056671632 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6f8c056870 con 0x7f6f9c0ca960 2024-03-18T16:27:57.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.025+0000 7f6f93e00640 1 -- 172.21.15.102:0/1056671632 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6f8c021020 con 0x7f6f9c0ca960 2024-03-18T16:27:57.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.321+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f9c0b2cb0 con 0x7f6f9c0ca960 2024-03-18T16:27:57.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.411+0000 7f6f93e00640 1 -- 172.21.15.102:0/1056671632 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f6f8c0010c0 con 0x7f6f9c0ca960 2024-03-18T16:27:57.413 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:57.413 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:57.414 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6f7c06bc20 msgr2=0x7f6f7c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6f7c06bc20 0x7f6f7c06e0e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c1ed560 tx=0x7f6f88006c90 comp rx=0 tx=0).stop 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f9c0ca960 msgr2=0x7f6f9c1f3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f9c0ca960 0x7f6f9c1f3110 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00c410 tx=0x7f6f8c00c8d0 comp rx=0 tx=0).stop 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 shutdown_connections 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6f7c06bc20 0x7f6f7c06e0e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6f9c15a1a0 0x7f6f9c1e3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f9c0cad40 0x7f6f9c1f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 --2- 172.21.15.102:0/1056671632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f9c0ca960 0x7f6f9c1f3110 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 >> 172.21.15.102:0/1056671632 conn(0x7f6f9c067190 msgr2=0x7f6f9c067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:57.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 shutdown_connections 2024-03-18T16:27:57.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:57.414+0000 7f6fa1400640 1 -- 172.21.15.102:0/1056671632 wait complete. 2024-03-18T16:27:58.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:27:58.035 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:27:59.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:27:59.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.210+0000 7f7bf6200640 1 Processor -- start 2024-03-18T16:27:59.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.210+0000 7f7bf6200640 1 -- start start 2024-03-18T16:27:59.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.211+0000 7f7bf6200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7bf0154520 0x7f7bf00c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:59.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.211+0000 7f7bf6200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7bf00c3890 0x7f7bf00c8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.211+0000 7f7bf6200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bf015a170 con 0x7f7bf00c3cd0 2024-03-18T16:27:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.211+0000 7f7bf6200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7bf0154900 con 0x7f7bf00c3890 2024-03-18T16:27:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.211+0000 7f7bf6200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7bf00c4160 con 0x7f7bf0154520 2024-03-18T16:27:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.212+0000 7f7bf4e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7bf0154520 0x7f7bf00c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.212+0000 7f7befe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7bf00c3890 0x7f7bf00c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.212+0000 7f7bf4e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7bf0154520 0x7f7bf00c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53432/0 (socket says 172.21.15.102:53432) 2024-03-18T16:27:59.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.212+0000 7f7bf4e00640 1 -- 172.21.15.102:0/1976424527 learned_addr learned my addr 172.21.15.102:0/1976424527 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:59.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.215+0000 7f7bf4e00640 1 -- 172.21.15.102:0/1976424527 >> v2:172.21.15.102:3300/0 conn(0x7f7bf00c3890 msgr2=0x7f7bf00c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:59.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.215+0000 7f7bf4e00640 1 --2- 172.21.15.102:0/1976424527 >> v2:172.21.15.102:3300/0 conn(0x7f7bf00c3890 0x7f7bf00c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:59.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.215+0000 7f7bf4e00640 1 -- 172.21.15.102:0/1976424527 >> v1:172.21.15.102:6789/0 conn(0x7f7bf00c3cd0 legacy=0x7f7bf00c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:27:59.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.215+0000 7f7bf4e00640 1 -- 172.21.15.102:0/1976424527 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf00c4440 con 0x7f7bf0154520 2024-03-18T16:27:59.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.217+0000 7f7bf4e00640 1 --2- 172.21.15.102:0/1976424527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 0x7f7bf00c9eb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00cb4c0 tx=0x7f7be402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0cfd26674c007af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.220+0000 7f7bef400640 1 -- 172.21.15.102:0/1976424527 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4035020 con 0x7f7bf0154520 2024-03-18T16:27:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.220+0000 7f7bef400640 1 -- 172.21.15.102:0/1976424527 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be4031870 con 0x7f7bf0154520 2024-03-18T16:27:59.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.220+0000 7f7bef400640 1 -- 172.21.15.102:0/1976424527 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4031c40 con 0x7f7bf0154520 2024-03-18T16:27:59.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.221+0000 7f7bf6200640 1 -- 172.21.15.102:0/1976424527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 msgr2=0x7f7bf00c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:59.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.221+0000 7f7bf6200640 1 --2- 172.21.15.102:0/1976424527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 0x7f7bf00c9eb0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00cb4c0 tx=0x7f7be402b910 comp rx=0 tx=0).stop 2024-03-18T16:27:59.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.221+0000 7f7bf6200640 1 -- 172.21.15.102:0/1976424527 shutdown_connections 2024-03-18T16:27:59.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.221+0000 7f7bf6200640 1 --2- 172.21.15.102:0/1976424527 >> v2:172.21.15.102:3300/0 conn(0x7f7bf00c3890 0x7f7bf00c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:59.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.221+0000 7f7bf6200640 1 --2- 172.21.15.102:0/1976424527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 0x7f7bf00c9eb0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:59.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.221+0000 7f7bf6200640 1 -- 172.21.15.102:0/1976424527 >> 172.21.15.102:0/1976424527 conn(0x7f7bf0066f80 msgr2=0x7f7bf0067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:59.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.222+0000 7f7bf6200640 1 -- 172.21.15.102:0/1976424527 shutdown_connections 2024-03-18T16:27:59.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.222+0000 7f7bf6200640 1 -- 172.21.15.102:0/1976424527 wait complete. 2024-03-18T16:27:59.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.224+0000 7f7bf6200640 1 Processor -- start 2024-03-18T16:27:59.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.224+0000 7f7bf6200640 1 -- start start 2024-03-18T16:27:59.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.225+0000 7f7bf6200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bf00c3890 0x7f7bf0146180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:59.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.225+0000 7f7bf6200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 0x7f7bf015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:59.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.225+0000 7f7bf6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bf00c4d10 0x7f7bf013d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:59.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.225+0000 7f7bf6200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf015a9f0 con 0x7f7bf00c4d10 2024-03-18T16:27:59.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.225+0000 7f7bf6200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf015a710 con 0x7f7bf0154520 2024-03-18T16:27:59.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.225+0000 7f7bf6200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf00ca3f0 con 0x7f7bf00c3890 2024-03-18T16:27:59.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.226+0000 7f7befe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 0x7f7bf015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:59.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.226+0000 7f7bf5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bf00c4d10 0x7f7bf013d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:59.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.226+0000 7f7bf4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bf00c3890 0x7f7bf0146180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:59.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.226+0000 7f7befe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 0x7f7bf015c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53448/0 (socket says 172.21.15.102:53448) 2024-03-18T16:27:59.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.226+0000 7f7befe00640 1 -- 172.21.15.102:0/4165101751 learned_addr learned my addr 172.21.15.102:0/4165101751 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:27:59.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.230+0000 7f7befe00640 1 -- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bf00c3890 msgr2=0x7f7bf0146180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:59.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.230+0000 7f7befe00640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bf00c3890 0x7f7bf0146180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:59.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.230+0000 7f7befe00640 1 -- 172.21.15.102:0/4165101751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bf00c4d10 msgr2=0x7f7bf013d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:59.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.230+0000 7f7befe00640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bf00c4d10 0x7f7bf013d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:59.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.230+0000 7f7befe00640 1 -- 172.21.15.102:0/4165101751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf01466c0 con 0x7f7bf0154520 2024-03-18T16:27:59.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.232+0000 7f7befe00640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 0x7f7bf015c1c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00e9b0 tx=0x7f7bdc00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:59.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.236+0000 7f7becc00640 1 -- 172.21.15.102:0/4165101751 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc017070 con 0x7f7bf0154520 2024-03-18T16:27:59.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.236+0000 7f7becc00640 1 -- 172.21.15.102:0/4165101751 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bdc00c9a0 con 0x7f7bf0154520 2024-03-18T16:27:59.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.236+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf015df80 con 0x7f7bf0154520 2024-03-18T16:27:59.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.236+0000 7f7becc00640 1 -- 172.21.15.102:0/4165101751 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc00cd90 con 0x7f7bf0154520 2024-03-18T16:27:59.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.237+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf0145980 con 0x7f7bf0154520 2024-03-18T16:27:59.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.238+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf0074fd0 con 0x7f7bf0154520 2024-03-18T16:27:59.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.503+0000 7f7becc00640 1 -- 172.21.15.102:0/4165101751 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7bdc01a070 con 0x7f7bf0154520 2024-03-18T16:27:59.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.504+0000 7f7becc00640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bd006bc50 0x7f7bd006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:27:59.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.504+0000 7f7becc00640 1 -- 172.21.15.102:0/4165101751 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7bdc08b560 con 0x7f7bf0154520 2024-03-18T16:27:59.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.504+0000 7f7bf4e00640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bd006bc50 0x7f7bd006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:27:59.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.505+0000 7f7bf4e00640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bd006bc50 0x7f7bd006e110 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7be4036000 tx=0x7f7be403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:27:59.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.514+0000 7f7becc00640 1 -- 172.21.15.102:0/4165101751 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7bdc0595f0 con 0x7f7bf0154520 2024-03-18T16:27:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.876+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bf014acb0 con 0x7f7bf0154520 2024-03-18T16:27:59.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.891+0000 7f7becc00640 1 -- 172.21.15.102:0/4165101751 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7bdc05d690 con 0x7f7bf0154520 2024-03-18T16:27:59.893 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:27:59.893 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:27:59.893 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:27:59.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.893+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bd006bc50 msgr2=0x7f7bd006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:59.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.893+0000 7f7bf6200640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bd006bc50 0x7f7bd006e110 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7be4036000 tx=0x7f7be403c040 comp rx=0 tx=0).stop 2024-03-18T16:27:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.893+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 msgr2=0x7f7bf015c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:27:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.893+0000 7f7bf6200640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 0x7f7bf015c1c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00e9b0 tx=0x7f7bdc00ee70 comp rx=0 tx=0).stop 2024-03-18T16:27:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.894+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 shutdown_connections 2024-03-18T16:27:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.894+0000 7f7bf6200640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bd006bc50 0x7f7bd006e110 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.894+0000 7f7bf6200640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bf00c4d10 0x7f7bf013d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.894+0000 7f7bf6200640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bf0154520 0x7f7bf015c1c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.894+0000 7f7bf6200640 1 --2- 172.21.15.102:0/4165101751 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bf00c3890 0x7f7bf0146180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:27:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.894+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 >> 172.21.15.102:0/4165101751 conn(0x7f7bf0066f80 msgr2=0x7f7bf0157e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:27:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.894+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 shutdown_connections 2024-03-18T16:27:59.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:27:59.894+0000 7f7bf6200640 1 -- 172.21.15.102:0/4165101751 wait complete. 2024-03-18T16:28:00.517 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:00.517 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:01.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:01.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.689+0000 7fa9c1a00640 1 Processor -- start 2024-03-18T16:28:01.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.690+0000 7fa9c1a00640 1 -- start start 2024-03-18T16:28:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.691+0000 7fa9c1a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa9bc1524c0 0x7fa9bc072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.691+0000 7fa9c1a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9bc15b8e0 0x7fa9bc0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.691+0000 7fa9c1a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9bc15c200 con 0x7fa9bc158ee0 2024-03-18T16:28:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.691+0000 7fa9c1a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa9bc159300 con 0x7fa9bc15b8e0 2024-03-18T16:28:01.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.691+0000 7fa9c1a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa9bc15c4e0 con 0x7fa9bc1524c0 2024-03-18T16:28:01.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.692+0000 7fa9baa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa9bc1524c0 0x7fa9bc072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:01.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.693+0000 7fa9baa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa9bc1524c0 0x7fa9bc072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40890/0 (socket says 172.21.15.102:40890) 2024-03-18T16:28:01.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.693+0000 7fa9baa00640 1 -- 172.21.15.102:0/2011084366 learned_addr learned my addr 172.21.15.102:0/2011084366 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:01.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.695+0000 7fa9baa00640 1 -- 172.21.15.102:0/2011084366 >> v2:172.21.15.102:3300/0 conn(0x7fa9bc15b8e0 msgr2=0x7fa9bc0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:01.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.695+0000 7fa9baa00640 1 --2- 172.21.15.102:0/2011084366 >> v2:172.21.15.102:3300/0 conn(0x7fa9bc15b8e0 0x7fa9bc0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:01.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.695+0000 7fa9baa00640 1 -- 172.21.15.102:0/2011084366 >> v1:172.21.15.154:6789/0 conn(0x7fa9bc158ee0 legacy=0x7fa9bc06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:01.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.695+0000 7fa9baa00640 1 -- 172.21.15.102:0/2011084366 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc15c7c0 con 0x7fa9bc1524c0 2024-03-18T16:28:01.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.702+0000 7fa9baa00640 1 --2- 172.21.15.102:0/2011084366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc1524c0 0x7fa9bc072700 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc06bf90 tx=0x7fa9ac02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f452aca0ee576c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:01.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.757+0000 7fa9b9600640 1 -- 172.21.15.102:0/2011084366 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac034070 con 0x7fa9bc1524c0 2024-03-18T16:28:01.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.757+0000 7fa9b9600640 1 -- 172.21.15.102:0/2011084366 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9ac030860 con 0x7fa9bc1524c0 2024-03-18T16:28:01.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.757+0000 7fa9b9600640 1 -- 172.21.15.102:0/2011084366 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac030c30 con 0x7fa9bc1524c0 2024-03-18T16:28:01.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.758+0000 7fa9c1a00640 1 -- 172.21.15.102:0/2011084366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc1524c0 msgr2=0x7fa9bc072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:01.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.758+0000 7fa9c1a00640 1 --2- 172.21.15.102:0/2011084366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc1524c0 0x7fa9bc072700 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc06bf90 tx=0x7fa9ac02b9a0 comp rx=0 tx=0).stop 2024-03-18T16:28:01.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.758+0000 7fa9c1a00640 1 -- 172.21.15.102:0/2011084366 shutdown_connections 2024-03-18T16:28:01.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.758+0000 7fa9c1a00640 1 --2- 172.21.15.102:0/2011084366 >> v2:172.21.15.102:3300/0 conn(0x7fa9bc15b8e0 0x7fa9bc0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:01.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.758+0000 7fa9c1a00640 1 --2- 172.21.15.102:0/2011084366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc1524c0 0x7fa9bc072700 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:01.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.758+0000 7fa9c1a00640 1 -- 172.21.15.102:0/2011084366 >> 172.21.15.102:0/2011084366 conn(0x7fa9bc0700d0 msgr2=0x7fa9bc06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:01.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.758+0000 7fa9c1a00640 1 -- 172.21.15.102:0/2011084366 shutdown_connections 2024-03-18T16:28:01.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.758+0000 7fa9c1a00640 1 -- 172.21.15.102:0/2011084366 wait complete. 2024-03-18T16:28:01.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.760+0000 7fa9c1a00640 1 Processor -- start 2024-03-18T16:28:01.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.760+0000 7fa9c1a00640 1 -- start start 2024-03-18T16:28:01.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.761+0000 7fa9c1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc1524c0 0x7fa9bc155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:01.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.762+0000 7fa9c1a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc158ee0 0x7fa9bc1f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:01.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.762+0000 7fa9c1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9bc15b8e0 0x7fa9bc0c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:01.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.762+0000 7fa9c1a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1589c0 con 0x7fa9bc1524c0 2024-03-18T16:28:01.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.762+0000 7fa9c1a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc1586e0 con 0x7fa9bc158ee0 2024-03-18T16:28:01.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.762+0000 7fa9c1a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa9bc1528a0 con 0x7fa9bc15b8e0 2024-03-18T16:28:01.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.763+0000 7fa9bb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9bc15b8e0 0x7fa9bc0c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:01.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.763+0000 7fa9bb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9bc15b8e0 0x7fa9bc0c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50244/0 (socket says 172.21.15.102:50244) 2024-03-18T16:28:01.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.763+0000 7fa9bb400640 1 -- 172.21.15.102:0/3815586473 learned_addr learned my addr 172.21.15.102:0/3815586473 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:01.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.763+0000 7fa9ba000640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc158ee0 0x7fa9bc1f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:01.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.767+0000 7fa9baa00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc1524c0 0x7fa9bc155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:01.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.768+0000 7fa9ba000640 1 -- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9bc15b8e0 msgr2=0x7fa9bc0c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:01.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.768+0000 7fa9ba000640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9bc15b8e0 0x7fa9bc0c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:01.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.768+0000 7fa9ba000640 1 -- 172.21.15.102:0/3815586473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc1524c0 msgr2=0x7fa9bc155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:01.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.768+0000 7fa9ba000640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc1524c0 0x7fa9bc155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:01.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.768+0000 7fa9ba000640 1 -- 172.21.15.102:0/3815586473 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc0c6840 con 0x7fa9bc158ee0 2024-03-18T16:28:01.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.770+0000 7fa9ba000640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc158ee0 0x7fa9bc1f4950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800c390 tx=0x7fa9a800c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:01.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.774+0000 7fa9b2a00640 1 -- 172.21.15.102:0/3815586473 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8017070 con 0x7fa9bc158ee0 2024-03-18T16:28:01.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.774+0000 7fa9b2a00640 1 -- 172.21.15.102:0/3815586473 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a80052d0 con 0x7fa9bc158ee0 2024-03-18T16:28:01.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.774+0000 7fa9b2a00640 1 -- 172.21.15.102:0/3815586473 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8003830 con 0x7fa9bc158ee0 2024-03-18T16:28:01.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.774+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9bc0c6b00 con 0x7fa9bc158ee0 2024-03-18T16:28:01.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.774+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9bc155140 con 0x7fa9bc158ee0 2024-03-18T16:28:01.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.776+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9bc074fd0 con 0x7fa9bc158ee0 2024-03-18T16:28:01.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.795+0000 7fa9b2a00640 1 -- 172.21.15.102:0/3815586473 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa9a801a070 con 0x7fa9bc158ee0 2024-03-18T16:28:01.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.797+0000 7fa9b2a00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa99c06bc50 0x7fa99c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:01.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.797+0000 7fa9baa00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa99c06bc50 0x7fa99c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:01.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:01.798+0000 7fa9baa00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa99c06bc50 0x7fa99c06e110 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac036000 tx=0x7fa9ac03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:02.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.038+0000 7fa9b2a00640 1 -- 172.21.15.102:0/3815586473 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa9a8056a00 con 0x7fa9bc158ee0 2024-03-18T16:28:02.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.049+0000 7fa9b2a00640 1 -- 172.21.15.102:0/3815586473 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa9a8021070 con 0x7fa9bc158ee0 2024-03-18T16:28:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.296+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9bc0c5950 con 0x7fa9bc158ee0 2024-03-18T16:28:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.309+0000 7fa9b2a00640 1 -- 172.21.15.102:0/3815586473 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa9a8058960 con 0x7fa9bc158ee0 2024-03-18T16:28:02.311 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:02.311 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa99c06bc50 msgr2=0x7fa99c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa99c06bc50 0x7fa99c06e110 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac036000 tx=0x7fa9ac03b040 comp rx=0 tx=0).stop 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc158ee0 msgr2=0x7fa9bc1f4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc158ee0 0x7fa9bc1f4950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800c390 tx=0x7fa9a800c850 comp rx=0 tx=0).stop 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 shutdown_connections 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa99c06bc50 0x7fa99c06e110 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa9bc15b8e0 0x7fa9bc0c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9bc158ee0 0x7fa9bc1f4950 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 --2- 172.21.15.102:0/3815586473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9bc1524c0 0x7fa9bc155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.312+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 >> 172.21.15.102:0/3815586473 conn(0x7fa9bc0700d0 msgr2=0x7fa9bc157b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.313+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 shutdown_connections 2024-03-18T16:28:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:02.313+0000 7fa9c1a00640 1 -- 172.21.15.102:0/3815586473 wait complete. 2024-03-18T16:28:02.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:02.936 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:03.937 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:04.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.114+0000 7f3be1a00640 1 Processor -- start 2024-03-18T16:28:04.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.114+0000 7f3be1a00640 1 -- start start 2024-03-18T16:28:04.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.115+0000 7f3be1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc141650 0x7f3bdc158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:04.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.115+0000 7f3be1a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0c4fa0 0x7f3bdc0c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:04.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.115+0000 7f3be1a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bdc0c72d0 con 0x7f3bdc15c3a0 2024-03-18T16:28:04.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.115+0000 7f3be1a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3bdc159370 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.115+0000 7f3be1a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bdc158ad0 con 0x7f3bdc141650 2024-03-18T16:28:04.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.116+0000 7f3bdbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc141650 0x7f3bdc158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:04.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.116+0000 7f3be1000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0c4fa0 0x7f3bdc0c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:04.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.116+0000 7f3bdbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc141650 0x7f3bdc158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50260/0 (socket says 172.21.15.102:50260) 2024-03-18T16:28:04.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.116+0000 7f3bdbe00640 1 -- 172.21.15.102:0/1378889850 learned_addr learned my addr 172.21.15.102:0/1378889850 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:04.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.119+0000 7f3bdbe00640 1 -- 172.21.15.102:0/1378889850 >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0c4fa0 msgr2=0x7f3bdc0c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:04.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.119+0000 7f3bdbe00640 1 --2- 172.21.15.102:0/1378889850 >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0c4fa0 0x7f3bdc0c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:04.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.119+0000 7f3bdbe00640 1 -- 172.21.15.102:0/1378889850 >> v1:172.21.15.102:6789/0 conn(0x7f3bdc15c3a0 legacy=0x7f3bdc0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:04.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.119+0000 7f3bdbe00640 1 -- 172.21.15.102:0/1378889850 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bdc158db0 con 0x7f3bdc141650 2024-03-18T16:28:04.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.121+0000 7f3bdbe00640 1 --2- 172.21.15.102:0/1378889850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc141650 0x7f3bdc158420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009870 tx=0x7f3bcc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d36956d803e70bc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:04.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.125+0000 7f3bdaa00640 1 -- 172.21.15.102:0/1378889850 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc034070 con 0x7f3bdc141650 2024-03-18T16:28:04.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.125+0000 7f3bdaa00640 1 -- 172.21.15.102:0/1378889850 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bcc030840 con 0x7f3bdc141650 2024-03-18T16:28:04.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.125+0000 7f3bdaa00640 1 -- 172.21.15.102:0/1378889850 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc030c10 con 0x7f3bdc141650 2024-03-18T16:28:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.126+0000 7f3be1a00640 1 -- 172.21.15.102:0/1378889850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc141650 msgr2=0x7f3bdc158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.126+0000 7f3be1a00640 1 --2- 172.21.15.102:0/1378889850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc141650 0x7f3bdc158420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009870 tx=0x7f3bcc02b980 comp rx=0 tx=0).stop 2024-03-18T16:28:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.126+0000 7f3be1a00640 1 -- 172.21.15.102:0/1378889850 shutdown_connections 2024-03-18T16:28:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.126+0000 7f3be1a00640 1 --2- 172.21.15.102:0/1378889850 >> v2:172.21.15.102:3300/0 conn(0x7f3bdc0c4fa0 0x7f3bdc0c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:04.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.126+0000 7f3be1a00640 1 --2- 172.21.15.102:0/1378889850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc141650 0x7f3bdc158420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.126+0000 7f3be1a00640 1 -- 172.21.15.102:0/1378889850 >> 172.21.15.102:0/1378889850 conn(0x7f3bdc0cb890 msgr2=0x7f3bdc0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.126+0000 7f3be1a00640 1 -- 172.21.15.102:0/1378889850 shutdown_connections 2024-03-18T16:28:04.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.126+0000 7f3be1a00640 1 -- 172.21.15.102:0/1378889850 wait complete. 2024-03-18T16:28:04.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.129+0000 7f3be1a00640 1 Processor -- start 2024-03-18T16:28:04.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.129+0000 7f3be1a00640 1 -- start start 2024-03-18T16:28:04.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.131+0000 7f3be1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc0c4fa0 0x7f3bdc13d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:04.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.131+0000 7f3be1a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bdc141650 0x7f3bdc1e5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:04.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.131+0000 7f3be1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc15c3a0 0x7f3bdc1e7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:04.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.131+0000 7f3be1a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc0c66a0 con 0x7f3bdc15c3a0 2024-03-18T16:28:04.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.131+0000 7f3be1a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bdc0c63c0 con 0x7f3bdc141650 2024-03-18T16:28:04.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.131+0000 7f3be1a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bdc15c7c0 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.131+0000 7f3bdbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc0c4fa0 0x7f3bdc13d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:04.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.131+0000 7f3bdbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc0c4fa0 0x7f3bdc13d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50274/0 (socket says 172.21.15.102:50274) 2024-03-18T16:28:04.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.132+0000 7f3bdbe00640 1 -- 172.21.15.102:0/1722789829 learned_addr learned my addr 172.21.15.102:0/1722789829 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:04.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.132+0000 7f3be1000640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc15c3a0 0x7f3bdc1e7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:04.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.132+0000 7f3bdb400640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bdc141650 0x7f3bdc1e5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.136+0000 7f3bdbe00640 1 -- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bdc141650 msgr2=0x7f3bdc1e5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.136+0000 7f3bdbe00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bdc141650 0x7f3bdc1e5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.136+0000 7f3bdbe00640 1 -- 172.21.15.102:0/1722789829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc15c3a0 msgr2=0x7f3bdc1e7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.136+0000 7f3bdbe00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc15c3a0 0x7f3bdc1e7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:04.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.136+0000 7f3bdbe00640 1 -- 172.21.15.102:0/1722789829 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bdc1e4f40 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.139+0000 7f3bdbe00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc0c4fa0 0x7f3bdc13d6a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009990 tx=0x7f3bcc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:04.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.142+0000 7f3bd3e00640 1 -- 172.21.15.102:0/1722789829 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc048070 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.142+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bdc1e6bc0 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.142+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bdc06fe80 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.143+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bdc074fd0 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.145+0000 7f3bd3e00640 1 -- 172.21.15.102:0/1722789829 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bcc039d90 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.145+0000 7f3bd3e00640 1 -- 172.21.15.102:0/1722789829 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc052060 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.405+0000 7f3bd3e00640 1 -- 172.21.15.102:0/1722789829 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3bcc04b070 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.407+0000 7f3bd3e00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bbc06bc00 0x7f3bbc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:04.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.408+0000 7f3bd3e00640 1 -- 172.21.15.102:0/1722789829 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3bcc0ad300 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.408+0000 7f3bdb400640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bbc06bc00 0x7f3bbc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:04.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.408+0000 7f3bdb400640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bbc06bc00 0x7f3bbc06e0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc141060 tx=0x7f3bc8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:04.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.418+0000 7f3bd3e00640 1 -- 172.21.15.102:0/1722789829 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3bcc07b3d0 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.657+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bdc1e3980 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.670+0000 7f3bd3e00640 1 -- 172.21.15.102:0/1722789829 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3bcc07f470 con 0x7f3bdc0c4fa0 2024-03-18T16:28:04.672 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:04.672 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.673+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bbc06bc00 msgr2=0x7f3bbc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.673+0000 7f3be1a00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bbc06bc00 0x7f3bbc06e0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc141060 tx=0x7f3bc8009040 comp rx=0 tx=0).stop 2024-03-18T16:28:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.673+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc0c4fa0 msgr2=0x7f3bdc13d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.673+0000 7f3be1a00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc0c4fa0 0x7f3bdc13d6a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009990 tx=0x7f3bcc004730 comp rx=0 tx=0).stop 2024-03-18T16:28:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.674+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 shutdown_connections 2024-03-18T16:28:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.674+0000 7f3be1a00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bbc06bc00 0x7f3bbc06e0c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.674+0000 7f3be1a00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bdc15c3a0 0x7f3bdc1e7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.674+0000 7f3be1a00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bdc141650 0x7f3bdc1e5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.674+0000 7f3be1a00640 1 --2- 172.21.15.102:0/1722789829 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bdc0c4fa0 0x7f3bdc13d6a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.674+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 >> 172.21.15.102:0/1722789829 conn(0x7f3bdc0cb890 msgr2=0x7f3bdc072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.674+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 shutdown_connections 2024-03-18T16:28:04.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:04.674+0000 7f3be1a00640 1 -- 172.21.15.102:0/1722789829 wait complete. 2024-03-18T16:28:05.304 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:05.304 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:06.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:06.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.479+0000 7f9384e00640 1 Processor -- start 2024-03-18T16:28:06.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.480+0000 7f9384e00640 1 -- start start 2024-03-18T16:28:06.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.481+0000 7f9384e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f938015a310 0x7f938006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:06.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.481+0000 7f9384e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f938015a6f0 0x7f938015ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:06.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.481+0000 7f9384e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93800ca500 con 0x7f938015baa0 2024-03-18T16:28:06.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.481+0000 7f9384e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f93800cd410 con 0x7f938015a310 2024-03-18T16:28:06.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.481+0000 7f9384e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f938015b080 con 0x7f938015a6f0 2024-03-18T16:28:06.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.482+0000 7f937ea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f938015a6f0 0x7f938015ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:06.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.482+0000 7f937f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f938015a310 0x7f938006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:06.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.482+0000 7f937ea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f938015a6f0 0x7f938015ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40924/0 (socket says 172.21.15.102:40924) 2024-03-18T16:28:06.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.482+0000 7f937ea00640 1 -- 172.21.15.102:0/1522610926 learned_addr learned my addr 172.21.15.102:0/1522610926 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:06.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.482+0000 7f937fe00640 1 --1- 172.21.15.102:0/1522610926 >> v1:172.21.15.102:6789/0 conn(0x7f938015baa0 0x7f938015bef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54602/0 (socket says 172.21.15.102:54602) 2024-03-18T16:28:06.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.485+0000 7f937ea00640 1 -- 172.21.15.102:0/1522610926 >> v2:172.21.15.102:3300/0 conn(0x7f938015a310 msgr2=0x7f938006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:06.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.485+0000 7f937ea00640 1 --2- 172.21.15.102:0/1522610926 >> v2:172.21.15.102:3300/0 conn(0x7f938015a310 0x7f938006d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:06.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.485+0000 7f937ea00640 1 -- 172.21.15.102:0/1522610926 >> v1:172.21.15.102:6789/0 conn(0x7f938015baa0 legacy=0x7f938015bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:06.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.485+0000 7f937ea00640 1 -- 172.21.15.102:0/1522610926 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93800ca7a0 con 0x7f938015a6f0 2024-03-18T16:28:06.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.489+0000 7f937ea00640 1 --2- 172.21.15.102:0/1522610926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 0x7f938015ab40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f93800d0c80 tx=0x7f936c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c43f030cf8ccf66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:06.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.491+0000 7f937e000640 1 -- 172.21.15.102:0/1522610926 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c034070 con 0x7f938015a6f0 2024-03-18T16:28:06.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.494+0000 7f937e000640 1 -- 172.21.15.102:0/1522610926 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f936c030860 con 0x7f938015a6f0 2024-03-18T16:28:06.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.494+0000 7f937e000640 1 -- 172.21.15.102:0/1522610926 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c030c30 con 0x7f938015a6f0 2024-03-18T16:28:06.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.494+0000 7f9384e00640 1 -- 172.21.15.102:0/1522610926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 msgr2=0x7f938015ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:06.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.494+0000 7f9384e00640 1 --2- 172.21.15.102:0/1522610926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 0x7f938015ab40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f93800d0c80 tx=0x7f936c02b9a0 comp rx=0 tx=0).stop 2024-03-18T16:28:06.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.494+0000 7f9384e00640 1 -- 172.21.15.102:0/1522610926 shutdown_connections 2024-03-18T16:28:06.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.494+0000 7f9384e00640 1 --2- 172.21.15.102:0/1522610926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 0x7f938015ab40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:06.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.494+0000 7f9384e00640 1 --2- 172.21.15.102:0/1522610926 >> v2:172.21.15.102:3300/0 conn(0x7f938015a310 0x7f938006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:06.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.494+0000 7f9384e00640 1 -- 172.21.15.102:0/1522610926 >> 172.21.15.102:0/1522610926 conn(0x7f938008c1a0 msgr2=0x7f9380066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:06.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.495+0000 7f9384e00640 1 -- 172.21.15.102:0/1522610926 shutdown_connections 2024-03-18T16:28:06.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.495+0000 7f9384e00640 1 -- 172.21.15.102:0/1522610926 wait complete. 2024-03-18T16:28:06.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.497+0000 7f9384e00640 1 Processor -- start 2024-03-18T16:28:06.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.497+0000 7f9384e00640 1 -- start start 2024-03-18T16:28:06.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.499+0000 7f9384e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f938015a310 0x7f938014e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:06.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.499+0000 7f9384e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 0x7f93801f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:06.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.499+0000 7f9384e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938015baa0 0x7f93801e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:06.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.499+0000 7f9384e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f93800cd130 con 0x7f938015baa0 2024-03-18T16:28:06.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.499+0000 7f9384e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f93800cce50 con 0x7f938015a6f0 2024-03-18T16:28:06.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.499+0000 7f9384e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f938006d920 con 0x7f938015a310 2024-03-18T16:28:06.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.499+0000 7f937ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 0x7f93801f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:06.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.500+0000 7f937f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f938015a310 0x7f938014e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:06.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.500+0000 7f937fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938015baa0 0x7f93801e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:06.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.500+0000 7f937ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 0x7f93801f41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40940/0 (socket says 172.21.15.102:40940) 2024-03-18T16:28:06.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.500+0000 7f937ea00640 1 -- 172.21.15.102:0/2081414520 learned_addr learned my addr 172.21.15.102:0/2081414520 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:06.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.503+0000 7f937ea00640 1 -- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f938015a310 msgr2=0x7f938014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:06.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.503+0000 7f937ea00640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f938015a310 0x7f938014e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:06.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.503+0000 7f937ea00640 1 -- 172.21.15.102:0/2081414520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938015baa0 msgr2=0x7f93801e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:06.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.503+0000 7f937ea00640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938015baa0 0x7f93801e70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:06.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.503+0000 7f937ea00640 1 -- 172.21.15.102:0/2081414520 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93801f4730 con 0x7f938015a6f0 2024-03-18T16:28:06.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.506+0000 7f937ea00640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 0x7f93801f41f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f936c02b9e0 tx=0x7f936c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:06.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.510+0000 7f9377400640 1 -- 172.21.15.102:0/2081414520 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c048070 con 0x7f938015a6f0 2024-03-18T16:28:06.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.510+0000 7f9377400640 1 -- 172.21.15.102:0/2081414520 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f936c030660 con 0x7f938015a6f0 2024-03-18T16:28:06.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.510+0000 7f9377400640 1 -- 172.21.15.102:0/2081414520 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c034040 con 0x7f938015a6f0 2024-03-18T16:28:06.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.510+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93800c7020 con 0x7f938015a6f0 2024-03-18T16:28:06.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.510+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93801f39f0 con 0x7f938015a6f0 2024-03-18T16:28:06.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.511+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9380150e60 con 0x7f938015a6f0 2024-03-18T16:28:06.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.531+0000 7f9377400640 1 -- 172.21.15.102:0/2081414520 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f936c04b070 con 0x7f938015a6f0 2024-03-18T16:28:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.531+0000 7f9377400640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f936006bc20 0x7f936006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:06.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.532+0000 7f937f400640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f936006bc20 0x7f936006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:06.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.533+0000 7f937f400640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f936006bc20 0x7f936006e0e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9370009b40 tx=0x7f9370009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:06.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.533+0000 7f9377400640 1 -- 172.21.15.102:0/2081414520 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f936c0793c0 con 0x7f938015a6f0 2024-03-18T16:28:06.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:06.783+0000 7f9377400640 1 -- 172.21.15.102:0/2081414520 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9380150e60 con 0x7f938015a6f0 2024-03-18T16:28:07.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.039+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9380074fd0 con 0x7f938015a6f0 2024-03-18T16:28:07.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.168+0000 7f9377400640 1 -- 172.21.15.102:0/2081414520 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f936c052020 con 0x7f938015a6f0 2024-03-18T16:28:07.170 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:07.170 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:07.170 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:07.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.171+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f936006bc20 msgr2=0x7f936006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.171+0000 7f9384e00640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f936006bc20 0x7f936006e0e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9370009b40 tx=0x7f9370009190 comp rx=0 tx=0).stop 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.171+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 msgr2=0x7f93801f41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.171+0000 7f9384e00640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 0x7f93801f41f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f936c02b9e0 tx=0x7f936c004730 comp rx=0 tx=0).stop 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.171+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 shutdown_connections 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.171+0000 7f9384e00640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f936006bc20 0x7f936006e0e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.171+0000 7f9384e00640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f938015baa0 0x7f93801e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.171+0000 7f9384e00640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f938015a6f0 0x7f93801f41f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.171+0000 7f9384e00640 1 --2- 172.21.15.102:0/2081414520 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f938015a310 0x7f938014e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.172+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 >> 172.21.15.102:0/2081414520 conn(0x7f938008c1a0 msgr2=0x7f9380066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.172+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 shutdown_connections 2024-03-18T16:28:07.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:07.172+0000 7f9384e00640 1 -- 172.21.15.102:0/2081414520 wait complete. 2024-03-18T16:28:07.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:07.803 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:08.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:08.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.979+0000 7fb992800640 1 Processor -- start 2024-03-18T16:28:08.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.979+0000 7fb992800640 1 -- start start 2024-03-18T16:28:08.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.980+0000 7fb992800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:08.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.980+0000 7fb992800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb98c0c3cd0 0x7fb98c0c9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:08.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.980+0000 7fb992800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb98c15a170 con 0x7fb98c0c3890 2024-03-18T16:28:08.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.980+0000 7fb992800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb98c154900 con 0x7fb98c0c3cd0 2024-03-18T16:28:08.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.980+0000 7fb992800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb98c0c4160 con 0x7fb98c154520 2024-03-18T16:28:08.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.981+0000 7fb991400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:08.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.981+0000 7fb991e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb98c0c3cd0 0x7fb98c0c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:08.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.981+0000 7fb991400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c0c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50286/0 (socket says 172.21.15.102:50286) 2024-03-18T16:28:08.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.981+0000 7fb991400640 1 -- 172.21.15.102:0/501679318 learned_addr learned my addr 172.21.15.102:0/501679318 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:08.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.984+0000 7fb991400640 1 -- 172.21.15.102:0/501679318 >> v2:172.21.15.154:3300/0 conn(0x7fb98c0c3cd0 msgr2=0x7fb98c0c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:08.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.984+0000 7fb991400640 1 --2- 172.21.15.102:0/501679318 >> v2:172.21.15.154:3300/0 conn(0x7fb98c0c3cd0 0x7fb98c0c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:08.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.984+0000 7fb991400640 1 -- 172.21.15.102:0/501679318 >> v1:172.21.15.154:6789/0 conn(0x7fb98c0c3890 legacy=0x7fb98c0c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:08.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.984+0000 7fb991400640 1 -- 172.21.15.102:0/501679318 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98c0c4440 con 0x7fb98c154520 2024-03-18T16:28:08.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.987+0000 7fb991400640 1 --2- 172.21.15.102:0/501679318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c0c9eb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb980009870 tx=0x7fb98002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69decc0d9cd3993e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:08.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.991+0000 7fb98be00640 1 -- 172.21.15.102:0/501679318 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980034070 con 0x7fb98c154520 2024-03-18T16:28:08.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.991+0000 7fb98be00640 1 -- 172.21.15.102:0/501679318 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb980030840 con 0x7fb98c154520 2024-03-18T16:28:08.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.991+0000 7fb98be00640 1 -- 172.21.15.102:0/501679318 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980030c10 con 0x7fb98c154520 2024-03-18T16:28:08.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.992+0000 7fb992800640 1 -- 172.21.15.102:0/501679318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 msgr2=0x7fb98c0c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:08.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.992+0000 7fb992800640 1 --2- 172.21.15.102:0/501679318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c0c9eb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb980009870 tx=0x7fb98002b980 comp rx=0 tx=0).stop 2024-03-18T16:28:08.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.992+0000 7fb992800640 1 -- 172.21.15.102:0/501679318 shutdown_connections 2024-03-18T16:28:08.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.992+0000 7fb992800640 1 --2- 172.21.15.102:0/501679318 >> v2:172.21.15.154:3300/0 conn(0x7fb98c0c3cd0 0x7fb98c0c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:08.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.992+0000 7fb992800640 1 --2- 172.21.15.102:0/501679318 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c0c9eb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:08.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.992+0000 7fb992800640 1 -- 172.21.15.102:0/501679318 >> 172.21.15.102:0/501679318 conn(0x7fb98c066f80 msgr2=0x7fb98c067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:08.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.992+0000 7fb992800640 1 -- 172.21.15.102:0/501679318 shutdown_connections 2024-03-18T16:28:08.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.992+0000 7fb992800640 1 -- 172.21.15.102:0/501679318 wait complete. 2024-03-18T16:28:08.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.994+0000 7fb992800640 1 Processor -- start 2024-03-18T16:28:08.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.995+0000 7fb992800640 1 -- start start 2024-03-18T16:28:08.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.996+0000 7fb992800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0c3890 0x7fb98c159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:08.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.996+0000 7fb992800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:08.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.996+0000 7fb992800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb98c153280 0x7fb98c1efda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:08.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.996+0000 7fb992800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb98c15a9f0 con 0x7fb98c0c3890 2024-03-18T16:28:08.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.996+0000 7fb992800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb98c15a710 con 0x7fb98c153280 2024-03-18T16:28:08.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.996+0000 7fb992800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb98c0ca3f0 con 0x7fb98c154520 2024-03-18T16:28:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.997+0000 7fb990a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.997+0000 7fb990a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50302/0 (socket says 172.21.15.102:50302) 2024-03-18T16:28:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.997+0000 7fb990a00640 1 -- 172.21.15.102:0/3270136567 learned_addr learned my addr 172.21.15.102:0/3270136567 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.997+0000 7fb991400640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0c3890 0x7fb98c159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:08.997+0000 7fb991e00640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb98c153280 0x7fb98c1efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.000+0000 7fb990a00640 1 -- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb98c153280 msgr2=0x7fb98c1efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.000+0000 7fb990a00640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb98c153280 0x7fb98c1efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.000+0000 7fb990a00640 1 -- 172.21.15.102:0/3270136567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0c3890 msgr2=0x7fb98c159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.000+0000 7fb990a00640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0c3890 0x7fb98c159830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.000+0000 7fb990a00640 1 -- 172.21.15.102:0/3270136567 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98c153790 con 0x7fb98c154520 2024-03-18T16:28:09.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.003+0000 7fb990a00640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c152d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb97800c460 tx=0x7fb97800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.006+0000 7fb989600640 1 -- 172.21.15.102:0/3270136567 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978016020 con 0x7fb98c154520 2024-03-18T16:28:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.006+0000 7fb989600640 1 -- 172.21.15.102:0/3270136567 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9780052f0 con 0x7fb98c154520 2024-03-18T16:28:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.006+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb98c1f32b0 con 0x7fb98c154520 2024-03-18T16:28:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.006+0000 7fb989600640 1 -- 172.21.15.102:0/3270136567 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978007260 con 0x7fb98c154520 2024-03-18T16:28:09.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.006+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb98c1f3870 con 0x7fb98c154520 2024-03-18T16:28:09.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.008+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb98c157fe0 con 0x7fb98c154520 2024-03-18T16:28:09.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.028+0000 7fb989600640 1 -- 172.21.15.102:0/3270136567 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb978007400 con 0x7fb98c154520 2024-03-18T16:28:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.029+0000 7fb989600640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb96c06bc50 0x7fb96c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.030+0000 7fb991400640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb96c06bc50 0x7fb96c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:09.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.031+0000 7fb991400640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb96c06bc50 0x7fb96c06e110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb980036000 tx=0x7fb98003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:09.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.270+0000 7fb989600640 1 -- 172.21.15.102:0/3270136567 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb978057670 con 0x7fb98c154520 2024-03-18T16:28:09.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.280+0000 7fb989600640 1 -- 172.21.15.102:0/3270136567 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb98c157fe0 con 0x7fb98c154520 2024-03-18T16:28:09.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.537+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb98c0c9150 con 0x7fb98c154520 2024-03-18T16:28:09.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.572+0000 7fb989600640 1 -- 172.21.15.102:0/3270136567 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb978020020 con 0x7fb98c154520 2024-03-18T16:28:09.574 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:09.574 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:09.575 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:09.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb96c06bc50 msgr2=0x7fb96c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb96c06bc50 0x7fb96c06e110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb980036000 tx=0x7fb98003b040 comp rx=0 tx=0).stop 2024-03-18T16:28:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 msgr2=0x7fb98c152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c152d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb97800c460 tx=0x7fb97800c920 comp rx=0 tx=0).stop 2024-03-18T16:28:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 shutdown_connections 2024-03-18T16:28:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb96c06bc50 0x7fb96c06e110 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb98c153280 0x7fb98c1efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb98c154520 0x7fb98c152d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 --2- 172.21.15.102:0/3270136567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0c3890 0x7fb98c159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:09.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 >> 172.21.15.102:0/3270136567 conn(0x7fb98c066f80 msgr2=0x7fb98c158380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:09.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 shutdown_connections 2024-03-18T16:28:09.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:09.575+0000 7fb992800640 1 -- 172.21.15.102:0/3270136567 wait complete. 2024-03-18T16:28:10.199 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:10.199 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:11.200 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:11.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.373+0000 7f3a9be00640 1 Processor -- start 2024-03-18T16:28:11.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.373+0000 7f3a9be00640 1 -- start start 2024-03-18T16:28:11.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.374+0000 7f3a9be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3a9c141650 0x7f3a9c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:11.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.374+0000 7f3a9be00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3a9c15c3a0 0x7f3a9c0c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:11.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.374+0000 7f3a9be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c0c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:11.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.374+0000 7f3a9be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3a9c159370 con 0x7f3a9c141650 2024-03-18T16:28:11.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.374+0000 7f3a9be00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3a9c158ad0 con 0x7f3a9c15c3a0 2024-03-18T16:28:11.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.374+0000 7f3a9be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9c158db0 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.375+0000 7f3a9b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c0c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:11.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.375+0000 7f3a9a000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3a9c15c3a0 0x7f3a9c0c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:11.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.375+0000 7f3a9aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3a9c141650 0x7f3a9c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:11.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.375+0000 7f3a9b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c0c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50268/0 (socket says 172.21.15.102:50268) 2024-03-18T16:28:11.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.375+0000 7f3a9b400640 1 -- 172.21.15.102:0/3516617054 learned_addr learned my addr 172.21.15.102:0/3516617054 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:11.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.378+0000 7f3a9b400640 1 -- 172.21.15.102:0/3516617054 >> v2:172.21.15.154:3300/0 conn(0x7f3a9c15c3a0 msgr2=0x7f3a9c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:11.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.378+0000 7f3a9b400640 1 --2- 172.21.15.102:0/3516617054 >> v2:172.21.15.154:3300/0 conn(0x7f3a9c15c3a0 0x7f3a9c0c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.378+0000 7f3a9b400640 1 -- 172.21.15.102:0/3516617054 >> v2:172.21.15.102:3300/0 conn(0x7f3a9c141650 msgr2=0x7f3a9c158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:11.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.378+0000 7f3a9b400640 1 --2- 172.21.15.102:0/3516617054 >> v2:172.21.15.102:3300/0 conn(0x7f3a9c141650 0x7f3a9c158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.378+0000 7f3a9b400640 1 -- 172.21.15.102:0/3516617054 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c159090 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.378+0000 7f3a9aa00640 1 --2- 172.21.15.102:0/3516617054 >> v2:172.21.15.102:3300/0 conn(0x7f3a9c141650 0x7f3a9c158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:28:11.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.381+0000 7f3a9b400640 1 --2- 172.21.15.102:0/3516617054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c0c5430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c068e80 tx=0x7f3a9402bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b88c1639090b65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.384+0000 7f3a99600640 1 -- 172.21.15.102:0/3516617054 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9402ca80 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.386+0000 7f3a99600640 1 -- 172.21.15.102:0/3516617054 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9402cc20 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.386+0000 7f3a99600640 1 -- 172.21.15.102:0/3516617054 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a940343f0 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.386+0000 7f3a9be00640 1 -- 172.21.15.102:0/3516617054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 msgr2=0x7f3a9c0c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:11.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.386+0000 7f3a9be00640 1 --2- 172.21.15.102:0/3516617054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c0c5430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c068e80 tx=0x7f3a9402bee0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.387+0000 7f3a9be00640 1 -- 172.21.15.102:0/3516617054 shutdown_connections 2024-03-18T16:28:11.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.387+0000 7f3a9be00640 1 --2- 172.21.15.102:0/3516617054 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c0c5430 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.387+0000 7f3a9be00640 1 --2- 172.21.15.102:0/3516617054 >> v2:172.21.15.154:3300/0 conn(0x7f3a9c15c3a0 0x7f3a9c0c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.387+0000 7f3a9be00640 1 --2- 172.21.15.102:0/3516617054 >> v2:172.21.15.102:3300/0 conn(0x7f3a9c141650 0x7f3a9c158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.387+0000 7f3a9be00640 1 -- 172.21.15.102:0/3516617054 >> 172.21.15.102:0/3516617054 conn(0x7f3a9c0cb890 msgr2=0x7f3a9c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:11.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.387+0000 7f3a9be00640 1 -- 172.21.15.102:0/3516617054 shutdown_connections 2024-03-18T16:28:11.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.387+0000 7f3a9be00640 1 -- 172.21.15.102:0/3516617054 wait complete. 2024-03-18T16:28:11.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.388+0000 7f3a9be00640 1 Processor -- start 2024-03-18T16:28:11.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.389+0000 7f3a9be00640 1 -- start start 2024-03-18T16:28:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.389+0000 7f3a9be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.389+0000 7f3a9be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a9c141650 0x7f3a9c13ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.389+0000 7f3a9be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c15c3a0 0x7f3a9c152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.389+0000 7f3a9be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c15c7c0 con 0x7f3a9c141650 2024-03-18T16:28:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.389+0000 7f3a9be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c0c66a0 con 0x7f3a9c15c3a0 2024-03-18T16:28:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.389+0000 7f3a9be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9c141a30 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.390+0000 7f3a9aa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:11.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.390+0000 7f3a9b400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c15c3a0 0x7f3a9c152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:11.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.390+0000 7f3a9aa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c1540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50282/0 (socket says 172.21.15.102:50282) 2024-03-18T16:28:11.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.390+0000 7f3a9aa00640 1 -- 172.21.15.102:0/1841258060 learned_addr learned my addr 172.21.15.102:0/1841258060 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:11.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.390+0000 7f3a9a000640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a9c141650 0x7f3a9c13ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:11.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.395+0000 7f3a9aa00640 1 -- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c15c3a0 msgr2=0x7f3a9c152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:11.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.395+0000 7f3a9aa00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c15c3a0 0x7f3a9c152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.395+0000 7f3a9aa00640 1 -- 172.21.15.102:0/1841258060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a9c141650 msgr2=0x7f3a9c13ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:11.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.395+0000 7f3a9aa00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a9c141650 0x7f3a9c13ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.395+0000 7f3a9aa00640 1 -- 172.21.15.102:0/1841258060 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c140800 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.398+0000 7f3a9aa00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c1540f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00e910 tx=0x7f3a8c00edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:11.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.401+0000 7f3a92a00640 1 -- 172.21.15.102:0/1841258060 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c017070 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.401+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9c1f3640 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.401+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9c1f3c00 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.402+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a9c074fd0 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.406+0000 7f3a92a00640 1 -- 172.21.15.102:0/1841258060 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a8c004020 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.406+0000 7f3a92a00640 1 -- 172.21.15.102:0/1841258060 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c021060 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.676+0000 7f3a92a00640 1 -- 172.21.15.102:0/1841258060 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3a8c01a070 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.678+0000 7f3a92a00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a7c06bc00 0x7f3a7c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.679+0000 7f3a92a00640 1 -- 172.21.15.102:0/1841258060 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3a8c08af80 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.679+0000 7f3a9a000640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a7c06bc00 0x7f3a7c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:11.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.679+0000 7f3a9a000640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a7c06bc00 0x7f3a7c06e0c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0ca5d0 tx=0x7f3a88009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:11.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.688+0000 7f3a92a00640 1 -- 172.21.15.102:0/1841258060 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3a8c0590a0 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.975+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a9c0b7f00 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.987+0000 7f3a92a00640 1 -- 172.21.15.102:0/1841258060 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3a8c05d140 con 0x7f3a9c0c4fa0 2024-03-18T16:28:11.990 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:11.990 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:11.990 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:11.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a7c06bc00 msgr2=0x7f3a7c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:11.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a7c06bc00 0x7f3a7c06e0c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0ca5d0 tx=0x7f3a88009190 comp rx=0 tx=0).stop 2024-03-18T16:28:11.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 msgr2=0x7f3a9c1540f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:11.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c1540f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00e910 tx=0x7f3a8c00edd0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 shutdown_connections 2024-03-18T16:28:11.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3a7c06bc00 0x7f3a7c06e0c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a9c15c3a0 0x7f3a9c152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3a9c141650 0x7f3a9c13ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 --2- 172.21.15.102:0/1841258060 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3a9c0c4fa0 0x7f3a9c1540f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:11.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 >> 172.21.15.102:0/1841258060 conn(0x7f3a9c0cb890 msgr2=0x7f3a9c13ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:11.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 shutdown_connections 2024-03-18T16:28:11.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:11.990+0000 7f3a9be00640 1 -- 172.21.15.102:0/1841258060 wait complete. 2024-03-18T16:28:12.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:12.620 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:13.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:13.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.792+0000 7fba06c00640 1 Processor -- start 2024-03-18T16:28:13.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.792+0000 7fba06c00640 1 -- start start 2024-03-18T16:28:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.793+0000 7fba06c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fba00141650 0x7fba00158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.793+0000 7fba06c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba0015c3a0 0x7fba000c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:13.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.794+0000 7fba06c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba00158ad0 con 0x7fba000c4fa0 2024-03-18T16:28:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.794+0000 7fba06c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fba00159370 con 0x7fba00141650 2024-03-18T16:28:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.794+0000 7fba06c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fba00158db0 con 0x7fba0015c3a0 2024-03-18T16:28:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.794+0000 7fba04e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba0015c3a0 0x7fba000c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.794+0000 7fba05800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fba00141650 0x7fba00158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:13.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.795+0000 7fba06200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fba000c4fa0 0x7fba0006f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49284/0 (socket says 172.21.15.102:49284) 2024-03-18T16:28:13.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.795+0000 7fba06200640 1 -- 172.21.15.102:0/2821922061 learned_addr learned my addr 172.21.15.102:0/2821922061 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:13.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.798+0000 7fba04e00640 1 -- 172.21.15.102:0/2821922061 >> v2:172.21.15.102:3300/0 conn(0x7fba00141650 msgr2=0x7fba00158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:13.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.798+0000 7fba04e00640 1 --2- 172.21.15.102:0/2821922061 >> v2:172.21.15.102:3300/0 conn(0x7fba00141650 0x7fba00158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:13.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.798+0000 7fba04e00640 1 -- 172.21.15.102:0/2821922061 >> v1:172.21.15.102:6789/0 conn(0x7fba000c4fa0 legacy=0x7fba0006f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:13.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.798+0000 7fba04e00640 1 -- 172.21.15.102:0/2821922061 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba00159090 con 0x7fba0015c3a0 2024-03-18T16:28:13.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.801+0000 7fba04e00640 1 --2- 172.21.15.102:0/2821922061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba0015c3a0 0x7fba000c6bc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec0097a0 tx=0x7fb9ec02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94f253790ec5704d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:13.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.805+0000 7fb9ffe00640 1 -- 172.21.15.102:0/2821922061 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec035020 con 0x7fba0015c3a0 2024-03-18T16:28:13.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.805+0000 7fb9ffe00640 1 -- 172.21.15.102:0/2821922061 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9ec031810 con 0x7fba0015c3a0 2024-03-18T16:28:13.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.805+0000 7fb9ffe00640 1 -- 172.21.15.102:0/2821922061 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec031be0 con 0x7fba0015c3a0 2024-03-18T16:28:13.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.806+0000 7fba06c00640 1 -- 172.21.15.102:0/2821922061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba0015c3a0 msgr2=0x7fba000c6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:13.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.806+0000 7fba06c00640 1 --2- 172.21.15.102:0/2821922061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba0015c3a0 0x7fba000c6bc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec0097a0 tx=0x7fb9ec02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:28:13.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.806+0000 7fba06c00640 1 -- 172.21.15.102:0/2821922061 shutdown_connections 2024-03-18T16:28:13.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.806+0000 7fba06c00640 1 --2- 172.21.15.102:0/2821922061 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba0015c3a0 0x7fba000c6bc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:13.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.806+0000 7fba06c00640 1 --2- 172.21.15.102:0/2821922061 >> v2:172.21.15.102:3300/0 conn(0x7fba00141650 0x7fba00158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:13.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.806+0000 7fba06c00640 1 -- 172.21.15.102:0/2821922061 >> 172.21.15.102:0/2821922061 conn(0x7fba000cb890 msgr2=0x7fba000cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:13.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.806+0000 7fba06c00640 1 -- 172.21.15.102:0/2821922061 shutdown_connections 2024-03-18T16:28:13.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.806+0000 7fba06c00640 1 -- 172.21.15.102:0/2821922061 wait complete. 2024-03-18T16:28:13.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.808+0000 7fba06c00640 1 Processor -- start 2024-03-18T16:28:13.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.808+0000 7fba06c00640 1 -- start start 2024-03-18T16:28:13.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.809+0000 7fba06c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba000c4fa0 0x7fba00157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:13.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.809+0000 7fba06c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba00141650 0x7fba00152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:13.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.809+0000 7fba06c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0015c3a0 0x7fba000caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:13.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.809+0000 7fba06c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fba0015c7c0 con 0x7fba00141650 2024-03-18T16:28:13.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.809+0000 7fba06c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fba000c66a0 con 0x7fba0015c3a0 2024-03-18T16:28:13.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.809+0000 7fba06c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fba00141a30 con 0x7fba000c4fa0 2024-03-18T16:28:13.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.810+0000 7fba05800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba000c4fa0 0x7fba00157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:13.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.810+0000 7fba04e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba00141650 0x7fba00152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:13.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.810+0000 7fba05800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba000c4fa0 0x7fba00157b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50304/0 (socket says 172.21.15.102:50304) 2024-03-18T16:28:13.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.810+0000 7fba05800640 1 -- 172.21.15.102:0/94303197 learned_addr learned my addr 172.21.15.102:0/94303197 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:13.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.810+0000 7fba06200640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0015c3a0 0x7fba000caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:13.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.813+0000 7fba05800640 1 -- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0015c3a0 msgr2=0x7fba000caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:13.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.813+0000 7fba05800640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0015c3a0 0x7fba000caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:13.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.813+0000 7fba05800640 1 -- 172.21.15.102:0/94303197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba00141650 msgr2=0x7fba00152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:13.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.813+0000 7fba05800640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba00141650 0x7fba00152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:13.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.813+0000 7fba05800640 1 -- 172.21.15.102:0/94303197 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba001580c0 con 0x7fba000c4fa0 2024-03-18T16:28:13.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.816+0000 7fba05800640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba000c4fa0 0x7fba00157b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400e9e0 tx=0x7fb9f400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.820+0000 7fb9fd600640 1 -- 172.21.15.102:0/94303197 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4017070 con 0x7fba000c4fa0 2024-03-18T16:28:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.820+0000 7fb9fd600640 1 -- 172.21.15.102:0/94303197 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f400c9d0 con 0x7fba000c4fa0 2024-03-18T16:28:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.820+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba001ef340 con 0x7fba000c4fa0 2024-03-18T16:28:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.820+0000 7fb9fd600640 1 -- 172.21.15.102:0/94303197 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f400cdc0 con 0x7fba000c4fa0 2024-03-18T16:28:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.820+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fba00154f40 con 0x7fba000c4fa0 2024-03-18T16:28:13.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.822+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba00074fd0 con 0x7fba000c4fa0 2024-03-18T16:28:13.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.853+0000 7fb9fd600640 1 -- 172.21.15.102:0/94303197 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb9f401a070 con 0x7fba000c4fa0 2024-03-18T16:28:13.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.854+0000 7fb9fd600640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb9e006bc50 0x7fb9e006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:13.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.854+0000 7fba04e00640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb9e006bc50 0x7fb9e006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:13.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.855+0000 7fba04e00640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb9e006bc50 0x7fb9e006e110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec036000 tx=0x7fb9ec03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:13.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:13.858+0000 7fb9fd600640 1 -- 172.21.15.102:0/94303197 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb9f40576d0 con 0x7fba000c4fa0 2024-03-18T16:28:14.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.117+0000 7fb9fd600640 1 -- 172.21.15.102:0/94303197 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb9f4059630 con 0x7fba000c4fa0 2024-03-18T16:28:14.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.406+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba00140370 con 0x7fba000c4fa0 2024-03-18T16:28:14.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.422+0000 7fb9fd600640 1 -- 172.21.15.102:0/94303197 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb9f405d6d0 con 0x7fba000c4fa0 2024-03-18T16:28:14.424 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:14.424 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:14.424 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:14.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb9e006bc50 msgr2=0x7fb9e006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:14.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb9e006bc50 0x7fb9e006e110 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec036000 tx=0x7fb9ec03c040 comp rx=0 tx=0).stop 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba000c4fa0 msgr2=0x7fba00157b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba000c4fa0 0x7fba00157b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400e9e0 tx=0x7fb9f400eea0 comp rx=0 tx=0).stop 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 shutdown_connections 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb9e006bc50 0x7fb9e006e110 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0015c3a0 0x7fba000caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba00141650 0x7fba00152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 --2- 172.21.15.102:0/94303197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fba000c4fa0 0x7fba00157b80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.424+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 >> 172.21.15.102:0/94303197 conn(0x7fba000cb890 msgr2=0x7fba000cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.425+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 shutdown_connections 2024-03-18T16:28:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:14.425+0000 7fba06c00640 1 -- 172.21.15.102:0/94303197 wait complete. 2024-03-18T16:28:15.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:15.048 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:16.049 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:16.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.216+0000 7f33cea00640 1 Processor -- start 2024-03-18T16:28:16.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.216+0000 7f33cea00640 1 -- start start 2024-03-18T16:28:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.217+0000 7f33cea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f33c806de50 0x7f33c815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.217+0000 7f33cea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33c8154150 0x7f33c8154580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.217+0000 7f33cea00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c806bf60 con 0x7f33c806b470 2024-03-18T16:28:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.217+0000 7f33cea00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f33c80ca0a0 con 0x7f33c8154150 2024-03-18T16:28:16.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.217+0000 7f33cea00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f33c80c94b0 con 0x7f33c806de50 2024-03-18T16:28:16.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.218+0000 7f33ccc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f33c806de50 0x7f33c815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:16.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.219+0000 7f33ccc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f33c806de50 0x7f33c815ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:58988/0 (socket says 172.21.15.102:58988) 2024-03-18T16:28:16.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.219+0000 7f33ccc00640 1 -- 172.21.15.102:0/3336939759 learned_addr learned my addr 172.21.15.102:0/3336939759 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:16.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.219+0000 7f33ce000640 1 --2- 172.21.15.102:0/3336939759 >> v2:172.21.15.102:3300/0 conn(0x7f33c8154150 0x7f33c8154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:16.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.222+0000 7f33ccc00640 1 -- 172.21.15.102:0/3336939759 >> v2:172.21.15.102:3300/0 conn(0x7f33c8154150 msgr2=0x7f33c8154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:16.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.222+0000 7f33ccc00640 1 --2- 172.21.15.102:0/3336939759 >> v2:172.21.15.102:3300/0 conn(0x7f33c8154150 0x7f33c8154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.222+0000 7f33ccc00640 1 -- 172.21.15.102:0/3336939759 >> v1:172.21.15.102:6789/0 conn(0x7f33c806b470 legacy=0x7f33c806b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:16.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.222+0000 7f33ccc00640 1 -- 172.21.15.102:0/3336939759 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c80c9770 con 0x7f33c806de50 2024-03-18T16:28:16.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.225+0000 7f33ccc00640 1 --2- 172.21.15.102:0/3336939759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806de50 0x7f33c815ad60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f33c806c1e0 tx=0x7f33b402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d43756c3a968892e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:16.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.229+0000 7f33c7e00640 1 -- 172.21.15.102:0/3336939759 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4034070 con 0x7f33c806de50 2024-03-18T16:28:16.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.229+0000 7f33c7e00640 1 -- 172.21.15.102:0/3336939759 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33b4030860 con 0x7f33c806de50 2024-03-18T16:28:16.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.229+0000 7f33c7e00640 1 -- 172.21.15.102:0/3336939759 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4030c30 con 0x7f33c806de50 2024-03-18T16:28:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.230+0000 7f33cea00640 1 -- 172.21.15.102:0/3336939759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806de50 msgr2=0x7f33c815ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.230+0000 7f33cea00640 1 --2- 172.21.15.102:0/3336939759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806de50 0x7f33c815ad60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f33c806c1e0 tx=0x7f33b402b9a0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.230+0000 7f33cea00640 1 -- 172.21.15.102:0/3336939759 shutdown_connections 2024-03-18T16:28:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.230+0000 7f33cea00640 1 --2- 172.21.15.102:0/3336939759 >> v2:172.21.15.102:3300/0 conn(0x7f33c8154150 0x7f33c8154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.230+0000 7f33cea00640 1 --2- 172.21.15.102:0/3336939759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806de50 0x7f33c815ad60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.230+0000 7f33cea00640 1 -- 172.21.15.102:0/3336939759 >> 172.21.15.102:0/3336939759 conn(0x7f33c80727e0 msgr2=0x7f33c8072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.230+0000 7f33cea00640 1 -- 172.21.15.102:0/3336939759 shutdown_connections 2024-03-18T16:28:16.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.230+0000 7f33cea00640 1 -- 172.21.15.102:0/3336939759 wait complete. 2024-03-18T16:28:16.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.233+0000 7f33cea00640 1 Processor -- start 2024-03-18T16:28:16.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.233+0000 7f33cea00640 1 -- start start 2024-03-18T16:28:16.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.235+0000 7f33cea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806b470 0x7f33c81f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:16.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.235+0000 7f33cea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c806de50 0x7f33c81526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:16.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.235+0000 7f33cea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33c8154150 0x7f33c8156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:16.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.235+0000 7f33cea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f33c80c5190 con 0x7f33c806de50 2024-03-18T16:28:16.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.235+0000 7f33cea00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f33c80c9d30 con 0x7f33c806b470 2024-03-18T16:28:16.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.235+0000 7f33cea00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f33c80c5310 con 0x7f33c8154150 2024-03-18T16:28:16.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.235+0000 7f33ccc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c806de50 0x7f33c81526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:16.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.235+0000 7f33cd600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806b470 0x7f33c81f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.235+0000 7f33ce000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33c8154150 0x7f33c8156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.236+0000 7f33cd600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806b470 0x7f33c81f0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:58998/0 (socket says 172.21.15.102:58998) 2024-03-18T16:28:16.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.236+0000 7f33cd600640 1 -- 172.21.15.102:0/3698375382 learned_addr learned my addr 172.21.15.102:0/3698375382 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:16.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.239+0000 7f33ccc00640 1 -- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33c8154150 msgr2=0x7f33c8156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:16.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.239+0000 7f33ccc00640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33c8154150 0x7f33c8156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.239+0000 7f33ccc00640 1 -- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806b470 msgr2=0x7f33c81f0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:16.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.239+0000 7f33ccc00640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806b470 0x7f33c81f0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.239+0000 7f33ccc00640 1 -- 172.21.15.102:0/3698375382 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c8155260 con 0x7f33c806de50 2024-03-18T16:28:16.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.242+0000 7f33ccc00640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c806de50 0x7f33c81526d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f33b4033000 tx=0x7f33b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.247+0000 7f33c5600640 1 -- 172.21.15.102:0/3698375382 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4048070 con 0x7f33c806de50 2024-03-18T16:28:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.247+0000 7f33c5600640 1 -- 172.21.15.102:0/3698375382 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33b4030660 con 0x7f33c806de50 2024-03-18T16:28:16.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.248+0000 7f33c5600640 1 -- 172.21.15.102:0/3698375382 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4034040 con 0x7f33c806de50 2024-03-18T16:28:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.248+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c8155500 con 0x7f33c806de50 2024-03-18T16:28:16.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.248+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c8156150 con 0x7f33c806de50 2024-03-18T16:28:16.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.249+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33c8074fd0 con 0x7f33c806de50 2024-03-18T16:28:16.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.510+0000 7f33c5600640 1 -- 172.21.15.102:0/3698375382 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f33b404b070 con 0x7f33c806de50 2024-03-18T16:28:16.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.511+0000 7f33c5600640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f33a806bbd0 0x7f33a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:16.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.512+0000 7f33cd600640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f33a806bbd0 0x7f33a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:16.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.513+0000 7f33cd600640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f33a806bbd0 0x7f33a806e090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f33bc001840 tx=0x7f33bc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:16.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.526+0000 7f33c5600640 1 -- 172.21.15.102:0/3698375382 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f33b4030810 con 0x7f33c806de50 2024-03-18T16:28:16.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.527+0000 7f33c5600640 1 -- 172.21.15.102:0/3698375382 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f33b4052020 con 0x7f33c806de50 2024-03-18T16:28:16.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.793+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33c80c7c70 con 0x7f33c806de50 2024-03-18T16:28:16.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.816+0000 7f33c5600640 1 -- 172.21.15.102:0/3698375382 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f33b407b5a0 con 0x7f33c806de50 2024-03-18T16:28:16.818 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:16.818 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:16.818 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.819+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f33a806bbd0 msgr2=0x7f33a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.819+0000 7f33cea00640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f33a806bbd0 0x7f33a806e090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f33bc001840 tx=0x7f33bc009040 comp rx=0 tx=0).stop 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.819+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c806de50 msgr2=0x7f33c81526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.819+0000 7f33cea00640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c806de50 0x7f33c81526d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f33b4033000 tx=0x7f33b4004730 comp rx=0 tx=0).stop 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.820+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 shutdown_connections 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.820+0000 7f33cea00640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f33a806bbd0 0x7f33a806e090 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.820+0000 7f33cea00640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f33c8154150 0x7f33c8156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.820+0000 7f33cea00640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c806de50 0x7f33c81526d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.820+0000 7f33cea00640 1 --2- 172.21.15.102:0/3698375382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33c806b470 0x7f33c81f0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.820+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 >> 172.21.15.102:0/3698375382 conn(0x7f33c80727e0 msgr2=0x7f33c8072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.820+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 shutdown_connections 2024-03-18T16:28:16.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:16.820+0000 7f33cea00640 1 -- 172.21.15.102:0/3698375382 wait complete. 2024-03-18T16:28:17.450 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:17.450 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:18.451 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:18.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.629+0000 7fca41a00640 1 Processor -- start 2024-03-18T16:28:18.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.629+0000 7fca41a00640 1 -- start start 2024-03-18T16:28:18.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.630+0000 7fca41a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fca3c15a6f0 0x7fca3c15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:18.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.630+0000 7fca41a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 0x7fca3c15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:18.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.630+0000 7fca41a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3c15c470 con 0x7fca3c15a310 2024-03-18T16:28:18.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.630+0000 7fca41a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fca3c0cd410 con 0x7fca3c15a6f0 2024-03-18T16:28:18.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.630+0000 7fca41a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca3c0ca500 con 0x7fca3c15baa0 2024-03-18T16:28:18.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.631+0000 7fca3be00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fca3c15a310 0x7fca3c06d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44662/0 (socket says 172.21.15.102:44662) 2024-03-18T16:28:18.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.631+0000 7fca3be00640 1 -- 172.21.15.102:0/105215238 learned_addr learned my addr 172.21.15.102:0/105215238 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:18.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.631+0000 7fca41000640 1 --2- 172.21.15.102:0/105215238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 0x7fca3c15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:18.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.631+0000 7fca3b400640 1 --2- 172.21.15.102:0/105215238 >> v2:172.21.15.102:3300/0 conn(0x7fca3c15a6f0 0x7fca3c15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:18.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.633+0000 7fca3aa00640 1 -- 172.21.15.102:0/105215238 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147820515 0 0) 0x7fca3c15c470 con 0x7fca3c15a310 2024-03-18T16:28:18.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.633+0000 7fca3aa00640 1 -- 172.21.15.102:0/105215238 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca20003520 con 0x7fca3c15a310 2024-03-18T16:28:18.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.634+0000 7fca3b400640 1 -- 172.21.15.102:0/105215238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 msgr2=0x7fca3c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:18.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.634+0000 7fca3b400640 1 --2- 172.21.15.102:0/105215238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 0x7fca3c15bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:18.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.634+0000 7fca3b400640 1 -- 172.21.15.102:0/105215238 >> v1:172.21.15.154:6789/0 conn(0x7fca3c15a310 legacy=0x7fca3c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:18.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.634+0000 7fca3b400640 1 -- 172.21.15.102:0/105215238 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c15b080 con 0x7fca3c15a6f0 2024-03-18T16:28:18.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.637+0000 7fca3b400640 1 --2- 172.21.15.102:0/105215238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c15a6f0 0x7fca3c15ab40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fca280097a0 tx=0x7fca2802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57ea378a445a7df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:18.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.641+0000 7fca3aa00640 1 -- 172.21.15.102:0/105215238 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28035020 con 0x7fca3c15a6f0 2024-03-18T16:28:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.642+0000 7fca3aa00640 1 -- 172.21.15.102:0/105215238 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca28031810 con 0x7fca3c15a6f0 2024-03-18T16:28:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.642+0000 7fca3aa00640 1 -- 172.21.15.102:0/105215238 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28031bc0 con 0x7fca3c15a6f0 2024-03-18T16:28:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.643+0000 7fca41a00640 1 -- 172.21.15.102:0/105215238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c15a6f0 msgr2=0x7fca3c15ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.643+0000 7fca41a00640 1 --2- 172.21.15.102:0/105215238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c15a6f0 0x7fca3c15ab40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fca280097a0 tx=0x7fca2802b8b0 comp rx=0 tx=0).stop 2024-03-18T16:28:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.643+0000 7fca41a00640 1 -- 172.21.15.102:0/105215238 shutdown_connections 2024-03-18T16:28:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.643+0000 7fca41a00640 1 --2- 172.21.15.102:0/105215238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 0x7fca3c15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:18.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.643+0000 7fca41a00640 1 --2- 172.21.15.102:0/105215238 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c15a6f0 0x7fca3c15ab40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:18.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.643+0000 7fca41a00640 1 -- 172.21.15.102:0/105215238 >> 172.21.15.102:0/105215238 conn(0x7fca3c08c1a0 msgr2=0x7fca3c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:18.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.643+0000 7fca41a00640 1 -- 172.21.15.102:0/105215238 shutdown_connections 2024-03-18T16:28:18.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.644+0000 7fca41a00640 1 -- 172.21.15.102:0/105215238 wait complete. 2024-03-18T16:28:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.645+0000 7fca41a00640 1 Processor -- start 2024-03-18T16:28:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.646+0000 7fca41a00640 1 -- start start 2024-03-18T16:28:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.646+0000 7fca41a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca3c15a310 0x7fca3c0c8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.646+0000 7fca41a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c15a6f0 0x7fca3c0c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.646+0000 7fca41a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 0x7fca3c1f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.646+0000 7fca41a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c0cce50 con 0x7fca3c15a6f0 2024-03-18T16:28:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.646+0000 7fca41a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c0caa80 con 0x7fca3c15a310 2024-03-18T16:28:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.646+0000 7fca41a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fca3c0cd130 con 0x7fca3c15baa0 2024-03-18T16:28:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.647+0000 7fca3be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca3c15a310 0x7fca3c0c8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.647+0000 7fca3b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c15a6f0 0x7fca3c0c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.647+0000 7fca41000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 0x7fca3c1f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.647+0000 7fca3be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca3c15a310 0x7fca3c0c8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59010/0 (socket says 172.21.15.102:59010) 2024-03-18T16:28:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.647+0000 7fca3be00640 1 -- 172.21.15.102:0/1625537336 learned_addr learned my addr 172.21.15.102:0/1625537336 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:18.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.651+0000 7fca3be00640 1 -- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 msgr2=0x7fca3c1f0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:18.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.651+0000 7fca3be00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 0x7fca3c1f0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:18.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.651+0000 7fca3be00640 1 -- 172.21.15.102:0/1625537336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c15a6f0 msgr2=0x7fca3c0c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:18.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.651+0000 7fca3be00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c15a6f0 0x7fca3c0c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:18.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.651+0000 7fca3be00640 1 -- 172.21.15.102:0/1625537336 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c0c9a10 con 0x7fca3c15a310 2024-03-18T16:28:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.653+0000 7fca3be00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca3c15a310 0x7fca3c0c8d50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00c3c0 tx=0x7fca2c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:18.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.656+0000 7fca33e00640 1 -- 172.21.15.102:0/1625537336 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c017070 con 0x7fca3c15a310 2024-03-18T16:28:18.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.657+0000 7fca33e00640 1 -- 172.21.15.102:0/1625537336 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2c005ce0 con 0x7fca3c15a310 2024-03-18T16:28:18.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.657+0000 7fca33e00640 1 -- 172.21.15.102:0/1625537336 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0072f0 con 0x7fca3c15a310 2024-03-18T16:28:18.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.657+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3c0c9cd0 con 0x7fca3c15a310 2024-03-18T16:28:18.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.657+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3c1f3bc0 con 0x7fca3c15a310 2024-03-18T16:28:18.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.658+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca3c0cd980 con 0x7fca3c15a310 2024-03-18T16:28:18.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.678+0000 7fca33e00640 1 -- 172.21.15.102:0/1625537336 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fca2c01a070 con 0x7fca3c15a310 2024-03-18T16:28:18.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.678+0000 7fca33e00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fca1c06bc50 0x7fca1c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:18.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.679+0000 7fca3b400640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fca1c06bc50 0x7fca1c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:18.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.680+0000 7fca3b400640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fca1c06bc50 0x7fca1c06e110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fca28036000 tx=0x7fca28008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:18.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.918+0000 7fca33e00640 1 -- 172.21.15.102:0/1625537336 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fca2c056700 con 0x7fca3c15a310 2024-03-18T16:28:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:18.930+0000 7fca33e00640 1 -- 172.21.15.102:0/1625537336 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fca3c0cd980 con 0x7fca3c15a310 2024-03-18T16:28:19.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.197+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca3c074fd0 con 0x7fca3c15a310 2024-03-18T16:28:19.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.325+0000 7fca33e00640 1 -- 172.21.15.102:0/1625537336 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fca2c021020 con 0x7fca3c15a310 2024-03-18T16:28:19.328 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:19.328 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:19.328 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fca1c06bc50 msgr2=0x7fca1c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fca1c06bc50 0x7fca1c06e110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fca28036000 tx=0x7fca28008030 comp rx=0 tx=0).stop 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca3c15a310 msgr2=0x7fca3c0c8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca3c15a310 0x7fca3c0c8d50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00c3c0 tx=0x7fca2c00c880 comp rx=0 tx=0).stop 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 shutdown_connections 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fca1c06bc50 0x7fca1c06e110 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fca3c15baa0 0x7fca3c1f0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c15a6f0 0x7fca3c0c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 --2- 172.21.15.102:0/1625537336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca3c15a310 0x7fca3c0c8d50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:19.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 >> 172.21.15.102:0/1625537336 conn(0x7fca3c08c1a0 msgr2=0x7fca3c14edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:19.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 shutdown_connections 2024-03-18T16:28:19.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:19.329+0000 7fca41a00640 1 -- 172.21.15.102:0/1625537336 wait complete. 2024-03-18T16:28:19.950 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:19.950 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:20.952 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:21.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.123+0000 7f88e7200640 1 Processor -- start 2024-03-18T16:28:21.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.124+0000 7f88e7200640 1 -- start start 2024-03-18T16:28:21.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.125+0000 7f88e7200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f88e8141650 0x7f88e8158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:21.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.125+0000 7f88e7200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88e815c3a0 0x7f88e80c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:21.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.125+0000 7f88e7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e80c4fa0 0x7f88e80c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:21.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.125+0000 7f88e7200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f88e8159370 con 0x7f88e8141650 2024-03-18T16:28:21.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.125+0000 7f88e7200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f88e8158ad0 con 0x7f88e815c3a0 2024-03-18T16:28:21.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.125+0000 7f88e7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f88e8158db0 con 0x7f88e80c4fa0 2024-03-18T16:28:21.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.125+0000 7f88e5400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88e815c3a0 0x7f88e80c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:21.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.126+0000 7f88e6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e80c4fa0 0x7f88e80c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:21.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.126+0000 7f88e5e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f88e8141650 0x7f88e8158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:21.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.126+0000 7f88e5400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88e815c3a0 0x7f88e80c6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40396/0 (socket says 172.21.15.102:40396) 2024-03-18T16:28:21.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.126+0000 7f88e5400640 1 -- 172.21.15.102:0/1876027903 learned_addr learned my addr 172.21.15.102:0/1876027903 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:21.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.128+0000 7f88e5400640 1 -- 172.21.15.102:0/1876027903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e80c4fa0 msgr2=0x7f88e80c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:21.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.128+0000 7f88e5400640 1 --2- 172.21.15.102:0/1876027903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e80c4fa0 0x7f88e80c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.128+0000 7f88e5400640 1 -- 172.21.15.102:0/1876027903 >> v2:172.21.15.102:3300/0 conn(0x7f88e8141650 msgr2=0x7f88e8158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:21.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.128+0000 7f88e5400640 1 --2- 172.21.15.102:0/1876027903 >> v2:172.21.15.102:3300/0 conn(0x7f88e8141650 0x7f88e8158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.128+0000 7f88e5400640 1 -- 172.21.15.102:0/1876027903 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e8159090 con 0x7f88e815c3a0 2024-03-18T16:28:21.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.130+0000 7f88e5400640 1 --2- 172.21.15.102:0/1876027903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e815c3a0 0x7f88e80c6bc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88e8068e80 tx=0x7f88d402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fd140e0a6b0fadd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:21.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e4a00640 1 -- 172.21.15.102:0/1876027903 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4035020 con 0x7f88e815c3a0 2024-03-18T16:28:21.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e4a00640 1 -- 172.21.15.102:0/1876027903 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d4031870 con 0x7f88e815c3a0 2024-03-18T16:28:21.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e4a00640 1 -- 172.21.15.102:0/1876027903 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4031c40 con 0x7f88e815c3a0 2024-03-18T16:28:21.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e7200640 1 -- 172.21.15.102:0/1876027903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e815c3a0 msgr2=0x7f88e80c6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:21.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e7200640 1 --2- 172.21.15.102:0/1876027903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e815c3a0 0x7f88e80c6bc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f88e8068e80 tx=0x7f88d402b910 comp rx=0 tx=0).stop 2024-03-18T16:28:21.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e7200640 1 -- 172.21.15.102:0/1876027903 shutdown_connections 2024-03-18T16:28:21.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e7200640 1 --2- 172.21.15.102:0/1876027903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e80c4fa0 0x7f88e80c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e7200640 1 --2- 172.21.15.102:0/1876027903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e815c3a0 0x7f88e80c6bc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e7200640 1 --2- 172.21.15.102:0/1876027903 >> v2:172.21.15.102:3300/0 conn(0x7f88e8141650 0x7f88e8158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.135+0000 7f88e7200640 1 -- 172.21.15.102:0/1876027903 >> 172.21.15.102:0/1876027903 conn(0x7f88e80cb890 msgr2=0x7f88e80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:21.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.136+0000 7f88e7200640 1 -- 172.21.15.102:0/1876027903 shutdown_connections 2024-03-18T16:28:21.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.136+0000 7f88e7200640 1 -- 172.21.15.102:0/1876027903 wait complete. 2024-03-18T16:28:21.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.137+0000 7f88e7200640 1 Processor -- start 2024-03-18T16:28:21.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.137+0000 7f88e7200640 1 -- start start 2024-03-18T16:28:21.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.138+0000 7f88e7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88e80c4fa0 0x7f88e8136cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:21.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.138+0000 7f88e7200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8141650 0x7f88e81349d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:21.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.138+0000 7f88e7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e815c3a0 0x7f88e8134f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:21.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.138+0000 7f88e7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f88e815c7c0 con 0x7f88e80c4fa0 2024-03-18T16:28:21.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.138+0000 7f88e7200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f88e80c66a0 con 0x7f88e8141650 2024-03-18T16:28:21.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.138+0000 7f88e7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f88e8141a30 con 0x7f88e815c3a0 2024-03-18T16:28:21.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.139+0000 7f88e5400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8141650 0x7f88e81349d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:21.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.139+0000 7f88e5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88e80c4fa0 0x7f88e8136cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:21.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.139+0000 7f88e5400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8141650 0x7f88e81349d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40402/0 (socket says 172.21.15.102:40402) 2024-03-18T16:28:21.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.139+0000 7f88e5400640 1 -- 172.21.15.102:0/2727987901 learned_addr learned my addr 172.21.15.102:0/2727987901 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:21.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.141+0000 7f88e6800640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e815c3a0 0x7f88e8134f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:21.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.143+0000 7f88e5400640 1 -- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e815c3a0 msgr2=0x7f88e8134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:21.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.143+0000 7f88e5400640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e815c3a0 0x7f88e8134f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.143+0000 7f88e5400640 1 -- 172.21.15.102:0/2727987901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88e80c4fa0 msgr2=0x7f88e8136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:21.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.143+0000 7f88e5400640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88e80c4fa0 0x7f88e8136cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.143+0000 7f88e5400640 1 -- 172.21.15.102:0/2727987901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e813ea50 con 0x7f88e8141650 2024-03-18T16:28:21.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.146+0000 7f88e5400640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8141650 0x7f88e81349d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f88d402b950 tx=0x7f88d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:21.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.149+0000 7f88de000640 1 -- 172.21.15.102:0/2727987901 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4041070 con 0x7f88e8141650 2024-03-18T16:28:21.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.149+0000 7f88de000640 1 -- 172.21.15.102:0/2727987901 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d4031e10 con 0x7f88e8141650 2024-03-18T16:28:21.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.149+0000 7f88de000640 1 -- 172.21.15.102:0/2727987901 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4035040 con 0x7f88e8141650 2024-03-18T16:28:21.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.149+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e8139720 con 0x7f88e8141650 2024-03-18T16:28:21.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.150+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e8138f80 con 0x7f88e8141650 2024-03-18T16:28:21.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.151+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e8074fd0 con 0x7f88e8141650 2024-03-18T16:28:21.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.170+0000 7f88de000640 1 -- 172.21.15.102:0/2727987901 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f88d4038020 con 0x7f88e8141650 2024-03-18T16:28:21.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.171+0000 7f88de000640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88c806bbd0 0x7f88c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:21.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.171+0000 7f88e5e00640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88c806bbd0 0x7f88c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:21.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.172+0000 7f88e5e00640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88c806bbd0 0x7f88c806e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009b40 tx=0x7f88d8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:21.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.174+0000 7f88de000640 1 -- 172.21.15.102:0/2727987901 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f88d4079540 con 0x7f88e8141650 2024-03-18T16:28:21.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.434+0000 7f88de000640 1 -- 172.21.15.102:0/2727987901 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f88d407b4a0 con 0x7f88e8141650 2024-03-18T16:28:21.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.701+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88e80b7f00 con 0x7f88e8141650 2024-03-18T16:28:21.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.747+0000 7f88de000640 1 -- 172.21.15.102:0/2727987901 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f88d407f540 con 0x7f88e8141650 2024-03-18T16:28:21.749 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:21.749 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:21.749 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:21.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88c806bbd0 msgr2=0x7f88c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:21.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88c806bbd0 0x7f88c806e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009b40 tx=0x7f88d8009190 comp rx=0 tx=0).stop 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8141650 msgr2=0x7f88e81349d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8141650 0x7f88e81349d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f88d402b950 tx=0x7f88d40046c0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 shutdown_connections 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f88c806bbd0 0x7f88c806e090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88e815c3a0 0x7f88e8134f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8141650 0x7f88e81349d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 --2- 172.21.15.102:0/2727987901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88e80c4fa0 0x7f88e8136cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 >> 172.21.15.102:0/2727987901 conn(0x7f88e80cb890 msgr2=0x7f88e80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 shutdown_connections 2024-03-18T16:28:21.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:21.749+0000 7f88e7200640 1 -- 172.21.15.102:0/2727987901 wait complete. 2024-03-18T16:28:22.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:28:22.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:28:22.373 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:22.374 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:23.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.552+0000 7f2507e00640 1 Processor -- start 2024-03-18T16:28:23.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.552+0000 7f2507e00640 1 -- start start 2024-03-18T16:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.553+0000 7f2507e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2508141650 0x7f2508158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.553+0000 7f2507e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f250815c3a0 0x7f25080c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.553+0000 7f2507e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2508158ad0 con 0x7f25080c4fa0 2024-03-18T16:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.553+0000 7f2507e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2508159370 con 0x7f250815c3a0 2024-03-18T16:28:23.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.553+0000 7f2507e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2508158db0 con 0x7f2508141650 2024-03-18T16:28:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.554+0000 7f2506000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f250815c3a0 0x7f25080c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.554+0000 7f2506a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2508141650 0x7f2508158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.555+0000 7f2506a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2508141650 0x7f2508158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48206/0 (socket says 172.21.15.102:48206) 2024-03-18T16:28:23.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.555+0000 7f2506a00640 1 -- 172.21.15.102:0/4031555077 learned_addr learned my addr 172.21.15.102:0/4031555077 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.557+0000 7f2506a00640 1 -- 172.21.15.102:0/4031555077 >> v2:172.21.15.154:3300/0 conn(0x7f250815c3a0 msgr2=0x7f25080c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.557+0000 7f2506a00640 1 --2- 172.21.15.102:0/4031555077 >> v2:172.21.15.154:3300/0 conn(0x7f250815c3a0 0x7f25080c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.557+0000 7f2506a00640 1 -- 172.21.15.102:0/4031555077 >> v1:172.21.15.154:6789/0 conn(0x7f25080c4fa0 legacy=0x7f250806f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:23.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.557+0000 7f2506a00640 1 -- 172.21.15.102:0/4031555077 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2508159090 con 0x7f2508141650 2024-03-18T16:28:23.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.560+0000 7f2506a00640 1 --2- 172.21.15.102:0/4031555077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2508141650 0x7f2508158420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009870 tx=0x7f24f802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1a4f2182192f5ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:23.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.564+0000 7f2505600640 1 -- 172.21.15.102:0/4031555077 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8034070 con 0x7f2508141650 2024-03-18T16:28:23.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2505600640 1 -- 172.21.15.102:0/4031555077 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24f8030840 con 0x7f2508141650 2024-03-18T16:28:23.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2505600640 1 -- 172.21.15.102:0/4031555077 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8030c10 con 0x7f2508141650 2024-03-18T16:28:23.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2507e00640 1 -- 172.21.15.102:0/4031555077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2508141650 msgr2=0x7f2508158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:23.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2507e00640 1 --2- 172.21.15.102:0/4031555077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2508141650 0x7f2508158420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f24f8009870 tx=0x7f24f802b980 comp rx=0 tx=0).stop 2024-03-18T16:28:23.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2507e00640 1 -- 172.21.15.102:0/4031555077 shutdown_connections 2024-03-18T16:28:23.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2507e00640 1 --2- 172.21.15.102:0/4031555077 >> v2:172.21.15.154:3300/0 conn(0x7f250815c3a0 0x7f25080c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:23.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2507e00640 1 --2- 172.21.15.102:0/4031555077 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2508141650 0x7f2508158420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:23.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2507e00640 1 -- 172.21.15.102:0/4031555077 >> 172.21.15.102:0/4031555077 conn(0x7f25080cb890 msgr2=0x7f25080cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:23.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2507e00640 1 -- 172.21.15.102:0/4031555077 shutdown_connections 2024-03-18T16:28:23.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.565+0000 7f2507e00640 1 -- 172.21.15.102:0/4031555077 wait complete. 2024-03-18T16:28:23.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.567+0000 7f2507e00640 1 Processor -- start 2024-03-18T16:28:23.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.567+0000 7f2507e00640 1 -- start start 2024-03-18T16:28:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.568+0000 7f2507e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25080c4fa0 0x7f25081554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.568+0000 7f2507e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2508141650 0x7f2508154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.568+0000 7f2507e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f250815c3a0 0x7f25080c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.568+0000 7f2507e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f250815c7c0 con 0x7f2508141650 2024-03-18T16:28:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.568+0000 7f2507e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25080c66a0 con 0x7f250815c3a0 2024-03-18T16:28:23.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.568+0000 7f2507e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2508141a30 con 0x7f25080c4fa0 2024-03-18T16:28:23.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.569+0000 7f2506a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25080c4fa0 0x7f25081554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:23.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.569+0000 7f2507400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f250815c3a0 0x7f25080c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:23.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.569+0000 7f2506000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2508141650 0x7f2508154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:23.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.569+0000 7f2507400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f250815c3a0 0x7f25080c8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40420/0 (socket says 172.21.15.102:40420) 2024-03-18T16:28:23.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.569+0000 7f2507400640 1 -- 172.21.15.102:0/3124778374 learned_addr learned my addr 172.21.15.102:0/3124778374 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:23.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.573+0000 7f2506a00640 1 -- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f250815c3a0 msgr2=0x7f25080c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:23.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.573+0000 7f2506a00640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f250815c3a0 0x7f25080c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:23.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.573+0000 7f2506a00640 1 -- 172.21.15.102:0/3124778374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2508141650 msgr2=0x7f2508154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:23.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.573+0000 7f2506a00640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2508141650 0x7f2508154c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:23.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.573+0000 7f2506a00640 1 -- 172.21.15.102:0/3124778374 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2508155ab0 con 0x7f25080c4fa0 2024-03-18T16:28:23.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.576+0000 7f2506a00640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25080c4fa0 0x7f25081554b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2508067fd0 tx=0x7f24f8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:23.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.578+0000 7f24ff400640 1 -- 172.21.15.102:0/3124778374 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8048070 con 0x7f25080c4fa0 2024-03-18T16:28:23.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.578+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25081f1650 con 0x7f25080c4fa0 2024-03-18T16:28:23.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.578+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25081f18f0 con 0x7f25080c4fa0 2024-03-18T16:28:23.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.579+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2508074fd0 con 0x7f25080c4fa0 2024-03-18T16:28:23.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.581+0000 7f24ff400640 1 -- 172.21.15.102:0/3124778374 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24f8039d90 con 0x7f25080c4fa0 2024-03-18T16:28:23.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.581+0000 7f24ff400640 1 -- 172.21.15.102:0/3124778374 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8052060 con 0x7f25080c4fa0 2024-03-18T16:28:23.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.858+0000 7f24ff400640 1 -- 172.21.15.102:0/3124778374 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f24f804b070 con 0x7f25080c4fa0 2024-03-18T16:28:23.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.861+0000 7f24ff400640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24e806bbd0 0x7f24e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:23.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.861+0000 7f2506000640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24e806bbd0 0x7f24e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:23.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.862+0000 7f2506000640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24e806bbd0 0x7f24e806e090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f25080c2680 tx=0x7f24f4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:23.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.862+0000 7f24ff400640 1 -- 172.21.15.102:0/3124778374 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f24f8079420 con 0x7f25080c4fa0 2024-03-18T16:28:23.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:23.874+0000 7f24ff400640 1 -- 172.21.15.102:0/3124778374 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f24f807b380 con 0x7f25080c4fa0 2024-03-18T16:28:24.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.160+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2508140520 con 0x7f25080c4fa0 2024-03-18T16:28:24.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.173+0000 7f24ff400640 1 -- 172.21.15.102:0/3124778374 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f24f807f420 con 0x7f25080c4fa0 2024-03-18T16:28:24.175 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:24.175 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:24.175 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:24.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24e806bbd0 msgr2=0x7f24e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24e806bbd0 0x7f24e806e090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f25080c2680 tx=0x7f24f4009190 comp rx=0 tx=0).stop 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25080c4fa0 msgr2=0x7f25081554b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25080c4fa0 0x7f25081554b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2508067fd0 tx=0x7f24f8004730 comp rx=0 tx=0).stop 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 shutdown_connections 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24e806bbd0 0x7f24e806e090 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f250815c3a0 0x7f25080c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2508141650 0x7f2508154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 --2- 172.21.15.102:0/3124778374 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f25080c4fa0 0x7f25081554b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 >> 172.21.15.102:0/3124778374 conn(0x7f25080cb890 msgr2=0x7f25080c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.175+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 shutdown_connections 2024-03-18T16:28:24.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:24.176+0000 7f2507e00640 1 -- 172.21.15.102:0/3124778374 wait complete. 2024-03-18T16:28:24.796 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:24.797 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:25.798 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:25.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.967+0000 7f3657200640 1 Processor -- start 2024-03-18T16:28:25.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.968+0000 7f3657200640 1 -- start start 2024-03-18T16:28:25.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.969+0000 7f3657200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f365806b470 0x7f365815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.969+0000 7f3657200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f365806bf60 con 0x7f365806de50 2024-03-18T16:28:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.969+0000 7f3657200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36580c94b0 con 0x7f3658154150 2024-03-18T16:28:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.969+0000 7f3657200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f36580ca0a0 con 0x7f365806b470 2024-03-18T16:28:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.969+0000 7f3656800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3658154150 0x7f36580c5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:45070/0 (socket says 172.21.15.102:45070) 2024-03-18T16:28:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.969+0000 7f3656800640 1 -- 172.21.15.102:0/606772722 learned_addr learned my addr 172.21.15.102:0/606772722 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:25.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.970+0000 7f3655e00640 1 --2- 172.21.15.102:0/606772722 >> v2:172.21.15.102:3300/0 conn(0x7f365806b470 0x7f365815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:25.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.972+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897519324 0 0) 0x7f36580c94b0 con 0x7f3658154150 2024-03-18T16:28:25.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.972+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f363c003520 con 0x7f3658154150 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1694615802 0 0) 0x7f363c003520 con 0x7f3658154150 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 >> v2:172.21.15.102:3300/0 conn(0x7f365806b470 msgr2=0x7f365815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 --2- 172.21.15.102:0/606772722 >> v2:172.21.15.102:3300/0 conn(0x7f365806b470 0x7f365815ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 >> v1:172.21.15.102:6789/0 conn(0x7f365806de50 legacy=0x7f365806b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36580c9770 con 0x7f3658154150 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f36500019f0 con 0x7f3658154150 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 >> v1:172.21.15.154:6789/0 conn(0x7f3658154150 legacy=0x7f36580c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f363c0039b0 0x7f363c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c004800 0x7f363c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 0x7f363c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f36580ca0a0 con 0x7f363c004800 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f363c00c060 con 0x7f363c0039b0 2024-03-18T16:28:25.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.976+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f363c00c260 con 0x7f363c0051c0 2024-03-18T16:28:25.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.977+0000 7f3656800640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 0x7f363c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:25.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.977+0000 7f3655e00640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f363c0039b0 0x7f363c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.981+0000 7f3656800640 1 -- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f363c0039b0 msgr2=0x7f363c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.981+0000 7f3656800640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f363c0039b0 0x7f363c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.981+0000 7f3656800640 1 -- 172.21.15.102:0/606772722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c004800 msgr2=0x7f363c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:25.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.981+0000 7f3656800640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f363c004800 0x7f363c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:25.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.981+0000 7f3656800640 1 -- 172.21.15.102:0/606772722 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36580c9770 con 0x7f363c0051c0 2024-03-18T16:28:25.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.982+0000 7f3656800640 1 -- 172.21.15.102:0/606772722 reap_dead start 2024-03-18T16:28:25.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.984+0000 7f3656800640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 0x7f363c00b9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f36500013d0 tx=0x7f363c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b3afbd74decd2d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:25.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.988+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f365000a480 con 0x7f363c0051c0 2024-03-18T16:28:25.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.988+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3650034630 con 0x7f363c0051c0 2024-03-18T16:28:25.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.988+0000 7f3654a00640 1 -- 172.21.15.102:0/606772722 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3650034a20 con 0x7f363c0051c0 2024-03-18T16:28:25.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.989+0000 7f3657200640 1 -- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 msgr2=0x7f363c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:25.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.989+0000 7f3657200640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 0x7f363c00b9c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f36500013d0 tx=0x7f363c005830 comp rx=0 tx=0).stop 2024-03-18T16:28:25.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.989+0000 7f3657200640 1 -- 172.21.15.102:0/606772722 shutdown_connections 2024-03-18T16:28:25.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.989+0000 7f3657200640 1 --2- 172.21.15.102:0/606772722 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 0x7f363c00b9c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:25.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.989+0000 7f3657200640 1 -- 172.21.15.102:0/606772722 >> 172.21.15.102:0/606772722 conn(0x7f36580727e0 msgr2=0x7f3658072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:25.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.989+0000 7f3657200640 1 -- 172.21.15.102:0/606772722 shutdown_connections 2024-03-18T16:28:25.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.990+0000 7f3657200640 1 -- 172.21.15.102:0/606772722 wait complete. 2024-03-18T16:28:25.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.991+0000 7f3657200640 1 Processor -- start 2024-03-18T16:28:25.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.991+0000 7f3657200640 1 -- start start 2024-03-18T16:28:25.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.992+0000 7f3657200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 0x7f36581504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:25.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.992+0000 7f3657200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f365815a320 0x7f36581509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:25.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.992+0000 7f3657200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36581f0650 0x7f365814c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:25.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.992+0000 7f3657200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f365815ad60 con 0x7f36581f0650 2024-03-18T16:28:25.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.992+0000 7f3657200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f36580c9d30 con 0x7f365815a320 2024-03-18T16:28:25.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.992+0000 7f3657200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f365815aee0 con 0x7f363c0051c0 2024-03-18T16:28:25.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.993+0000 7f3655400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f365815a320 0x7f36581509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:25.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.993+0000 7f3655e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 0x7f36581504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:25.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.993+0000 7f3655400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f365815a320 0x7f36581509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40440/0 (socket says 172.21.15.102:40440) 2024-03-18T16:28:25.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.993+0000 7f3655400640 1 -- 172.21.15.102:0/812308952 learned_addr learned my addr 172.21.15.102:0/812308952 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:25.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.993+0000 7f3656800640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36581f0650 0x7f365814c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:25.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.997+0000 7f3655400640 1 -- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 msgr2=0x7f36581504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:25.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.997+0000 7f3655400640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 0x7f36581504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:25.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.997+0000 7f3655400640 1 -- 172.21.15.102:0/812308952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36581f0650 msgr2=0x7f365814c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:25.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.997+0000 7f3655400640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36581f0650 0x7f365814c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:25.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.997+0000 7f3655400640 1 -- 172.21.15.102:0/812308952 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f365814fcc0 con 0x7f365815a320 2024-03-18T16:28:26.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:25.999+0000 7f3655400640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f365815a320 0x7f36581509e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f364400c530 tx=0x7f364400c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:26.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.003+0000 7f364e000640 1 -- 172.21.15.102:0/812308952 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36440052d0 con 0x7f365815a320 2024-03-18T16:28:26.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.003+0000 7f364e000640 1 -- 172.21.15.102:0/812308952 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3644005490 con 0x7f365815a320 2024-03-18T16:28:26.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.003+0000 7f364e000640 1 -- 172.21.15.102:0/812308952 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644012870 con 0x7f365815a320 2024-03-18T16:28:26.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.003+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36581e67c0 con 0x7f365815a320 2024-03-18T16:28:26.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.003+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36581e6d80 con 0x7f365815a320 2024-03-18T16:28:26.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.004+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3658074fd0 con 0x7f365815a320 2024-03-18T16:28:26.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.023+0000 7f364e000640 1 -- 172.21.15.102:0/812308952 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3644012a10 con 0x7f365815a320 2024-03-18T16:28:26.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.024+0000 7f364e000640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f363806bc50 0x7f363806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:26.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.025+0000 7f3655e00640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f363806bc50 0x7f363806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:26.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.025+0000 7f3655e00640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f363806bc50 0x7f363806e110 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3648009b40 tx=0x7f3648009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:26.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.026+0000 7f364e000640 1 -- 172.21.15.102:0/812308952 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3644057490 con 0x7f365815a320 2024-03-18T16:28:26.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.283+0000 7f364e000640 1 -- 172.21.15.102:0/812308952 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3644059af0 con 0x7f365815a320 2024-03-18T16:28:26.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.519+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36580c8d50 con 0x7f365815a320 2024-03-18T16:28:26.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.534+0000 7f364e000640 1 -- 172.21.15.102:0/812308952 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f36440593f0 con 0x7f365815a320 2024-03-18T16:28:26.536 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:26.536 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:26.536 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:26.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f363806bc50 msgr2=0x7f363806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:26.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f363806bc50 0x7f363806e110 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3648009b40 tx=0x7f3648009190 comp rx=0 tx=0).stop 2024-03-18T16:28:26.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f365815a320 msgr2=0x7f36581509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:26.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f365815a320 0x7f36581509e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f364400c530 tx=0x7f364400c930 comp rx=0 tx=0).stop 2024-03-18T16:28:26.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 shutdown_connections 2024-03-18T16:28:26.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f363806bc50 0x7f363806e110 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:26.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36581f0650 0x7f365814c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:26.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f365815a320 0x7f36581509e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:26.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 --2- 172.21.15.102:0/812308952 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f363c0051c0 0x7f36581504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:26.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 >> 172.21.15.102:0/812308952 conn(0x7f36580727e0 msgr2=0x7f36580c5070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:26.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 shutdown_connections 2024-03-18T16:28:26.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:26.537+0000 7f3657200640 1 -- 172.21.15.102:0/812308952 wait complete. 2024-03-18T16:28:27.167 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:27.168 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:28.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:28.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.340+0000 7f344e200640 1 Processor -- start 2024-03-18T16:28:28.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.340+0000 7f344e200640 1 -- start start 2024-03-18T16:28:28.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.341+0000 7f344e200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 0x7f344815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:28.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.341+0000 7f344e200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f34480c4fa0 0x7f34480c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:28.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.341+0000 7f344e200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34480c72d0 con 0x7f3448141650 2024-03-18T16:28:28.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.342+0000 7f344e200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f344815c8e0 con 0x7f34480c4fa0 2024-03-18T16:28:28.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.342+0000 7f344e200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3448159370 con 0x7f3448158420 2024-03-18T16:28:28.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.342+0000 7f344ce00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3448141650 0x7f34480c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58498/0 (socket says 172.21.15.102:58498) 2024-03-18T16:28:28.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.342+0000 7f344ce00640 1 -- 172.21.15.102:0/2172988603 learned_addr learned my addr 172.21.15.102:0/2172988603 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:28.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.342+0000 7f344d800640 1 --2- 172.21.15.102:0/2172988603 >> v2:172.21.15.154:3300/0 conn(0x7f34480c4fa0 0x7f34480c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:28.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.342+0000 7f3447e00640 1 --2- 172.21.15.102:0/2172988603 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 0x7f344815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:28.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.345+0000 7f3447400640 1 -- 172.21.15.102:0/2172988603 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3705691882 0 0) 0x7f34480c72d0 con 0x7f3448141650 2024-03-18T16:28:28.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.345+0000 7f3447400640 1 -- 172.21.15.102:0/2172988603 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3430003520 con 0x7f3448141650 2024-03-18T16:28:28.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.346+0000 7f3447e00640 1 -- 172.21.15.102:0/2172988603 >> v2:172.21.15.154:3300/0 conn(0x7f34480c4fa0 msgr2=0x7f34480c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:28.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.346+0000 7f3447e00640 1 --2- 172.21.15.102:0/2172988603 >> v2:172.21.15.154:3300/0 conn(0x7f34480c4fa0 0x7f34480c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.346+0000 7f3447e00640 1 -- 172.21.15.102:0/2172988603 >> v1:172.21.15.102:6789/0 conn(0x7f3448141650 legacy=0x7f34480c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:28.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.346+0000 7f3447e00640 1 -- 172.21.15.102:0/2172988603 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3448158840 con 0x7f3448158420 2024-03-18T16:28:28.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.349+0000 7f3447e00640 1 --2- 172.21.15.102:0/2172988603 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 0x7f344815c3a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f34340097a0 tx=0x7f343402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d65cb52f67994fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:28.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.352+0000 7f3447400640 1 -- 172.21.15.102:0/2172988603 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434035020 con 0x7f3448158420 2024-03-18T16:28:28.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.352+0000 7f3447400640 1 -- 172.21.15.102:0/2172988603 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3434031810 con 0x7f3448158420 2024-03-18T16:28:28.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.352+0000 7f3447400640 1 -- 172.21.15.102:0/2172988603 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434031be0 con 0x7f3448158420 2024-03-18T16:28:28.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.353+0000 7f344e200640 1 -- 172.21.15.102:0/2172988603 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 msgr2=0x7f344815c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:28.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.353+0000 7f344e200640 1 --2- 172.21.15.102:0/2172988603 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 0x7f344815c3a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f34340097a0 tx=0x7f343402b8b0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.353+0000 7f344e200640 1 -- 172.21.15.102:0/2172988603 shutdown_connections 2024-03-18T16:28:28.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.353+0000 7f344e200640 1 --2- 172.21.15.102:0/2172988603 >> v2:172.21.15.154:3300/0 conn(0x7f34480c4fa0 0x7f34480c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.354+0000 7f344e200640 1 --2- 172.21.15.102:0/2172988603 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 0x7f344815c3a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.354+0000 7f344e200640 1 -- 172.21.15.102:0/2172988603 >> 172.21.15.102:0/2172988603 conn(0x7f34480cb890 msgr2=0x7f34480cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:28.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.354+0000 7f344e200640 1 -- 172.21.15.102:0/2172988603 shutdown_connections 2024-03-18T16:28:28.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.354+0000 7f344e200640 1 -- 172.21.15.102:0/2172988603 wait complete. 2024-03-18T16:28:28.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.356+0000 7f344e200640 1 Processor -- start 2024-03-18T16:28:28.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.356+0000 7f344e200640 1 -- start start 2024-03-18T16:28:28.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.358+0000 7f344e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34480c4fa0 0x7f34480c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:28.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.358+0000 7f344e200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3448141650 0x7f34481e7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:28.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.358+0000 7f344e200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 0x7f3448139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:28.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.358+0000 7f344e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3448159090 con 0x7f34480c4fa0 2024-03-18T16:28:28.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.358+0000 7f344e200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3448158db0 con 0x7f3448141650 2024-03-18T16:28:28.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.358+0000 7f344e200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f34480c63c0 con 0x7f3448158420 2024-03-18T16:28:28.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.359+0000 7f3447e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3448141650 0x7f34481e7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:28.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.359+0000 7f344ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34480c4fa0 0x7f34480c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:28.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.359+0000 7f344d800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 0x7f3448139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:28.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.359+0000 7f3447e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3448141650 0x7f34481e7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40466/0 (socket says 172.21.15.102:40466) 2024-03-18T16:28:28.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.359+0000 7f3447e00640 1 -- 172.21.15.102:0/347613859 learned_addr learned my addr 172.21.15.102:0/347613859 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:28.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.363+0000 7f344ce00640 1 -- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 msgr2=0x7f3448139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:28.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.363+0000 7f344ce00640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 0x7f3448139140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.363+0000 7f344ce00640 1 -- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3448141650 msgr2=0x7f34481e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:28.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.363+0000 7f344ce00640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3448141650 0x7f34481e7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.363+0000 7f344ce00640 1 -- 172.21.15.102:0/347613859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34481f0860 con 0x7f34480c4fa0 2024-03-18T16:28:28.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.366+0000 7f344ce00640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34480c4fa0 0x7f34480c9690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c400 tx=0x7f343c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:28.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.369+0000 7f3444c00640 1 -- 172.21.15.102:0/347613859 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c017070 con 0x7f34480c4fa0 2024-03-18T16:28:28.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.369+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34481f0b40 con 0x7f34480c4fa0 2024-03-18T16:28:28.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.369+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34481f1100 con 0x7f34480c4fa0 2024-03-18T16:28:28.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.370+0000 7f3444c00640 1 -- 172.21.15.102:0/347613859 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f343c005ce0 con 0x7f34480c4fa0 2024-03-18T16:28:28.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.370+0000 7f3444c00640 1 -- 172.21.15.102:0/347613859 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c021060 con 0x7f34480c4fa0 2024-03-18T16:28:28.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.371+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3424005290 con 0x7f34480c4fa0 2024-03-18T16:28:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.394+0000 7f3444c00640 1 -- 172.21.15.102:0/347613859 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f343c01a070 con 0x7f34480c4fa0 2024-03-18T16:28:28.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.395+0000 7f3444c00640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f342806bbd0 0x7f342806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:28.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.395+0000 7f3447e00640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f342806bbd0 0x7f342806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:28.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.396+0000 7f3447e00640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f342806bbd0 0x7f342806e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f34340097a0 tx=0x7f343403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:28.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.396+0000 7f3444c00640 1 -- 172.21.15.102:0/347613859 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f343c056af0 con 0x7f34480c4fa0 2024-03-18T16:28:28.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.651+0000 7f3444c00640 1 -- 172.21.15.102:0/347613859 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f343c058a50 con 0x7f34480c4fa0 2024-03-18T16:28:28.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.883+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3424005580 con 0x7f34480c4fa0 2024-03-18T16:28:28.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.904+0000 7f3444c00640 1 -- 172.21.15.102:0/347613859 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f343c05caf0 con 0x7f34480c4fa0 2024-03-18T16:28:28.906 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:28.906 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:28.907 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:28.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f342806bbd0 msgr2=0x7f342806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:28.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f342806bbd0 0x7f342806e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f34340097a0 tx=0x7f343403c040 comp rx=0 tx=0).stop 2024-03-18T16:28:28.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34480c4fa0 msgr2=0x7f34480c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:28.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34480c4fa0 0x7f34480c9690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c400 tx=0x7f343c00c8c0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 shutdown_connections 2024-03-18T16:28:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f342806bbd0 0x7f342806e090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3448158420 0x7f3448139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3448141650 0x7f34481e7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 --2- 172.21.15.102:0/347613859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34480c4fa0 0x7f34480c9690 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.908+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 >> 172.21.15.102:0/347613859 conn(0x7f34480cb890 msgr2=0x7f34481570a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.909+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 shutdown_connections 2024-03-18T16:28:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:28.909+0000 7f344e200640 1 -- 172.21.15.102:0/347613859 wait complete. 2024-03-18T16:28:29.537 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:29.537 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:30.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:30.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.713+0000 7f8b9be00640 1 Processor -- start 2024-03-18T16:28:30.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.714+0000 7f8b9be00640 1 -- start start 2024-03-18T16:28:30.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.715+0000 7f8b9be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8b9c0ca960 0x7f8b9c06c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:30.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.715+0000 7f8b9be00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b9c0cad40 0x7f8b9c15e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:30.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.715+0000 7f8b9be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c15a1a0 0x7f8b9c0705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:30.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.715+0000 7f8b9be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8b9c0c8680 con 0x7f8b9c0ca960 2024-03-18T16:28:30.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.715+0000 7f8b9be00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8b9c15e670 con 0x7f8b9c0cad40 2024-03-18T16:28:30.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.715+0000 7f8b9be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8b9c06c9c0 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.716+0000 7f8b9aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8b9c0ca960 0x7f8b9c06c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:30.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.716+0000 7f8b9b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c15a1a0 0x7f8b9c0705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:30.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.716+0000 7f8b9a000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b9c0cad40 0x7f8b9c15e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:30.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.716+0000 7f8b9aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8b9c0ca960 0x7f8b9c06c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51514/0 (socket says 172.21.15.102:51514) 2024-03-18T16:28:30.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.716+0000 7f8b9aa00640 1 -- 172.21.15.102:0/3372335158 learned_addr learned my addr 172.21.15.102:0/3372335158 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.719+0000 7f8b9a000640 1 -- 172.21.15.102:0/3372335158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c15a1a0 msgr2=0x7f8b9c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.719+0000 7f8b9a000640 1 --2- 172.21.15.102:0/3372335158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c15a1a0 0x7f8b9c0705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.719+0000 7f8b9a000640 1 -- 172.21.15.102:0/3372335158 >> v2:172.21.15.102:3300/0 conn(0x7f8b9c0ca960 msgr2=0x7f8b9c06c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.719+0000 7f8b9a000640 1 --2- 172.21.15.102:0/3372335158 >> v2:172.21.15.102:3300/0 conn(0x7f8b9c0ca960 0x7f8b9c06c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:30.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.719+0000 7f8b9a000640 1 -- 172.21.15.102:0/3372335158 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c06cca0 con 0x7f8b9c0cad40 2024-03-18T16:28:30.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.722+0000 7f8b9a000640 1 --2- 172.21.15.102:0/3372335158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c0cad40 0x7f8b9c15e130 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0cb220 tx=0x7f8b8802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=774cd9e5cfb9711a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:30.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.727+0000 7f8b99600640 1 -- 172.21.15.102:0/3372335158 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b88035020 con 0x7f8b9c0cad40 2024-03-18T16:28:30.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.727+0000 7f8b99600640 1 -- 172.21.15.102:0/3372335158 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b88031830 con 0x7f8b9c0cad40 2024-03-18T16:28:30.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.727+0000 7f8b99600640 1 -- 172.21.15.102:0/3372335158 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b88031c00 con 0x7f8b9c0cad40 2024-03-18T16:28:30.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.728+0000 7f8b9be00640 1 -- 172.21.15.102:0/3372335158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c0cad40 msgr2=0x7f8b9c15e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.728+0000 7f8b9be00640 1 --2- 172.21.15.102:0/3372335158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c0cad40 0x7f8b9c15e130 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0cb220 tx=0x7f8b8802b8d0 comp rx=0 tx=0).stop 2024-03-18T16:28:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.728+0000 7f8b9be00640 1 -- 172.21.15.102:0/3372335158 shutdown_connections 2024-03-18T16:28:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.728+0000 7f8b9be00640 1 --2- 172.21.15.102:0/3372335158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c15a1a0 0x7f8b9c0705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.728+0000 7f8b9be00640 1 --2- 172.21.15.102:0/3372335158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c0cad40 0x7f8b9c15e130 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.728+0000 7f8b9be00640 1 --2- 172.21.15.102:0/3372335158 >> v2:172.21.15.102:3300/0 conn(0x7f8b9c0ca960 0x7f8b9c06c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.728+0000 7f8b9be00640 1 -- 172.21.15.102:0/3372335158 >> 172.21.15.102:0/3372335158 conn(0x7f8b9c067190 msgr2=0x7f8b9c0675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.728+0000 7f8b9be00640 1 -- 172.21.15.102:0/3372335158 shutdown_connections 2024-03-18T16:28:30.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.728+0000 7f8b9be00640 1 -- 172.21.15.102:0/3372335158 wait complete. 2024-03-18T16:28:30.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.731+0000 7f8b9be00640 1 Processor -- start 2024-03-18T16:28:30.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.731+0000 7f8b9be00640 1 -- start start 2024-03-18T16:28:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.732+0000 7f8b9be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c0ca960 0x7f8b9c13f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.732+0000 7f8b9be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b9c0cad40 0x7f8b9c141db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.732+0000 7f8b9be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c15a1a0 0x7f8b9c13dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.732+0000 7f8b9be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c15dc10 con 0x7f8b9c0cad40 2024-03-18T16:28:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.732+0000 7f8b9be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9c15d930 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.732+0000 7f8b9be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8b9c070ae0 con 0x7f8b9c0ca960 2024-03-18T16:28:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.733+0000 7f8b9aa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c0ca960 0x7f8b9c13f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.733+0000 7f8b9b400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c15a1a0 0x7f8b9c13dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.733+0000 7f8b9a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b9c0cad40 0x7f8b9c141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.733+0000 7f8b9b400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c15a1a0 0x7f8b9c13dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40478/0 (socket says 172.21.15.102:40478) 2024-03-18T16:28:30.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.733+0000 7f8b9b400640 1 -- 172.21.15.102:0/4134799630 learned_addr learned my addr 172.21.15.102:0/4134799630 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.737+0000 7f8b9b400640 1 -- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c0ca960 msgr2=0x7f8b9c13f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:30.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.737+0000 7f8b9b400640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c0ca960 0x7f8b9c13f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.737+0000 7f8b9b400640 1 -- 172.21.15.102:0/4134799630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b9c0cad40 msgr2=0x7f8b9c141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.737+0000 7f8b9b400640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b9c0cad40 0x7f8b9c141db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:30.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.737+0000 7f8b9b400640 1 -- 172.21.15.102:0/4134799630 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b9c0cc540 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.740+0000 7f8b9b400640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c15a1a0 0x7f8b9c13dc50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009dd0 tx=0x7f8b9400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.744+0000 7f8b92a00640 1 -- 172.21.15.102:0/4134799630 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94018020 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.744+0000 7f8b92a00640 1 -- 172.21.15.102:0/4134799630 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b94014480 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.744+0000 7f8b92a00640 1 -- 172.21.15.102:0/4134799630 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94014870 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.745+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9c1f0a70 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.745+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b9c140200 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.746+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b9c074fd0 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.772+0000 7f8b92a00640 1 -- 172.21.15.102:0/4134799630 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8b94005d10 con 0x7f8b9c15a1a0 2024-03-18T16:28:30.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.773+0000 7f8b92a00640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b7c06bc20 0x7f8b7c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.773+0000 7f8b9aa00640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b7c06bc20 0x7f8b7c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:30.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:30.774+0000 7f8b9aa00640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b7c06bc20 0x7f8b7c06e0e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c006cc0 tx=0x7f8b8c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:31.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.033+0000 7f8b92a00640 1 -- 172.21.15.102:0/4134799630 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8b94058120 con 0x7f8b9c15a1a0 2024-03-18T16:28:31.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.044+0000 7f8b92a00640 1 -- 172.21.15.102:0/4134799630 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8b94022070 con 0x7f8b9c15a1a0 2024-03-18T16:28:31.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.292+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b9c140440 con 0x7f8b9c15a1a0 2024-03-18T16:28:31.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.396+0000 7f8b92a00640 1 -- 172.21.15.102:0/4134799630 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8b94010040 con 0x7f8b9c15a1a0 2024-03-18T16:28:31.398 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:31.398 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:31.398 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.399+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b7c06bc20 msgr2=0x7f8b7c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.399+0000 7f8b9be00640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b7c06bc20 0x7f8b7c06e0e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c006cc0 tx=0x7f8b8c009240 comp rx=0 tx=0).stop 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.399+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c15a1a0 msgr2=0x7f8b9c13dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.399+0000 7f8b9be00640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c15a1a0 0x7f8b9c13dc50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009dd0 tx=0x7f8b9400c490 comp rx=0 tx=0).stop 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.400+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 shutdown_connections 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.400+0000 7f8b9be00640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b7c06bc20 0x7f8b7c06e0e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.400+0000 7f8b9be00640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b9c15a1a0 0x7f8b9c13dc50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.400+0000 7f8b9be00640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b9c0cad40 0x7f8b9c141db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.400+0000 7f8b9be00640 1 --2- 172.21.15.102:0/4134799630 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b9c0ca960 0x7f8b9c13f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.400+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 >> 172.21.15.102:0/4134799630 conn(0x7f8b9c067190 msgr2=0x7f8b9c067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.400+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 shutdown_connections 2024-03-18T16:28:31.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:31.400+0000 7f8b9be00640 1 -- 172.21.15.102:0/4134799630 wait complete. 2024-03-18T16:28:32.029 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:32.029 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:33.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:33.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.207+0000 7fdbbd400640 1 Processor -- start 2024-03-18T16:28:33.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.207+0000 7fdbbd400640 1 -- start start 2024-03-18T16:28:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.208+0000 7fdbbd400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdbb80cfe70 0x7fdbb815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.208+0000 7fdbbd400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbb80d0250 0x7fdbb80c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.208+0000 7fdbbd400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbb80cabd0 con 0x7fdbb80cc2f0 2024-03-18T16:28:33.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.208+0000 7fdbbd400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fdbb80caea0 con 0x7fdbb80d0250 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.208+0000 7fdbbd400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdbb80cb180 con 0x7fdbb80cfe70 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.209+0000 7fdbb7e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdbb80cfe70 0x7fdbb815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.209+0000 7fdbb7e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdbb80cfe70 0x7fdbb815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54318/0 (socket says 172.21.15.102:54318) 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.209+0000 7fdbb7e00640 1 -- 172.21.15.102:0/3291474782 learned_addr learned my addr 172.21.15.102:0/3291474782 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.209+0000 7fdbb7400640 1 --2- 172.21.15.102:0/3291474782 >> v2:172.21.15.102:3300/0 conn(0x7fdbb80d0250 0x7fdbb80c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.211+0000 7fdbb7e00640 1 -- 172.21.15.102:0/3291474782 >> v2:172.21.15.102:3300/0 conn(0x7fdbb80d0250 msgr2=0x7fdbb80c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.211+0000 7fdbb7e00640 1 --2- 172.21.15.102:0/3291474782 >> v2:172.21.15.102:3300/0 conn(0x7fdbb80d0250 0x7fdbb80c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.211+0000 7fdbb7e00640 1 -- 172.21.15.102:0/3291474782 >> v1:172.21.15.102:6789/0 conn(0x7fdbb80cc2f0 legacy=0x7fdbb80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.212+0000 7fdbb7e00640 1 -- 172.21.15.102:0/3291474782 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb80cb460 con 0x7fdbb80cfe70 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.214+0000 7fdbb7e00640 1 --2- 172.21.15.102:0/3291474782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80cfe70 0x7fdbb815ae40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80cd800 tx=0x7fdba802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fa5025435e5630b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.218+0000 7fdbb6a00640 1 -- 172.21.15.102:0/3291474782 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba802f850 con 0x7fdbb80cfe70 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.218+0000 7fdbb6a00640 1 -- 172.21.15.102:0/3291474782 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdba802f9f0 con 0x7fdbb80cfe70 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.218+0000 7fdbb6a00640 1 -- 172.21.15.102:0/3291474782 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba802fdc0 con 0x7fdbb80cfe70 2024-03-18T16:28:33.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.219+0000 7fdbbd400640 1 -- 172.21.15.102:0/3291474782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80cfe70 msgr2=0x7fdbb815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.219+0000 7fdbbd400640 1 --2- 172.21.15.102:0/3291474782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80cfe70 0x7fdbb815ae40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80cd800 tx=0x7fdba802ba30 comp rx=0 tx=0).stop 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.219+0000 7fdbbd400640 1 -- 172.21.15.102:0/3291474782 shutdown_connections 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.219+0000 7fdbbd400640 1 --2- 172.21.15.102:0/3291474782 >> v2:172.21.15.102:3300/0 conn(0x7fdbb80d0250 0x7fdbb80c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.219+0000 7fdbbd400640 1 --2- 172.21.15.102:0/3291474782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80cfe70 0x7fdbb815ae40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.219+0000 7fdbbd400640 1 -- 172.21.15.102:0/3291474782 >> 172.21.15.102:0/3291474782 conn(0x7fdbb808c3e0 msgr2=0x7fdbb806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.219+0000 7fdbbd400640 1 -- 172.21.15.102:0/3291474782 shutdown_connections 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.219+0000 7fdbbd400640 1 -- 172.21.15.102:0/3291474782 wait complete. 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.221+0000 7fdbbd400640 1 Processor -- start 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.221+0000 7fdbbd400640 1 -- start start 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.222+0000 7fdbbd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb80cc2f0 0x7fdbb81e26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.222+0000 7fdbbd400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb80cfe70 0x7fdbb8150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.222+0000 7fdbbd400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80d0250 0x7fdbb81591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.222+0000 7fdbbd400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb80cbd00 con 0x7fdbb80cc2f0 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.222+0000 7fdbbd400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb80cba20 con 0x7fdbb80d0250 2024-03-18T16:28:33.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.222+0000 7fdbbd400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb80c6b20 con 0x7fdbb80cfe70 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.222+0000 7fdbbca00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80d0250 0x7fdbb81591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.223+0000 7fdbb7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb80cc2f0 0x7fdbb81e26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.223+0000 7fdbb7400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb80cfe70 0x7fdbb8150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.223+0000 7fdbbca00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80d0250 0x7fdbb81591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54332/0 (socket says 172.21.15.102:54332) 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.223+0000 7fdbbca00640 1 -- 172.21.15.102:0/1752039404 learned_addr learned my addr 172.21.15.102:0/1752039404 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.226+0000 7fdbbca00640 1 -- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb80cfe70 msgr2=0x7fdbb8150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.226+0000 7fdbbca00640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb80cfe70 0x7fdbb8150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.226+0000 7fdbbca00640 1 -- 172.21.15.102:0/1752039404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb80cc2f0 msgr2=0x7fdbb81e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.226+0000 7fdbbca00640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb80cc2f0 0x7fdbb81e26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.226+0000 7fdbbca00640 1 -- 172.21.15.102:0/1752039404 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb8151270 con 0x7fdbb80d0250 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.229+0000 7fdbbca00640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80d0250 0x7fdbb81591c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000c730 tx=0x7fdbb000cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.231+0000 7fdbafe00640 1 -- 172.21.15.102:0/1752039404 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0016020 con 0x7fdbb80d0250 2024-03-18T16:28:33.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.231+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb8158a00 con 0x7fdbb80d0250 2024-03-18T16:28:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.231+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb81e1450 con 0x7fdbb80d0250 2024-03-18T16:28:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.232+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbb8074fd0 con 0x7fdbb80d0250 2024-03-18T16:28:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.253+0000 7fdbafe00640 1 -- 172.21.15.102:0/1752039404 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbb0006d50 con 0x7fdbb80d0250 2024-03-18T16:28:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.253+0000 7fdbafe00640 1 -- 172.21.15.102:0/1752039404 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0003b90 con 0x7fdbb80d0250 2024-03-18T16:28:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.255+0000 7fdbafe00640 1 -- 172.21.15.102:0/1752039404 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdbb00050d0 con 0x7fdbb80d0250 2024-03-18T16:28:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.256+0000 7fdbafe00640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9806bbd0 0x7fdb9806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.256+0000 7fdbafe00640 1 -- 172.21.15.102:0/1752039404 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fdbb0057850 con 0x7fdbb80d0250 2024-03-18T16:28:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.257+0000 7fdbb7e00640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9806bbd0 0x7fdb9806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:33.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.257+0000 7fdbb7e00640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9806bbd0 0x7fdb9806e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdba8036000 tx=0x7fdba8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:33.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.509+0000 7fdbafe00640 1 -- 172.21.15.102:0/1752039404 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdbb0020020 con 0x7fdbb80d0250 2024-03-18T16:28:33.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.752+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbb81519b0 con 0x7fdbb80d0250 2024-03-18T16:28:33.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.780+0000 7fdbafe00640 1 -- 172.21.15.102:0/1752039404 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fdbb00597b0 con 0x7fdbb80d0250 2024-03-18T16:28:33.783 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:33.784 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:33.784 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.783+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9806bbd0 msgr2=0x7fdb9806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.783+0000 7fdbbd400640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9806bbd0 0x7fdb9806e090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdba8036000 tx=0x7fdba8008030 comp rx=0 tx=0).stop 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.783+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80d0250 msgr2=0x7fdbb81591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.783+0000 7fdbbd400640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80d0250 0x7fdbb81591c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fdbb000c730 tx=0x7fdbb000cb30 comp rx=0 tx=0).stop 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.784+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 shutdown_connections 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.784+0000 7fdbbd400640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb9806bbd0 0x7fdb9806e090 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.784+0000 7fdbbd400640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbb80d0250 0x7fdbb81591c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.784+0000 7fdbbd400640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdbb80cfe70 0x7fdbb8150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.784+0000 7fdbbd400640 1 --2- 172.21.15.102:0/1752039404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbb80cc2f0 0x7fdbb81e26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.784+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 >> 172.21.15.102:0/1752039404 conn(0x7fdbb808c3e0 msgr2=0x7fdbb80c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.784+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 shutdown_connections 2024-03-18T16:28:33.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:33.784+0000 7fdbbd400640 1 -- 172.21.15.102:0/1752039404 wait complete. 2024-03-18T16:28:34.406 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:34.407 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:35.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:35.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.581+0000 7f7ef5400640 1 Processor -- start 2024-03-18T16:28:35.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.581+0000 7f7ef5400640 1 -- start start 2024-03-18T16:28:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.582+0000 7f7ef5400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ef00ca960 0x7f7ef006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.582+0000 7f7ef5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef015a1a0 0x7f7ef00705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.582+0000 7f7ef5400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ef006c9c0 con 0x7f7ef00cad40 2024-03-18T16:28:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.582+0000 7f7ef5400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7ef00c8680 con 0x7f7ef00ca960 2024-03-18T16:28:35.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.582+0000 7f7ef5400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef006cca0 con 0x7f7ef015a1a0 2024-03-18T16:28:35.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.583+0000 7f7ef4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef015a1a0 0x7f7ef00705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:35.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.584+0000 7f7eef400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7ef00cad40 0x7f7ef015e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:34668/0 (socket says 172.21.15.102:34668) 2024-03-18T16:28:35.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.584+0000 7f7eef400640 1 -- 172.21.15.102:0/3087887150 learned_addr learned my addr 172.21.15.102:0/3087887150 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:35.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.584+0000 7f7eefe00640 1 --2- 172.21.15.102:0/3087887150 >> v2:172.21.15.102:3300/0 conn(0x7f7ef00ca960 0x7f7ef006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:35.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.586+0000 7f7ef4a00640 1 -- 172.21.15.102:0/3087887150 >> v2:172.21.15.102:3300/0 conn(0x7f7ef00ca960 msgr2=0x7f7ef006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:35.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.586+0000 7f7ef4a00640 1 --2- 172.21.15.102:0/3087887150 >> v2:172.21.15.102:3300/0 conn(0x7f7ef00ca960 0x7f7ef006c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:35.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.586+0000 7f7ef4a00640 1 -- 172.21.15.102:0/3087887150 >> v1:172.21.15.102:6789/0 conn(0x7f7ef00cad40 legacy=0x7f7ef015e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:35.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.586+0000 7f7ef4a00640 1 -- 172.21.15.102:0/3087887150 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef006cf80 con 0x7f7ef015a1a0 2024-03-18T16:28:35.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.588+0000 7f7ef4a00640 1 --2- 172.21.15.102:0/3087887150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef015a1a0 0x7f7ef00705a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009d90 tx=0x7f7ee802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41742895dc598573 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:35.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.593+0000 7f7eeea00640 1 -- 172.21.15.102:0/3087887150 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8039070 con 0x7f7ef015a1a0 2024-03-18T16:28:35.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.593+0000 7f7eeea00640 1 -- 172.21.15.102:0/3087887150 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee802cce0 con 0x7f7ef015a1a0 2024-03-18T16:28:35.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.593+0000 7f7eeea00640 1 -- 172.21.15.102:0/3087887150 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8034660 con 0x7f7ef015a1a0 2024-03-18T16:28:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.594+0000 7f7ef5400640 1 -- 172.21.15.102:0/3087887150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef015a1a0 msgr2=0x7f7ef00705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.594+0000 7f7ef5400640 1 --2- 172.21.15.102:0/3087887150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef015a1a0 0x7f7ef00705a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009d90 tx=0x7f7ee802bf30 comp rx=0 tx=0).stop 2024-03-18T16:28:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.594+0000 7f7ef5400640 1 -- 172.21.15.102:0/3087887150 shutdown_connections 2024-03-18T16:28:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.594+0000 7f7ef5400640 1 --2- 172.21.15.102:0/3087887150 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef015a1a0 0x7f7ef00705a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.594+0000 7f7ef5400640 1 --2- 172.21.15.102:0/3087887150 >> v2:172.21.15.102:3300/0 conn(0x7f7ef00ca960 0x7f7ef006c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.594+0000 7f7ef5400640 1 -- 172.21.15.102:0/3087887150 >> 172.21.15.102:0/3087887150 conn(0x7f7ef0067190 msgr2=0x7f7ef00675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.594+0000 7f7ef5400640 1 -- 172.21.15.102:0/3087887150 shutdown_connections 2024-03-18T16:28:35.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.594+0000 7f7ef5400640 1 -- 172.21.15.102:0/3087887150 wait complete. 2024-03-18T16:28:35.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.596+0000 7f7ef5400640 1 Processor -- start 2024-03-18T16:28:35.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.596+0000 7f7ef5400640 1 -- start start 2024-03-18T16:28:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.597+0000 7f7ef5400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef00ca960 0x7f7ef01f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.597+0000 7f7ef5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef00cad40 0x7f7ef00cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.597+0000 7f7ef5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef015a1a0 0x7f7ef0158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.597+0000 7f7ef5400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef0070ae0 con 0x7f7ef015a1a0 2024-03-18T16:28:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.597+0000 7f7ef5400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef015dc10 con 0x7f7ef00ca960 2024-03-18T16:28:35.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.597+0000 7f7ef5400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef0070c60 con 0x7f7ef00cad40 2024-03-18T16:28:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.598+0000 7f7eefe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef00ca960 0x7f7ef01f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.598+0000 7f7eef400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef00cad40 0x7f7ef00cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.598+0000 7f7eefe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef00ca960 0x7f7ef01f0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54336/0 (socket says 172.21.15.102:54336) 2024-03-18T16:28:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.598+0000 7f7eefe00640 1 -- 172.21.15.102:0/1263612972 learned_addr learned my addr 172.21.15.102:0/1263612972 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:35.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.598+0000 7f7ef4a00640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef015a1a0 0x7f7ef0158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.601+0000 7f7eefe00640 1 -- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef00cad40 msgr2=0x7f7ef00cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.601+0000 7f7eefe00640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef00cad40 0x7f7ef00cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.601+0000 7f7eefe00640 1 -- 172.21.15.102:0/1263612972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef015a1a0 msgr2=0x7f7ef0158d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.601+0000 7f7eefe00640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef015a1a0 0x7f7ef0158d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:35.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.601+0000 7f7eefe00640 1 -- 172.21.15.102:0/1263612972 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef0157630 con 0x7f7ef00ca960 2024-03-18T16:28:35.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.604+0000 7f7eefe00640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef00ca960 0x7f7ef01f0680 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000e870 tx=0x7f7ee000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:35.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.606+0000 7f7ee7e00640 1 -- 172.21.15.102:0/1263612972 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0017070 con 0x7f7ef00ca960 2024-03-18T16:28:35.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.606+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef0157910 con 0x7f7ef00ca960 2024-03-18T16:28:35.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.606+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef01f36e0 con 0x7f7ef00ca960 2024-03-18T16:28:35.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.607+0000 7f7ee7e00640 1 -- 172.21.15.102:0/1263612972 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee000c940 con 0x7f7ef00ca960 2024-03-18T16:28:35.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.607+0000 7f7ee7e00640 1 -- 172.21.15.102:0/1263612972 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0021060 con 0x7f7ef00ca960 2024-03-18T16:28:35.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.608+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ec8005290 con 0x7f7ef00ca960 2024-03-18T16:28:35.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.867+0000 7f7ee7e00640 1 -- 172.21.15.102:0/1263612972 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7ee001a070 con 0x7f7ef00ca960 2024-03-18T16:28:35.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.869+0000 7f7ee7e00640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ed006bbd0 0x7f7ed006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.869+0000 7f7eef400640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ed006bbd0 0x7f7ed006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:35.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.869+0000 7f7ee7e00640 1 -- 172.21.15.102:0/1263612972 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7ee0057770 con 0x7f7ef00ca960 2024-03-18T16:28:35.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.870+0000 7f7eef400640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ed006bbd0 0x7f7ed006e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00678c0 tx=0x7f7edc007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:35.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:35.880+0000 7f7ee7e00640 1 -- 172.21.15.102:0/1263612972 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7ee00596d0 con 0x7f7ef00ca960 2024-03-18T16:28:36.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.125+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ec8005580 con 0x7f7ef00ca960 2024-03-18T16:28:36.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.141+0000 7f7ee7e00640 1 -- 172.21.15.102:0/1263612972 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7ee005d770 con 0x7f7ef00ca960 2024-03-18T16:28:36.144 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:36.144 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:36.145 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:36.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ed006bbd0 msgr2=0x7f7ed006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:36.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ed006bbd0 0x7f7ed006e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00678c0 tx=0x7f7edc007e80 comp rx=0 tx=0).stop 2024-03-18T16:28:36.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef00ca960 msgr2=0x7f7ef01f0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:36.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef00ca960 0x7f7ef01f0680 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7ee000e870 tx=0x7f7ee000ed30 comp rx=0 tx=0).stop 2024-03-18T16:28:36.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 shutdown_connections 2024-03-18T16:28:36.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ed006bbd0 0x7f7ed006e090 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef015a1a0 0x7f7ef0158d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ef00cad40 0x7f7ef00cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 --2- 172.21.15.102:0/1263612972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ef00ca960 0x7f7ef01f0680 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 >> 172.21.15.102:0/1263612972 conn(0x7f7ef0067190 msgr2=0x7f7ef0142500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 shutdown_connections 2024-03-18T16:28:36.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:36.145+0000 7f7ef5400640 1 -- 172.21.15.102:0/1263612972 wait complete. 2024-03-18T16:28:36.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:36.768 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:37.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:37.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.942+0000 7f8067200640 1 Processor -- start 2024-03-18T16:28:37.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.942+0000 7f8067200640 1 -- start start 2024-03-18T16:28:37.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.943+0000 7f8067200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f80680cfe70 0x7f806815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:37.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.943+0000 7f8067200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f80680cc2f0 0x7f80680cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:37.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.943+0000 7f8067200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80680ccc80 con 0x7f80680d0250 2024-03-18T16:28:37.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.943+0000 7f8067200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f80680cabd0 con 0x7f80680cfe70 2024-03-18T16:28:37.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.943+0000 7f8067200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f80680caea0 con 0x7f80680cc2f0 2024-03-18T16:28:37.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.944+0000 7f8065e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f80680cfe70 0x7f806815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.944+0000 7f8066800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f80680cc2f0 0x7f80680cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.944+0000 7f8065400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f80680d0250 0x7f80680c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:49154/0 (socket says 172.21.15.102:49154) 2024-03-18T16:28:37.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.944+0000 7f8065400640 1 -- 172.21.15.102:0/2347270278 learned_addr learned my addr 172.21.15.102:0/2347270278 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:37.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.947+0000 7f8065e00640 1 -- 172.21.15.102:0/2347270278 >> v2:172.21.15.154:3300/0 conn(0x7f80680cc2f0 msgr2=0x7f80680cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:37.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.947+0000 7f8065e00640 1 --2- 172.21.15.102:0/2347270278 >> v2:172.21.15.154:3300/0 conn(0x7f80680cc2f0 0x7f80680cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:37.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.947+0000 7f8065e00640 1 -- 172.21.15.102:0/2347270278 >> v1:172.21.15.154:6789/0 conn(0x7f80680d0250 legacy=0x7f80680c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:37.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.947+0000 7f8065e00640 1 -- 172.21.15.102:0/2347270278 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80680cb180 con 0x7f80680cfe70 2024-03-18T16:28:37.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.950+0000 7f8065e00640 1 --2- 172.21.15.102:0/2347270278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 0x7f806815ae40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80680ca160 tx=0x7f805802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=328c39c9de1bc98a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.954+0000 7f8064a00640 1 -- 172.21.15.102:0/2347270278 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058034070 con 0x7f80680cfe70 2024-03-18T16:28:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.954+0000 7f8064a00640 1 -- 172.21.15.102:0/2347270278 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8058030610 con 0x7f80680cfe70 2024-03-18T16:28:37.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.954+0000 7f8064a00640 1 -- 172.21.15.102:0/2347270278 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80580309e0 con 0x7f80680cfe70 2024-03-18T16:28:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.955+0000 7f8067200640 1 -- 172.21.15.102:0/2347270278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 msgr2=0x7f806815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.955+0000 7f8067200640 1 --2- 172.21.15.102:0/2347270278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 0x7f806815ae40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f80680ca160 tx=0x7f805802b960 comp rx=0 tx=0).stop 2024-03-18T16:28:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.955+0000 7f8067200640 1 -- 172.21.15.102:0/2347270278 shutdown_connections 2024-03-18T16:28:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.956+0000 7f8067200640 1 --2- 172.21.15.102:0/2347270278 >> v2:172.21.15.154:3300/0 conn(0x7f80680cc2f0 0x7f80680cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.956+0000 7f8067200640 1 --2- 172.21.15.102:0/2347270278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 0x7f806815ae40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.956+0000 7f8067200640 1 -- 172.21.15.102:0/2347270278 >> 172.21.15.102:0/2347270278 conn(0x7f806808c3e0 msgr2=0x7f806806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.956+0000 7f8067200640 1 -- 172.21.15.102:0/2347270278 shutdown_connections 2024-03-18T16:28:37.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.956+0000 7f8067200640 1 -- 172.21.15.102:0/2347270278 wait complete. 2024-03-18T16:28:37.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.958+0000 7f8067200640 1 Processor -- start 2024-03-18T16:28:37.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.958+0000 7f8067200640 1 -- start start 2024-03-18T16:28:37.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.960+0000 7f8067200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80680cc2f0 0x7f80681e27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:37.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.960+0000 7f8067200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 0x7f80681e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:37.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.960+0000 7f8067200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80680d0250 0x7f8068150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:37.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.960+0000 7f8067200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f80680cba20 con 0x7f80680cfe70 2024-03-18T16:28:37.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.960+0000 7f8067200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80680cb740 con 0x7f80680d0250 2024-03-18T16:28:37.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.960+0000 7f8067200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f80680cbd00 con 0x7f80680cc2f0 2024-03-18T16:28:37.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.961+0000 7f8065400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 0x7f80681e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:37.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.961+0000 7f8065e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80680cc2f0 0x7f80681e27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:37.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.961+0000 7f8066800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80680d0250 0x7f8068150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:37.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.961+0000 7f8065400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 0x7f80681e2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48170/0 (socket says 172.21.15.102:48170) 2024-03-18T16:28:37.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.961+0000 7f8065400640 1 -- 172.21.15.102:0/902371025 learned_addr learned my addr 172.21.15.102:0/902371025 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:37.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.965+0000 7f8065400640 1 -- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80680cc2f0 msgr2=0x7f80681e27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:37.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.965+0000 7f8065400640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80680cc2f0 0x7f80681e27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:37.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.965+0000 7f8065400640 1 -- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80680d0250 msgr2=0x7f8068150ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:37.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.965+0000 7f8065400640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80680d0250 0x7f8068150ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:37.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.965+0000 7f8065400640 1 -- 172.21.15.102:0/902371025 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8068156e30 con 0x7f80680cfe70 2024-03-18T16:28:37.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.967+0000 7f8065400640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 0x7f80681e2cf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f805400c470 tx=0x7f805400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:37.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.970+0000 7f805e000640 1 -- 172.21.15.102:0/902371025 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054017070 con 0x7f80680cfe70 2024-03-18T16:28:37.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.970+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8068157110 con 0x7f80680cfe70 2024-03-18T16:28:37.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.970+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8068157ec0 con 0x7f80680cfe70 2024-03-18T16:28:37.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.972+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8068074fd0 con 0x7f80680cfe70 2024-03-18T16:28:37.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.973+0000 7f805e000640 1 -- 172.21.15.102:0/902371025 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8054005ce0 con 0x7f80680cfe70 2024-03-18T16:28:37.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:37.973+0000 7f805e000640 1 -- 172.21.15.102:0/902371025 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054021060 con 0x7f80680cfe70 2024-03-18T16:28:38.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.277+0000 7f805e000640 1 -- 172.21.15.102:0/902371025 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f805401a070 con 0x7f80680cfe70 2024-03-18T16:28:38.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.279+0000 7f805e000640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f804806bbd0 0x7f804806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:38.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.280+0000 7f8065e00640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f804806bbd0 0x7f804806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:38.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.280+0000 7f8065e00640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f804806bbd0 0x7f804806e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8058036000 tx=0x7f805803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:38.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.300+0000 7f805e000640 1 -- 172.21.15.102:0/902371025 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8054056750 con 0x7f80680cfe70 2024-03-18T16:28:38.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.301+0000 7f805e000640 1 -- 172.21.15.102:0/902371025 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f80540586b0 con 0x7f80680cfe70 2024-03-18T16:28:38.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.535+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8068157c70 con 0x7f80680cfe70 2024-03-18T16:28:38.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.550+0000 7f805e000640 1 -- 172.21.15.102:0/902371025 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f805405c750 con 0x7f80680cfe70 2024-03-18T16:28:38.553 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:38.553 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:38.553 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f804806bbd0 msgr2=0x7f804806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f804806bbd0 0x7f804806e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8058036000 tx=0x7f805803b040 comp rx=0 tx=0).stop 2024-03-18T16:28:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 msgr2=0x7f80681e2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:38.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 0x7f80681e2cf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f805400c470 tx=0x7f805400c930 comp rx=0 tx=0).stop 2024-03-18T16:28:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 shutdown_connections 2024-03-18T16:28:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f804806bbd0 0x7f804806e090 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8058036000 tx=0x7f805803b040 comp rx=0 tx=0).stop 2024-03-18T16:28:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80680d0250 0x7f8068150ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80680cfe70 0x7f80681e2cf0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 --2- 172.21.15.102:0/902371025 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80680cc2f0 0x7f80681e27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.552+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 >> 172.21.15.102:0/902371025 conn(0x7f806808c3e0 msgr2=0x7f8068158860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.553+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 shutdown_connections 2024-03-18T16:28:38.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:38.553+0000 7f8067200640 1 -- 172.21.15.102:0/902371025 wait complete. 2024-03-18T16:28:39.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:39.176 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:40.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.349+0000 7f1352200640 1 Processor -- start 2024-03-18T16:28:40.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.349+0000 7f1352200640 1 -- start start 2024-03-18T16:28:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.350+0000 7f1352200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f134c0cca00 0x7f134c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.350+0000 7f1352200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f134c0cabd0 con 0x7f134c0cfe70 2024-03-18T16:28:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.350+0000 7f1352200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f134c0caea0 con 0x7f134c0d0250 2024-03-18T16:28:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.350+0000 7f1352200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f134c0cb180 con 0x7f134c0cca00 2024-03-18T16:28:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.351+0000 7f134be00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f134c0d0250 0x7f134c0cc2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:49166/0 (socket says 172.21.15.102:49166) 2024-03-18T16:28:40.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.351+0000 7f134be00640 1 -- 172.21.15.102:0/2599142616 learned_addr learned my addr 172.21.15.102:0/2599142616 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:40.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.352+0000 7f1351800640 1 --2- 172.21.15.102:0/2599142616 >> v2:172.21.15.102:3300/0 conn(0x7f134c0cca00 0x7f134c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:40.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.354+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2021430540 0 0) 0x7f134c0caea0 con 0x7f134c0d0250 2024-03-18T16:28:40.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.354+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1334003520 con 0x7f134c0d0250 2024-03-18T16:28:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.356+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 323751058 0 0) 0x7f134c0cabd0 con 0x7f134c0cfe70 2024-03-18T16:28:40.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.356+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f134c0caea0 con 0x7f134c0cfe70 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.357+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1488060760 0 0) 0x7f1334003520 con 0x7f134c0d0250 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.357+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 >> v2:172.21.15.102:3300/0 conn(0x7f134c0cca00 msgr2=0x7f134c15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.357+0000 7f134b400640 1 --2- 172.21.15.102:0/2599142616 >> v2:172.21.15.102:3300/0 conn(0x7f134c0cca00 0x7f134c15ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.357+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 >> v1:172.21.15.102:6789/0 conn(0x7f134c0cfe70 legacy=0x7f134c0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.357+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c0cb460 con 0x7f134c0d0250 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.357+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f1338001670 con 0x7f134c0d0250 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.358+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 >> v1:172.21.15.154:6789/0 conn(0x7f134c0d0250 legacy=0x7f134c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.358+0000 7f134b400640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1334003b40 0x7f1334003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.358+0000 7f134b400640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1334004800 0x7f1334004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.358+0000 7f134b400640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13340051c0 0x7f133400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.358+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f134c0cb180 con 0x7f1334003b40 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.358+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f133400bfb0 con 0x7f1334004800 2024-03-18T16:28:40.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.358+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f133400c1b0 con 0x7f13340051c0 2024-03-18T16:28:40.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.359+0000 7f1351800640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13340051c0 0x7f133400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:40.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.359+0000 7f134be00640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1334004800 0x7f1334004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:40.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.362+0000 7f1351800640 1 -- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1334004800 msgr2=0x7f1334004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:40.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.362+0000 7f1351800640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1334004800 0x7f1334004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.362+0000 7f1351800640 1 -- 172.21.15.102:0/2599142616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1334003b40 msgr2=0x7f1334003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.362+0000 7f1351800640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1334003b40 0x7f1334003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.362+0000 7f1351800640 1 -- 172.21.15.102:0/2599142616 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c0cb460 con 0x7f13340051c0 2024-03-18T16:28:40.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.363+0000 7f1351800640 1 -- 172.21.15.102:0/2599142616 reap_dead start 2024-03-18T16:28:40.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.364+0000 7f1351800640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13340051c0 0x7f133400b9c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1344009fc0 tx=0x7f1334005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc79d8a37522b2e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:40.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.368+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344009c10 con 0x7f13340051c0 2024-03-18T16:28:40.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.368+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f134402cc30 con 0x7f13340051c0 2024-03-18T16:28:40.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.368+0000 7f134b400640 1 -- 172.21.15.102:0/2599142616 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344003990 con 0x7f13340051c0 2024-03-18T16:28:40.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.369+0000 7f1352200640 1 -- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13340051c0 msgr2=0x7f133400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:40.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.369+0000 7f1352200640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13340051c0 0x7f133400b9c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1344009fc0 tx=0x7f1334005830 comp rx=0 tx=0).stop 2024-03-18T16:28:40.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.369+0000 7f1352200640 1 -- 172.21.15.102:0/2599142616 shutdown_connections 2024-03-18T16:28:40.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.369+0000 7f1352200640 1 --2- 172.21.15.102:0/2599142616 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13340051c0 0x7f133400b9c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.369+0000 7f1352200640 1 -- 172.21.15.102:0/2599142616 >> 172.21.15.102:0/2599142616 conn(0x7f134c08c3e0 msgr2=0x7f134c06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:40.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.369+0000 7f1352200640 1 -- 172.21.15.102:0/2599142616 shutdown_connections 2024-03-18T16:28:40.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.369+0000 7f1352200640 1 -- 172.21.15.102:0/2599142616 wait complete. 2024-03-18T16:28:40.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.371+0000 7f1352200640 1 Processor -- start 2024-03-18T16:28:40.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.371+0000 7f1352200640 1 -- start start 2024-03-18T16:28:40.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.372+0000 7f1352200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13340051c0 0x7f134c1e2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:40.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.372+0000 7f1352200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f134c158e60 0x7f134c150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:40.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.372+0000 7f1352200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134c0c4430 0x7f134c0c48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:40.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.372+0000 7f1352200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f134c0cbd00 con 0x7f13340051c0 2024-03-18T16:28:40.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.372+0000 7f1352200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f134c0cba20 con 0x7f134c0c4430 2024-03-18T16:28:40.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.372+0000 7f1352200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f134c15ae40 con 0x7f134c158e60 2024-03-18T16:28:40.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.373+0000 7f1350e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13340051c0 0x7f134c1e2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:40.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.373+0000 7f134be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f134c158e60 0x7f134c150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:40.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.373+0000 7f1351800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134c0c4430 0x7f134c0c48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:40.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.373+0000 7f1350e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13340051c0 0x7f134c1e2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48206/0 (socket says 172.21.15.102:48206) 2024-03-18T16:28:40.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.373+0000 7f1350e00640 1 -- 172.21.15.102:0/2753741649 learned_addr learned my addr 172.21.15.102:0/2753741649 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:40.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.377+0000 7f1351800640 1 -- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f134c158e60 msgr2=0x7f134c150eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:40.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.377+0000 7f1351800640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f134c158e60 0x7f134c150eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.377+0000 7f1351800640 1 -- 172.21.15.102:0/2753741649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13340051c0 msgr2=0x7f134c1e2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:40.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.377+0000 7f1351800640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13340051c0 0x7f134c1e2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.377+0000 7f1351800640 1 -- 172.21.15.102:0/2753741649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c1f3040 con 0x7f134c0c4430 2024-03-18T16:28:40.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.379+0000 7f1351800640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134c0c4430 0x7f134c0c48b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f134c0ca920 tx=0x7f1344009750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:40.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.382+0000 7f1348c00640 1 -- 172.21.15.102:0/2753741649 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344009c10 con 0x7f134c0c4430 2024-03-18T16:28:40.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.382+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134c1f32e0 con 0x7f134c0c4430 2024-03-18T16:28:40.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.382+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134c1f38a0 con 0x7f134c0c4430 2024-03-18T16:28:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.384+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134c074fd0 con 0x7f134c0c4430 2024-03-18T16:28:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.404+0000 7f1348c00640 1 -- 172.21.15.102:0/2753741649 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1344045070 con 0x7f134c0c4430 2024-03-18T16:28:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.404+0000 7f1348c00640 1 -- 172.21.15.102:0/2753741649 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344003990 con 0x7f134c0c4430 2024-03-18T16:28:40.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.405+0000 7f1348c00640 1 -- 172.21.15.102:0/2753741649 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1344005d10 con 0x7f134c0c4430 2024-03-18T16:28:40.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.406+0000 7f1348c00640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f132c06bc00 0x7f132c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:40.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.407+0000 7f1350e00640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f132c06bc00 0x7f132c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:40.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.407+0000 7f1348c00640 1 -- 172.21.15.102:0/2753741649 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f134407a630 con 0x7f134c0c4430 2024-03-18T16:28:40.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.408+0000 7f1350e00640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f132c06bc00 0x7f132c06e0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1340000f60 tx=0x7f1340007e10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:40.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.656+0000 7f1348c00640 1 -- 172.21.15.102:0/2753741649 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f134407cc90 con 0x7f134c0c4430 2024-03-18T16:28:40.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.932+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f134c0cd3c0 con 0x7f134c0c4430 2024-03-18T16:28:40.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.945+0000 7f1348c00640 1 -- 172.21.15.102:0/2753741649 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f134407c590 con 0x7f134c0c4430 2024-03-18T16:28:40.947 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:40.947 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:40.948 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f132c06bc00 msgr2=0x7f132c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f132c06bc00 0x7f132c06e0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1340000f60 tx=0x7f1340007e10 comp rx=0 tx=0).stop 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134c0c4430 msgr2=0x7f134c0c48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134c0c4430 0x7f134c0c48b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f134c0ca920 tx=0x7f1344009750 comp rx=0 tx=0).stop 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 shutdown_connections 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f132c06bc00 0x7f132c06e0c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134c0c4430 0x7f134c0c48b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f134c158e60 0x7f134c150eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 --2- 172.21.15.102:0/2753741649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13340051c0 0x7f134c1e2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 >> 172.21.15.102:0/2753741649 conn(0x7f134c08c3e0 msgr2=0x7f134c0c6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 shutdown_connections 2024-03-18T16:28:40.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:40.949+0000 7f1352200640 1 -- 172.21.15.102:0/2753741649 wait complete. 2024-03-18T16:28:41.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:41.572 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:42.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:42.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.747+0000 7fa565e00640 1 Processor -- start 2024-03-18T16:28:42.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.748+0000 7fa565e00640 1 -- start start 2024-03-18T16:28:42.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.749+0000 7fa565e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa56006b470 0x7fa56015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:42.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.749+0000 7fa565e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa56006de50 0x7fa56006b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:42.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.749+0000 7fa565e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006bd90 0x7fa560154150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:42.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.749+0000 7fa565e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa5600ca0a0 con 0x7fa56006b470 2024-03-18T16:28:42.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.749+0000 7fa565e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa5600c94b0 con 0x7fa56006de50 2024-03-18T16:28:42.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.749+0000 7fa565e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa5600c9770 con 0x7fa56006bd90 2024-03-18T16:28:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.750+0000 7fa564a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa56006b470 0x7fa56015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.750+0000 7fa565400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006bd90 0x7fa560154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.750+0000 7fa55fe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa56006de50 0x7fa56006b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.750+0000 7fa564a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa56006b470 0x7fa56015ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37488/0 (socket says 172.21.15.102:37488) 2024-03-18T16:28:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.750+0000 7fa565400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006bd90 0x7fa560154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34524/0 (socket says 172.21.15.102:34524) 2024-03-18T16:28:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.750+0000 7fa564a00640 1 -- 172.21.15.102:0/4179031308 learned_addr learned my addr 172.21.15.102:0/4179031308 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.752+0000 7fa565400640 1 -- 172.21.15.102:0/4179031308 >> v2:172.21.15.154:3300/0 conn(0x7fa56006de50 msgr2=0x7fa56006b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.753+0000 7fa565400640 1 --2- 172.21.15.102:0/4179031308 >> v2:172.21.15.154:3300/0 conn(0x7fa56006de50 0x7fa56006b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:42.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.753+0000 7fa565400640 1 -- 172.21.15.102:0/4179031308 >> v2:172.21.15.102:3300/0 conn(0x7fa56006b470 msgr2=0x7fa56015ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:42.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.753+0000 7fa565400640 1 --2- 172.21.15.102:0/4179031308 >> v2:172.21.15.102:3300/0 conn(0x7fa56006b470 0x7fa56015ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:42.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.753+0000 7fa565400640 1 -- 172.21.15.102:0/4179031308 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5600c9a50 con 0x7fa56006bd90 2024-03-18T16:28:42.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.755+0000 7fa565400640 1 --2- 172.21.15.102:0/4179031308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006bd90 0x7fa560154150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa56006c1e0 tx=0x7fa55802be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=315a55fc7e1dc669 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:42.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.759+0000 7fa55f400640 1 -- 172.21.15.102:0/4179031308 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558039070 con 0x7fa56006bd90 2024-03-18T16:28:42.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.759+0000 7fa55f400640 1 -- 172.21.15.102:0/4179031308 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55802cbd0 con 0x7fa56006bd90 2024-03-18T16:28:42.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.759+0000 7fa55f400640 1 -- 172.21.15.102:0/4179031308 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558034440 con 0x7fa56006bd90 2024-03-18T16:28:42.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.760+0000 7fa565e00640 1 -- 172.21.15.102:0/4179031308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006bd90 msgr2=0x7fa560154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:42.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.760+0000 7fa565e00640 1 --2- 172.21.15.102:0/4179031308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006bd90 0x7fa560154150 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa56006c1e0 tx=0x7fa55802be20 comp rx=0 tx=0).stop 2024-03-18T16:28:42.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.760+0000 7fa565e00640 1 -- 172.21.15.102:0/4179031308 shutdown_connections 2024-03-18T16:28:42.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.760+0000 7fa565e00640 1 --2- 172.21.15.102:0/4179031308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006bd90 0x7fa560154150 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:42.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.760+0000 7fa565e00640 1 --2- 172.21.15.102:0/4179031308 >> v2:172.21.15.154:3300/0 conn(0x7fa56006de50 0x7fa56006b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:42.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.760+0000 7fa565e00640 1 --2- 172.21.15.102:0/4179031308 >> v2:172.21.15.102:3300/0 conn(0x7fa56006b470 0x7fa56015ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:42.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.760+0000 7fa565e00640 1 -- 172.21.15.102:0/4179031308 >> 172.21.15.102:0/4179031308 conn(0x7fa5600727e0 msgr2=0x7fa560072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:42.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.760+0000 7fa565e00640 1 -- 172.21.15.102:0/4179031308 shutdown_connections 2024-03-18T16:28:42.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.760+0000 7fa565e00640 1 -- 172.21.15.102:0/4179031308 wait complete. 2024-03-18T16:28:42.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.763+0000 7fa565e00640 1 Processor -- start 2024-03-18T16:28:42.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.763+0000 7fa565e00640 1 -- start start 2024-03-18T16:28:42.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.765+0000 7fa565e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56006b470 0x7fa5601e1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.765+0000 7fa565e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa56006bd90 0x7fa5601e2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.765+0000 7fa565e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006de50 0x7fa5601f0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.765+0000 7fa565e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa560154810 con 0x7fa56006bd90 2024-03-18T16:28:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.765+0000 7fa565e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa560154690 con 0x7fa56006b470 2024-03-18T16:28:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.765+0000 7fa565e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa560154990 con 0x7fa56006de50 2024-03-18T16:28:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.765+0000 7fa55fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa56006bd90 0x7fa5601e2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:42.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.765+0000 7fa565400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006de50 0x7fa5601f0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:42.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.766+0000 7fa564a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56006b470 0x7fa5601e1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:42.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.766+0000 7fa565400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006de50 0x7fa5601f0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34530/0 (socket says 172.21.15.102:34530) 2024-03-18T16:28:42.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.766+0000 7fa565400640 1 -- 172.21.15.102:0/3506438360 learned_addr learned my addr 172.21.15.102:0/3506438360 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:42.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.769+0000 7fa565400640 1 -- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56006b470 msgr2=0x7fa5601e1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:42.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.769+0000 7fa565400640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56006b470 0x7fa5601e1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:42.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.769+0000 7fa565400640 1 -- 172.21.15.102:0/3506438360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa56006bd90 msgr2=0x7fa5601e2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:42.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.769+0000 7fa565400640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa56006bd90 0x7fa5601e2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:42.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.769+0000 7fa565400640 1 -- 172.21.15.102:0/3506438360 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5601f3be0 con 0x7fa56006de50 2024-03-18T16:28:42.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.772+0000 7fa565400640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006de50 0x7fa5601f0db0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa55802c430 tx=0x7fa558004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:42.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.776+0000 7fa55cc00640 1 -- 172.21.15.102:0/3506438360 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558039050 con 0x7fa56006de50 2024-03-18T16:28:42.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.776+0000 7fa55cc00640 1 -- 172.21.15.102:0/3506438360 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa558045070 con 0x7fa56006de50 2024-03-18T16:28:42.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.776+0000 7fa55cc00640 1 -- 172.21.15.102:0/3506438360 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa558009450 con 0x7fa56006de50 2024-03-18T16:28:42.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.776+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5601e0680 con 0x7fa56006de50 2024-03-18T16:28:42.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.777+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5601f12f0 con 0x7fa56006de50 2024-03-18T16:28:42.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.778+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5601e7250 con 0x7fa56006de50 2024-03-18T16:28:42.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.798+0000 7fa55cc00640 1 -- 172.21.15.102:0/3506438360 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa55803e850 con 0x7fa56006de50 2024-03-18T16:28:42.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.799+0000 7fa55cc00640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc50 0x7fa54006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:42.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.800+0000 7fa564a00640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc50 0x7fa54006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:42.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:42.800+0000 7fa564a00640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc50 0x7fa54006e110 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa554006cc0 tx=0x7fa554009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:43.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.050+0000 7fa55cc00640 1 -- 172.21.15.102:0/3506438360 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa55807a610 con 0x7fa56006de50 2024-03-18T16:28:43.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.061+0000 7fa55cc00640 1 -- 172.21.15.102:0/3506438360 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa5601e7250 con 0x7fa56006de50 2024-03-18T16:28:43.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.477+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa56006f9a0 con 0x7fa56006de50 2024-03-18T16:28:43.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.492+0000 7fa55cc00640 1 -- 172.21.15.102:0/3506438360 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa55807c570 con 0x7fa56006de50 2024-03-18T16:28:43.494 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:43.494 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:43.494 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:43.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc50 msgr2=0x7fa54006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:43.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc50 0x7fa54006e110 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa554006cc0 tx=0x7fa554009240 comp rx=0 tx=0).stop 2024-03-18T16:28:43.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006de50 msgr2=0x7fa5601f0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:43.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006de50 0x7fa5601f0db0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa55802c430 tx=0x7fa558004a40 comp rx=0 tx=0).stop 2024-03-18T16:28:43.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 shutdown_connections 2024-03-18T16:28:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa54006bc50 0x7fa54006e110 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa56006de50 0x7fa5601f0db0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa56006bd90 0x7fa5601e2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 --2- 172.21.15.102:0/3506438360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa56006b470 0x7fa5601e1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.495+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 >> 172.21.15.102:0/3506438360 conn(0x7fa5600727e0 msgr2=0x7fa5600c7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.496+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 shutdown_connections 2024-03-18T16:28:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:43.496+0000 7fa565e00640 1 -- 172.21.15.102:0/3506438360 wait complete. 2024-03-18T16:28:44.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:44.119 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:45.119 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:45.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.298+0000 7f6fd0e00640 1 Processor -- start 2024-03-18T16:28:45.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.298+0000 7f6fd0e00640 1 -- start start 2024-03-18T16:28:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.299+0000 7f6fd0e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6fcc06de50 0x7f6fcc15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:45.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.299+0000 7f6fd0e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fcc06bf60 con 0x7f6fcc06b470 2024-03-18T16:28:45.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.299+0000 7f6fd0e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fcc0c94b0 con 0x7f6fcc154150 2024-03-18T16:28:45.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.299+0000 7f6fd0e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6fcc0ca0a0 con 0x7f6fcc06de50 2024-03-18T16:28:45.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.311+0000 7f6fcbe00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f6fcc154150 0x7f6fcc0c5190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58734/0 (socket says 172.21.15.102:58734) 2024-03-18T16:28:45.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.311+0000 7f6fcbe00640 1 -- 172.21.15.102:0/2344089885 learned_addr learned my addr 172.21.15.102:0/2344089885 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:45.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.312+0000 7f6fcaa00640 1 --2- 172.21.15.102:0/2344089885 >> v2:172.21.15.154:3300/0 conn(0x7f6fcc06de50 0x7f6fcc15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:45.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.316+0000 7f6fcaa00640 1 -- 172.21.15.102:0/2344089885 >> v1:172.21.15.154:6789/0 conn(0x7f6fcc154150 legacy=0x7f6fcc0c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:45.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.316+0000 7f6fcaa00640 1 -- 172.21.15.102:0/2344089885 >> v1:172.21.15.102:6789/0 conn(0x7f6fcc06b470 legacy=0x7f6fcc06b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:45.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.316+0000 7f6fcaa00640 1 -- 172.21.15.102:0/2344089885 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc0c9770 con 0x7f6fcc06de50 2024-03-18T16:28:45.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.321+0000 7f6fcaa00640 1 --2- 172.21.15.102:0/2344089885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 0x7f6fcc15ad60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80097a0 tx=0x7f6fb802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98254ade9019e76c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.325+0000 7f6fca000640 1 -- 172.21.15.102:0/2344089885 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8035020 con 0x7f6fcc06de50 2024-03-18T16:28:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.326+0000 7f6fca000640 1 -- 172.21.15.102:0/2344089885 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb8031830 con 0x7f6fcc06de50 2024-03-18T16:28:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.326+0000 7f6fca000640 1 -- 172.21.15.102:0/2344089885 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb8031c00 con 0x7f6fcc06de50 2024-03-18T16:28:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.326+0000 7f6fd0e00640 1 -- 172.21.15.102:0/2344089885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 msgr2=0x7f6fcc15ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.326+0000 7f6fd0e00640 1 --2- 172.21.15.102:0/2344089885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 0x7f6fcc15ad60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80097a0 tx=0x7f6fb802b8d0 comp rx=0 tx=0).stop 2024-03-18T16:28:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.326+0000 7f6fd0e00640 1 -- 172.21.15.102:0/2344089885 shutdown_connections 2024-03-18T16:28:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.326+0000 7f6fd0e00640 1 --2- 172.21.15.102:0/2344089885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 0x7f6fcc15ad60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.326+0000 7f6fd0e00640 1 -- 172.21.15.102:0/2344089885 >> 172.21.15.102:0/2344089885 conn(0x7f6fcc0727e0 msgr2=0x7f6fcc072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.327+0000 7f6fd0e00640 1 -- 172.21.15.102:0/2344089885 shutdown_connections 2024-03-18T16:28:45.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.327+0000 7f6fd0e00640 1 -- 172.21.15.102:0/2344089885 wait complete. 2024-03-18T16:28:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.328+0000 7f6fd0e00640 1 Processor -- start 2024-03-18T16:28:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.328+0000 7f6fd0e00640 1 -- start start 2024-03-18T16:28:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.330+0000 7f6fd0e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fcc06b470 0x7f6fcc15a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.330+0000 7f6fd0e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 0x7f6fcc14a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.330+0000 7f6fd0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fcc154150 0x7f6fcc1519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.330+0000 7f6fd0e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc0c58a0 con 0x7f6fcc154150 2024-03-18T16:28:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.330+0000 7f6fd0e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fcc0c9d30 con 0x7f6fcc06de50 2024-03-18T16:28:45.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.330+0000 7f6fd0e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6fcc0c5a20 con 0x7f6fcc06b470 2024-03-18T16:28:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.331+0000 7f6fcb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fcc06b470 0x7f6fcc15a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.331+0000 7f6fcaa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 0x7f6fcc14a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.331+0000 7f6fcb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fcc06b470 0x7f6fcc15a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:34542/0 (socket says 172.21.15.102:34542) 2024-03-18T16:28:45.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.331+0000 7f6fcb400640 1 -- 172.21.15.102:0/1165498258 learned_addr learned my addr 172.21.15.102:0/1165498258 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:45.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.335+0000 7f6fcaa00640 1 -- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fcc06b470 msgr2=0x7f6fcc15a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.335+0000 7f6fcaa00640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fcc06b470 0x7f6fcc15a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.335+0000 7f6fcaa00640 1 -- 172.21.15.102:0/1165498258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fcc154150 msgr2=0x7f6fcc1519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.335+0000 7f6fcaa00640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fcc154150 0x7f6fcc1519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.335+0000 7f6fcaa00640 1 -- 172.21.15.102:0/1165498258 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fcc1512b0 con 0x7f6fcc06de50 2024-03-18T16:28:45.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.335+0000 7f6fcb400640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fcc06b470 0x7f6fcc15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:28:45.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.338+0000 7f6fcaa00640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 0x7f6fcc14a0e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8033000 tx=0x7f6fb80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:45.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.340+0000 7f6fc3400640 1 -- 172.21.15.102:0/1165498258 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb803ba80 con 0x7f6fcc06de50 2024-03-18T16:28:45.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.340+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fcc1f0cb0 con 0x7f6fcc06de50 2024-03-18T16:28:45.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.340+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fcc1f01d0 con 0x7f6fcc06de50 2024-03-18T16:28:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.342+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fcc06b850 con 0x7f6fcc06de50 2024-03-18T16:28:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.562+0000 7f6fc3400640 1 -- 172.21.15.102:0/1165498258 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb8031830 con 0x7f6fcc06de50 2024-03-18T16:28:45.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.562+0000 7f6fc3400640 1 -- 172.21.15.102:0/1165498258 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb803be60 con 0x7f6fcc06de50 2024-03-18T16:28:45.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.612+0000 7f6fc3400640 1 -- 172.21.15.102:0/1165498258 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6fb804c060 con 0x7f6fcc06de50 2024-03-18T16:28:45.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.614+0000 7f6fc3400640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fac06bbd0 0x7f6fac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:45.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.615+0000 7f6fc3400640 1 -- 172.21.15.102:0/1165498258 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6fb80acf40 con 0x7f6fcc06de50 2024-03-18T16:28:45.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.615+0000 7f6fcb400640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fac06bbd0 0x7f6fac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:45.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.615+0000 7f6fcb400640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fac06bbd0 0x7f6fac06e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc001c80 tx=0x7f6fbc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:45.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.624+0000 7f6fc3400640 1 -- 172.21.15.102:0/1165498258 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6fcc06b850 con 0x7f6fcc06de50 2024-03-18T16:28:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.978+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fcc0c3d40 con 0x7f6fcc06de50 2024-03-18T16:28:45.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.991+0000 7f6fc3400640 1 -- 172.21.15.102:0/1165498258 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f6fb8040070 con 0x7f6fcc06de50 2024-03-18T16:28:45.993 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:45.993 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:45.993 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.993+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fac06bbd0 msgr2=0x7f6fac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.993+0000 7f6fd0e00640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fac06bbd0 0x7f6fac06e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc001c80 tx=0x7f6fbc009040 comp rx=0 tx=0).stop 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.993+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 msgr2=0x7f6fcc14a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.993+0000 7f6fd0e00640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 0x7f6fcc14a0e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6fb8033000 tx=0x7f6fb80046c0 comp rx=0 tx=0).stop 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.994+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 shutdown_connections 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.994+0000 7f6fd0e00640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6fac06bbd0 0x7f6fac06e090 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.994+0000 7f6fd0e00640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6fcc154150 0x7f6fcc1519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.994+0000 7f6fd0e00640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fcc06de50 0x7f6fcc14a0e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.994+0000 7f6fd0e00640 1 --2- 172.21.15.102:0/1165498258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6fcc06b470 0x7f6fcc15a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:45.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.994+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 >> 172.21.15.102:0/1165498258 conn(0x7f6fcc0727e0 msgr2=0x7f6fcc0cc010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:45.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.994+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 shutdown_connections 2024-03-18T16:28:45.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:45.994+0000 7f6fd0e00640 1 -- 172.21.15.102:0/1165498258 wait complete. 2024-03-18T16:28:46.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:46.616 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:47.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:47.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.809+0000 7fe12a200640 1 Processor -- start 2024-03-18T16:28:47.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.809+0000 7fe12a200640 1 -- start start 2024-03-18T16:28:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.811+0000 7fe12a200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe124154730 0x7fe12415acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.811+0000 7fe12a200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe12415d8c0 0x7fe1240c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.811+0000 7fe12a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1240ce0a0 con 0x7fe1240cdbd0 2024-03-18T16:28:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.811+0000 7fe12a200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe12415a830 con 0x7fe12415d8c0 2024-03-18T16:28:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.811+0000 7fe12a200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe1240ce380 con 0x7fe124154730 2024-03-18T16:28:47.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.828+0000 7fe123400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe124154730 0x7fe12415acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:47.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.828+0000 7fe123400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe124154730 0x7fe12415acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:46948/0 (socket says 172.21.15.102:46948) 2024-03-18T16:28:47.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.828+0000 7fe123400640 1 -- 172.21.15.102:0/2571163638 learned_addr learned my addr 172.21.15.102:0/2571163638 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:47.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.830+0000 7fe123400640 1 -- 172.21.15.102:0/2571163638 >> v2:172.21.15.102:3300/0 conn(0x7fe12415d8c0 msgr2=0x7fe1240c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:47.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.830+0000 7fe123400640 1 --2- 172.21.15.102:0/2571163638 >> v2:172.21.15.102:3300/0 conn(0x7fe12415d8c0 0x7fe1240c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:47.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.830+0000 7fe123400640 1 -- 172.21.15.102:0/2571163638 >> v1:172.21.15.102:6789/0 conn(0x7fe1240cdbd0 legacy=0x7fe1240c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:47.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.830+0000 7fe123400640 1 -- 172.21.15.102:0/2571163638 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1240ce660 con 0x7fe124154730 2024-03-18T16:28:47.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.833+0000 7fe123400640 1 --2- 172.21.15.102:0/2571163638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe124154730 0x7fe12415acc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe1240c8350 tx=0x7fe11402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bde979dadedb1a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:47.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.837+0000 7fe122000640 1 -- 172.21.15.102:0/2571163638 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11402e070 con 0x7fe124154730 2024-03-18T16:28:47.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.837+0000 7fe122000640 1 -- 172.21.15.102:0/2571163638 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe114037070 con 0x7fe124154730 2024-03-18T16:28:47.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.837+0000 7fe122000640 1 -- 172.21.15.102:0/2571163638 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114031a00 con 0x7fe124154730 2024-03-18T16:28:47.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.838+0000 7fe12a200640 1 -- 172.21.15.102:0/2571163638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe124154730 msgr2=0x7fe12415acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:47.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.838+0000 7fe12a200640 1 --2- 172.21.15.102:0/2571163638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe124154730 0x7fe12415acc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe1240c8350 tx=0x7fe11402b880 comp rx=0 tx=0).stop 2024-03-18T16:28:47.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.838+0000 7fe12a200640 1 -- 172.21.15.102:0/2571163638 shutdown_connections 2024-03-18T16:28:47.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.838+0000 7fe12a200640 1 --2- 172.21.15.102:0/2571163638 >> v2:172.21.15.102:3300/0 conn(0x7fe12415d8c0 0x7fe1240c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:47.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.838+0000 7fe12a200640 1 --2- 172.21.15.102:0/2571163638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe124154730 0x7fe12415acc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:47.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.838+0000 7fe12a200640 1 -- 172.21.15.102:0/2571163638 >> 172.21.15.102:0/2571163638 conn(0x7fe124066f80 msgr2=0x7fe124067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:47.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.838+0000 7fe12a200640 1 -- 172.21.15.102:0/2571163638 shutdown_connections 2024-03-18T16:28:47.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.838+0000 7fe12a200640 1 -- 172.21.15.102:0/2571163638 wait complete. 2024-03-18T16:28:47.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.840+0000 7fe12a200640 1 Processor -- start 2024-03-18T16:28:47.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.840+0000 7fe12a200640 1 -- start start 2024-03-18T16:28:47.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.841+0000 7fe12a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cdbd0 0x7fe124151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:47.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.841+0000 7fe12a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe124154730 0x7fe1240cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:47.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.841+0000 7fe12a200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe12415d8c0 0x7fe1241efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:47.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.841+0000 7fe12a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe1240c9dc0 con 0x7fe124154730 2024-03-18T16:28:47.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.841+0000 7fe12a200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe12415b200 con 0x7fe12415d8c0 2024-03-18T16:28:47.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.841+0000 7fe12a200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe1240c9f40 con 0x7fe1240cdbd0 2024-03-18T16:28:47.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.842+0000 7fe122a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe124154730 0x7fe1240cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:47.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.843+0000 7fe122a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe124154730 0x7fe1240cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37530/0 (socket says 172.21.15.102:37530) 2024-03-18T16:28:47.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.843+0000 7fe122a00640 1 -- 172.21.15.102:0/1708291541 learned_addr learned my addr 172.21.15.102:0/1708291541 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:47.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.845+0000 7fe123e00640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe12415d8c0 0x7fe1241efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:47.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.850+0000 7fe122a00640 1 -- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cdbd0 msgr2=0x7fe124151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:47.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.850+0000 7fe122a00640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cdbd0 0x7fe124151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:47.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.850+0000 7fe122a00640 1 -- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe12415d8c0 msgr2=0x7fe1241efdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:47.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.850+0000 7fe122a00640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe12415d8c0 0x7fe1241efdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:47.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.850+0000 7fe122a00640 1 -- 172.21.15.102:0/1708291541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe124151800 con 0x7fe124154730 2024-03-18T16:28:47.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.856+0000 7fe122a00640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe124154730 0x7fe1240cced0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c3f0 tx=0x7fe11000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:47.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.862+0000 7fe11b400640 1 -- 172.21.15.102:0/1708291541 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110017070 con 0x7fe124154730 2024-03-18T16:28:47.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.862+0000 7fe11b400640 1 -- 172.21.15.102:0/1708291541 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1100035f0 con 0x7fe124154730 2024-03-18T16:28:47.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.862+0000 7fe11b400640 1 -- 172.21.15.102:0/1708291541 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1100039e0 con 0x7fe124154730 2024-03-18T16:28:47.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.862+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1241f34d0 con 0x7fe124154730 2024-03-18T16:28:47.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.862+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1241f3a90 con 0x7fe124154730 2024-03-18T16:28:47.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:47.864+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1240c9490 con 0x7fe124154730 2024-03-18T16:28:48.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.136+0000 7fe11b400640 1 -- 172.21.15.102:0/1708291541 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe11001a070 con 0x7fe124154730 2024-03-18T16:28:48.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.138+0000 7fe11b400640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe10406bc20 0x7fe10406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.138+0000 7fe123400640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe10406bc20 0x7fe10406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:48.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.139+0000 7fe123400640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe10406bc20 0x7fe10406e0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe114036000 tx=0x7fe114030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:48.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.159+0000 7fe11b400640 1 -- 172.21.15.102:0/1708291541 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe110056b00 con 0x7fe124154730 2024-03-18T16:28:48.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.160+0000 7fe11b400640 1 -- 172.21.15.102:0/1708291541 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe110058a60 con 0x7fe124154730 2024-03-18T16:28:48.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.414+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1240c4b80 con 0x7fe124154730 2024-03-18T16:28:48.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.430+0000 7fe11b400640 1 -- 172.21.15.102:0/1708291541 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe11005cb00 con 0x7fe124154730 2024-03-18T16:28:48.432 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:48.432 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:48.432 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe10406bc20 msgr2=0x7fe10406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe10406bc20 0x7fe10406e0e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe114036000 tx=0x7fe114030000 comp rx=0 tx=0).stop 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe124154730 msgr2=0x7fe1240cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe124154730 0x7fe1240cced0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c3f0 tx=0x7fe11000c8b0 comp rx=0 tx=0).stop 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 shutdown_connections 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe10406bc20 0x7fe10406e0e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe12415d8c0 0x7fe1241efdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe124154730 0x7fe1240cced0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 --2- 172.21.15.102:0/1708291541 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe1240cdbd0 0x7fe124151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:48.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 >> 172.21.15.102:0/1708291541 conn(0x7fe124066f80 msgr2=0x7fe1241526a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:48.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.433+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 shutdown_connections 2024-03-18T16:28:48.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:48.434+0000 7fe12a200640 1 -- 172.21.15.102:0/1708291541 wait complete. 2024-03-18T16:28:49.056 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:49.056 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:50.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:50.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.228+0000 7f9926600640 1 Processor -- start 2024-03-18T16:28:50.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.228+0000 7f9926600640 1 -- start start 2024-03-18T16:28:50.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.229+0000 7f9926600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f99200cfe70 0x7f992015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:50.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.229+0000 7f9926600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f99200cc2f0 0x7f99200cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:50.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.229+0000 7f9926600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99200ccc80 con 0x7f99200d0250 2024-03-18T16:28:50.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.229+0000 7f9926600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f99200cabd0 con 0x7f99200cc2f0 2024-03-18T16:28:50.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.229+0000 7f9926600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f99200caea0 con 0x7f99200cfe70 2024-03-18T16:28:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.230+0000 7f9925200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f99200cfe70 0x7f992015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.230+0000 7f9925c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f99200cc2f0 0x7f99200cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.231+0000 7f9925200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f99200cfe70 0x7f992015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:46960/0 (socket says 172.21.15.102:46960) 2024-03-18T16:28:50.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.231+0000 7f9925200640 1 -- 172.21.15.102:0/3370709165 learned_addr learned my addr 172.21.15.102:0/3370709165 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.234+0000 7f9925200640 1 -- 172.21.15.102:0/3370709165 >> v2:172.21.15.102:3300/0 conn(0x7f99200cc2f0 msgr2=0x7f99200cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:50.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.234+0000 7f9925200640 1 --2- 172.21.15.102:0/3370709165 >> v2:172.21.15.102:3300/0 conn(0x7f99200cc2f0 0x7f99200cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:50.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.234+0000 7f9925200640 1 -- 172.21.15.102:0/3370709165 >> v1:172.21.15.102:6789/0 conn(0x7f99200d0250 legacy=0x7f99200c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:50.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.234+0000 7f9925200640 1 -- 172.21.15.102:0/3370709165 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99200cb180 con 0x7f99200cfe70 2024-03-18T16:28:50.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.237+0000 7f9925200640 1 --2- 172.21.15.102:0/3370709165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cfe70 0x7f992015ae40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f99200cd800 tx=0x7f991402ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d583539bebd8786 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:50.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.241+0000 7f991f400640 1 -- 172.21.15.102:0/3370709165 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991402f850 con 0x7f99200cfe70 2024-03-18T16:28:50.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.241+0000 7f991f400640 1 -- 172.21.15.102:0/3370709165 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f991402f9f0 con 0x7f99200cfe70 2024-03-18T16:28:50.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.241+0000 7f991f400640 1 -- 172.21.15.102:0/3370709165 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991402fdc0 con 0x7f99200cfe70 2024-03-18T16:28:50.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.242+0000 7f9926600640 1 -- 172.21.15.102:0/3370709165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cfe70 msgr2=0x7f992015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:50.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.242+0000 7f9926600640 1 --2- 172.21.15.102:0/3370709165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cfe70 0x7f992015ae40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f99200cd800 tx=0x7f991402ba30 comp rx=0 tx=0).stop 2024-03-18T16:28:50.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.242+0000 7f9926600640 1 -- 172.21.15.102:0/3370709165 shutdown_connections 2024-03-18T16:28:50.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.242+0000 7f9926600640 1 --2- 172.21.15.102:0/3370709165 >> v2:172.21.15.102:3300/0 conn(0x7f99200cc2f0 0x7f99200cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:50.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.242+0000 7f9926600640 1 --2- 172.21.15.102:0/3370709165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cfe70 0x7f992015ae40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:50.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.242+0000 7f9926600640 1 -- 172.21.15.102:0/3370709165 >> 172.21.15.102:0/3370709165 conn(0x7f992008c3e0 msgr2=0x7f992006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:50.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.242+0000 7f9926600640 1 -- 172.21.15.102:0/3370709165 shutdown_connections 2024-03-18T16:28:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.242+0000 7f9926600640 1 -- 172.21.15.102:0/3370709165 wait complete. 2024-03-18T16:28:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.244+0000 7f9926600640 1 Processor -- start 2024-03-18T16:28:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.244+0000 7f9926600640 1 -- start start 2024-03-18T16:28:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.245+0000 7f9926600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cc2f0 0x7f99200c3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.245+0000 7f9926600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99200cfe70 0x7f9920153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.245+0000 7f9926600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99200d0250 0x7f99201f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.245+0000 7f9926600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f99200cba20 con 0x7f99200cfe70 2024-03-18T16:28:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.245+0000 7f9926600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99200cb740 con 0x7f99200cc2f0 2024-03-18T16:28:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.245+0000 7f9926600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f99200cbd00 con 0x7f99200d0250 2024-03-18T16:28:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.246+0000 7f9925200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cc2f0 0x7f99200c3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.246+0000 7f991fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99200cfe70 0x7f9920153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.246+0000 7f9925200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cc2f0 0x7f99200c3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:46974/0 (socket says 172.21.15.102:46974) 2024-03-18T16:28:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.246+0000 7f9925200640 1 -- 172.21.15.102:0/4220709711 learned_addr learned my addr 172.21.15.102:0/4220709711 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:50.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.246+0000 7f9925c00640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99200d0250 0x7f99201f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:50.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.251+0000 7f9925c00640 1 -- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cc2f0 msgr2=0x7f99200c3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:50.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.251+0000 7f9925c00640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cc2f0 0x7f99200c3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:50.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.251+0000 7f9925c00640 1 -- 172.21.15.102:0/4220709711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99200cfe70 msgr2=0x7f9920153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:50.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.251+0000 7f9925c00640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99200cfe70 0x7f9920153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:50.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.251+0000 7f9925c00640 1 -- 172.21.15.102:0/4220709711 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99200c3d60 con 0x7f99200d0250 2024-03-18T16:28:50.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.253+0000 7f9925c00640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99200d0250 0x7f99201f34b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f991800efc0 tx=0x7f991800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:50.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.257+0000 7f991cc00640 1 -- 172.21.15.102:0/4220709711 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918018020 con 0x7f99200d0250 2024-03-18T16:28:50.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.257+0000 7f991cc00640 1 -- 172.21.15.102:0/4220709711 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9918014480 con 0x7f99200d0250 2024-03-18T16:28:50.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.257+0000 7f991cc00640 1 -- 172.21.15.102:0/4220709711 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918014870 con 0x7f99200d0250 2024-03-18T16:28:50.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.257+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99201f4390 con 0x7f99200d0250 2024-03-18T16:28:50.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.257+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f99201f40b0 con 0x7f99200d0250 2024-03-18T16:28:50.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.258+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99200735d0 con 0x7f99200d0250 2024-03-18T16:28:50.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.277+0000 7f991cc00640 1 -- 172.21.15.102:0/4220709711 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9918005d10 con 0x7f99200d0250 2024-03-18T16:28:50.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.279+0000 7f991cc00640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f990006bc50 0x7f990006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:50.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.279+0000 7f9925200640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f990006bc50 0x7f990006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:50.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.280+0000 7f9925200640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f990006bc50 0x7f990006e110 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9914009630 tx=0x7f9914008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:50.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.517+0000 7f991cc00640 1 -- 172.21.15.102:0/4220709711 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f99180581c0 con 0x7f99200d0250 2024-03-18T16:28:50.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.528+0000 7f991cc00640 1 -- 172.21.15.102:0/4220709711 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f99200735d0 con 0x7f99200d0250 2024-03-18T16:28:50.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.768+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99201548e0 con 0x7f99200d0250 2024-03-18T16:28:50.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.781+0000 7f991cc00640 1 -- 172.21.15.102:0/4220709711 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f9918022030 con 0x7f99200d0250 2024-03-18T16:28:50.783 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:50.783 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:50.783 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:50.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.783+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f990006bc50 msgr2=0x7f990006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.783+0000 7f9926600640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f990006bc50 0x7f990006e110 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9914009630 tx=0x7f9914008030 comp rx=0 tx=0).stop 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.783+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99200d0250 msgr2=0x7f99201f34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.783+0000 7f9926600640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99200d0250 0x7f99201f34b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f991800efc0 tx=0x7f991800c490 comp rx=0 tx=0).stop 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.784+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 shutdown_connections 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.784+0000 7f9926600640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f990006bc50 0x7f990006e110 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.784+0000 7f9926600640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f99200d0250 0x7f99201f34b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.784+0000 7f9926600640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99200cfe70 0x7f9920153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.784+0000 7f9926600640 1 --2- 172.21.15.102:0/4220709711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99200cc2f0 0x7f99200c3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.784+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 >> 172.21.15.102:0/4220709711 conn(0x7f992008c3e0 msgr2=0x7f9920158880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.784+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 shutdown_connections 2024-03-18T16:28:50.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:50.784+0000 7f9926600640 1 -- 172.21.15.102:0/4220709711 wait complete. 2024-03-18T16:28:51.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:51.408 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:52.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:52.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:28:52.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:28:52.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.601+0000 7f1a97a00640 1 Processor -- start 2024-03-18T16:28:52.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.602+0000 7f1a97a00640 1 -- start start 2024-03-18T16:28:52.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.604+0000 7f1a97a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900c8b20 0x7f1a90072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:52.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.604+0000 7f1a97a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a90158760 0x7f1a9015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:52.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.604+0000 7f1a97a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a90157ac0 con 0x7f1a900c8740 2024-03-18T16:28:52.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.604+0000 7f1a97a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1a9015a860 con 0x7f1a90158760 2024-03-18T16:28:52.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.604+0000 7f1a97a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1a90072840 con 0x7f1a900c8b20 2024-03-18T16:28:52.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.604+0000 7f1a95a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a90158760 0x7f1a9015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:52.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.605+0000 7f1a8fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900c8b20 0x7f1a90072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:52.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.605+0000 7f1a95a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a90158760 0x7f1a9015b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54786/0 (socket says 172.21.15.102:54786) 2024-03-18T16:28:52.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.605+0000 7f1a95a00640 1 -- 172.21.15.102:0/3074225891 learned_addr learned my addr 172.21.15.102:0/3074225891 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:52.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.608+0000 7f1a8fe00640 1 -- 172.21.15.102:0/3074225891 >> v2:172.21.15.154:3300/0 conn(0x7f1a90158760 msgr2=0x7f1a9015b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:52.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.608+0000 7f1a8fe00640 1 --2- 172.21.15.102:0/3074225891 >> v2:172.21.15.154:3300/0 conn(0x7f1a90158760 0x7f1a9015b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:52.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.608+0000 7f1a8fe00640 1 -- 172.21.15.102:0/3074225891 >> v1:172.21.15.154:6789/0 conn(0x7f1a900c8740 legacy=0x7f1a901572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:52.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.608+0000 7f1a8fe00640 1 -- 172.21.15.102:0/3074225891 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a90157da0 con 0x7f1a900c8b20 2024-03-18T16:28:52.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.610+0000 7f1a8fe00640 1 --2- 172.21.15.102:0/3074225891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900c8b20 0x7f1a90072300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c0097a0 tx=0x7f1a7c02dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc713319a5e28620 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:52.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.612+0000 7f1a8f400640 1 -- 172.21.15.102:0/3074225891 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c02f8a0 con 0x7f1a900c8b20 2024-03-18T16:28:52.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.614+0000 7f1a8f400640 1 -- 172.21.15.102:0/3074225891 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a7c02fa40 con 0x7f1a900c8b20 2024-03-18T16:28:52.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.614+0000 7f1a8f400640 1 -- 172.21.15.102:0/3074225891 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c02fe10 con 0x7f1a900c8b20 2024-03-18T16:28:52.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.615+0000 7f1a97a00640 1 -- 172.21.15.102:0/3074225891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900c8b20 msgr2=0x7f1a90072300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:52.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.615+0000 7f1a97a00640 1 --2- 172.21.15.102:0/3074225891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900c8b20 0x7f1a90072300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c0097a0 tx=0x7f1a7c02dae0 comp rx=0 tx=0).stop 2024-03-18T16:28:52.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.615+0000 7f1a97a00640 1 -- 172.21.15.102:0/3074225891 shutdown_connections 2024-03-18T16:28:52.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.615+0000 7f1a97a00640 1 --2- 172.21.15.102:0/3074225891 >> v2:172.21.15.154:3300/0 conn(0x7f1a90158760 0x7f1a9015b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:52.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.615+0000 7f1a97a00640 1 --2- 172.21.15.102:0/3074225891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a900c8b20 0x7f1a90072300 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:52.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.615+0000 7f1a97a00640 1 -- 172.21.15.102:0/3074225891 >> 172.21.15.102:0/3074225891 conn(0x7f1a900700d0 msgr2=0x7f1a9006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:52.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.615+0000 7f1a97a00640 1 -- 172.21.15.102:0/3074225891 shutdown_connections 2024-03-18T16:28:52.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.616+0000 7f1a97a00640 1 -- 172.21.15.102:0/3074225891 wait complete. 2024-03-18T16:28:52.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.618+0000 7f1a97a00640 1 Processor -- start 2024-03-18T16:28:52.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.618+0000 7f1a97a00640 1 -- start start 2024-03-18T16:28:52.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.620+0000 7f1a97a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a900c8740 0x7f1a9014ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:52.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.620+0000 7f1a97a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a900c8b20 0x7f1a901e2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:52.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.620+0000 7f1a97a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a90158760 0x7f1a901e0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:52.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.620+0000 7f1a97a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9015aab0 con 0x7f1a900c8b20 2024-03-18T16:28:52.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.620+0000 7f1a97a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9015b7f0 con 0x7f1a900c8740 2024-03-18T16:28:52.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.620+0000 7f1a97a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9015ad90 con 0x7f1a90158760 2024-03-18T16:28:52.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.621+0000 7f1a95a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a90158760 0x7f1a901e0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:52.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.621+0000 7f1a95000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a900c8740 0x7f1a9014ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:52.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.621+0000 7f1a8fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a900c8b20 0x7f1a901e2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:52.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.621+0000 7f1a95000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a900c8740 0x7f1a9014ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54802/0 (socket says 172.21.15.102:54802) 2024-03-18T16:28:52.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.621+0000 7f1a95000640 1 -- 172.21.15.102:0/3966288075 learned_addr learned my addr 172.21.15.102:0/3966288075 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:52.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.625+0000 7f1a95a00640 1 -- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a900c8740 msgr2=0x7f1a9014ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:52.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.625+0000 7f1a95a00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a900c8740 0x7f1a9014ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:52.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.625+0000 7f1a95a00640 1 -- 172.21.15.102:0/3966288075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a900c8b20 msgr2=0x7f1a901e2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:52.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.625+0000 7f1a95a00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a900c8b20 0x7f1a901e2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:52.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.625+0000 7f1a95a00640 1 -- 172.21.15.102:0/3966288075 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a901ef7a0 con 0x7f1a90158760 2024-03-18T16:28:52.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.627+0000 7f1a95a00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a90158760 0x7f1a901e0d40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800efc0 tx=0x7f1a8800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:52.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.631+0000 7f1a8cc00640 1 -- 172.21.15.102:0/3966288075 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8800cd20 con 0x7f1a90158760 2024-03-18T16:28:52.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.631+0000 7f1a8cc00640 1 -- 172.21.15.102:0/3966288075 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a88003990 con 0x7f1a90158760 2024-03-18T16:28:52.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.631+0000 7f1a8cc00640 1 -- 172.21.15.102:0/3966288075 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a88003d80 con 0x7f1a90158760 2024-03-18T16:28:52.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.631+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a901efa80 con 0x7f1a90158760 2024-03-18T16:28:52.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.631+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a90157230 con 0x7f1a90158760 2024-03-18T16:28:52.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.633+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a90074fd0 con 0x7f1a90158760 2024-03-18T16:28:52.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.657+0000 7f1a8cc00640 1 -- 172.21.15.102:0/3966288075 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1a88005d10 con 0x7f1a90158760 2024-03-18T16:28:52.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.658+0000 7f1a8cc00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a7006bc20 0x7f1a7006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.658+0000 7f1a95000640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a7006bc20 0x7f1a7006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.659+0000 7f1a95000640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a7006bc20 0x7f1a7006e0e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1a84007fd0 tx=0x7f1a84009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:52.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.923+0000 7f1a8cc00640 1 -- 172.21.15.102:0/3966288075 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1a88058170 con 0x7f1a90158760 2024-03-18T16:28:52.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:52.935+0000 7f1a8cc00640 1 -- 172.21.15.102:0/3966288075 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1a88022020 con 0x7f1a90158760 2024-03-18T16:28:53.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.340+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a9006c590 con 0x7f1a90158760 2024-03-18T16:28:53.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.366+0000 7f1a8cc00640 1 -- 172.21.15.102:0/3966288075 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1a88010040 con 0x7f1a90158760 2024-03-18T16:28:53.368 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:53.368 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:53.368 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:53.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a7006bc20 msgr2=0x7f1a7006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:53.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a7006bc20 0x7f1a7006e0e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1a84007fd0 tx=0x7f1a84009040 comp rx=0 tx=0).stop 2024-03-18T16:28:53.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a90158760 msgr2=0x7f1a901e0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:53.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a90158760 0x7f1a901e0d40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1a8800efc0 tx=0x7f1a8800c490 comp rx=0 tx=0).stop 2024-03-18T16:28:53.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 shutdown_connections 2024-03-18T16:28:53.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a7006bc20 0x7f1a7006e0e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a90158760 0x7f1a901e0d40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a900c8b20 0x7f1a901e2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 --2- 172.21.15.102:0/3966288075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a900c8740 0x7f1a9014ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 >> 172.21.15.102:0/3966288075 conn(0x7f1a900700d0 msgr2=0x7f1a900c0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 shutdown_connections 2024-03-18T16:28:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:53.370+0000 7f1a97a00640 1 -- 172.21.15.102:0/3966288075 wait complete. 2024-03-18T16:28:54.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:54.000 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:55.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.174+0000 7f9d71800640 1 Processor -- start 2024-03-18T16:28:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.174+0000 7f9d71800640 1 -- start start 2024-03-18T16:28:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.175+0000 7f9d71800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.175+0000 7f9d71800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9d6c15c3a0 0x7f9d6c0c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.175+0000 7f9d71800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c158ad0 con 0x7f9d6c0c4fa0 2024-03-18T16:28:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.175+0000 7f9d71800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9d6c159370 con 0x7f9d6c15c3a0 2024-03-18T16:28:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.175+0000 7f9d71800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9d6c158db0 con 0x7f9d6c141650 2024-03-18T16:28:55.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.176+0000 7f9d6b400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9d6c15c3a0 0x7f9d6c0c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:55.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.176+0000 7f9d6be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:55.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.176+0000 7f9d70e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f9d6c0c4fa0 0x7f9d6c06f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:37688/0 (socket says 172.21.15.102:37688) 2024-03-18T16:28:55.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.176+0000 7f9d70e00640 1 -- 172.21.15.102:0/3944337465 learned_addr learned my addr 172.21.15.102:0/3944337465 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.179+0000 7f9d6be00640 1 -- 172.21.15.102:0/3944337465 >> v2:172.21.15.154:3300/0 conn(0x7f9d6c15c3a0 msgr2=0x7f9d6c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.179+0000 7f9d6be00640 1 --2- 172.21.15.102:0/3944337465 >> v2:172.21.15.154:3300/0 conn(0x7f9d6c15c3a0 0x7f9d6c0c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.179+0000 7f9d6be00640 1 -- 172.21.15.102:0/3944337465 >> v1:172.21.15.154:6789/0 conn(0x7f9d6c0c4fa0 legacy=0x7f9d6c06f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.179+0000 7f9d6be00640 1 -- 172.21.15.102:0/3944337465 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c159090 con 0x7f9d6c141650 2024-03-18T16:28:55.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.181+0000 7f9d6be00640 1 --2- 172.21.15.102:0/3944337465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c158420 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0097a0 tx=0x7f9d5c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3aaf2d0e9aeef7e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:55.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.185+0000 7f9d6aa00640 1 -- 172.21.15.102:0/3944337465 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c035020 con 0x7f9d6c141650 2024-03-18T16:28:55.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.185+0000 7f9d6aa00640 1 -- 172.21.15.102:0/3944337465 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d5c031810 con 0x7f9d6c141650 2024-03-18T16:28:55.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.185+0000 7f9d6aa00640 1 -- 172.21.15.102:0/3944337465 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c031be0 con 0x7f9d6c141650 2024-03-18T16:28:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.187+0000 7f9d71800640 1 -- 172.21.15.102:0/3944337465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 msgr2=0x7f9d6c158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.187+0000 7f9d71800640 1 --2- 172.21.15.102:0/3944337465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c158420 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c0097a0 tx=0x7f9d5c02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.187+0000 7f9d71800640 1 -- 172.21.15.102:0/3944337465 shutdown_connections 2024-03-18T16:28:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.187+0000 7f9d71800640 1 --2- 172.21.15.102:0/3944337465 >> v2:172.21.15.154:3300/0 conn(0x7f9d6c15c3a0 0x7f9d6c0c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.187+0000 7f9d71800640 1 --2- 172.21.15.102:0/3944337465 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c158420 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.187+0000 7f9d71800640 1 -- 172.21.15.102:0/3944337465 >> 172.21.15.102:0/3944337465 conn(0x7f9d6c0cb890 msgr2=0x7f9d6c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.187+0000 7f9d71800640 1 -- 172.21.15.102:0/3944337465 shutdown_connections 2024-03-18T16:28:55.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.187+0000 7f9d71800640 1 -- 172.21.15.102:0/3944337465 wait complete. 2024-03-18T16:28:55.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.189+0000 7f9d71800640 1 Processor -- start 2024-03-18T16:28:55.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.190+0000 7f9d71800640 1 -- start start 2024-03-18T16:28:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.191+0000 7f9d71800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d6c0c4fa0 0x7f9d6c153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.191+0000 7f9d71800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c13e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.191+0000 7f9d71800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d6c15c3a0 0x7f9d6c1ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.191+0000 7f9d71800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c15c7c0 con 0x7f9d6c15c3a0 2024-03-18T16:28:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.191+0000 7f9d71800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6c0c66a0 con 0x7f9d6c0c4fa0 2024-03-18T16:28:55.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.191+0000 7f9d71800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9d6c141a30 con 0x7f9d6c141650 2024-03-18T16:28:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.192+0000 7f9d6b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c13e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.192+0000 7f9d6be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d6c0c4fa0 0x7f9d6c153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.192+0000 7f9d6b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c13e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58782/0 (socket says 172.21.15.102:58782) 2024-03-18T16:28:55.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.192+0000 7f9d6b400640 1 -- 172.21.15.102:0/2257149179 learned_addr learned my addr 172.21.15.102:0/2257149179 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.196+0000 7f9d70e00640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d6c15c3a0 0x7f9d6c1ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.196+0000 7f9d6b400640 1 -- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d6c0c4fa0 msgr2=0x7f9d6c153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.196+0000 7f9d6b400640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d6c0c4fa0 0x7f9d6c153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.196+0000 7f9d6b400640 1 -- 172.21.15.102:0/2257149179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d6c15c3a0 msgr2=0x7f9d6c1ebed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.196+0000 7f9d6b400640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d6c15c3a0 0x7f9d6c1ebed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.196+0000 7f9d6b400640 1 -- 172.21.15.102:0/2257149179 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c1584b0 con 0x7f9d6c141650 2024-03-18T16:28:55.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.198+0000 7f9d6b400640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c13e3e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800e8a0 tx=0x7f9d5800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.202+0000 7f9d63e00640 1 -- 172.21.15.102:0/2257149179 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58017070 con 0x7f9d6c141650 2024-03-18T16:28:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.202+0000 7f9d63e00640 1 -- 172.21.15.102:0/2257149179 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d5800c940 con 0x7f9d6c141650 2024-03-18T16:28:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.202+0000 7f9d63e00640 1 -- 172.21.15.102:0/2257149179 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5800cd30 con 0x7f9d6c141650 2024-03-18T16:28:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.203+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6c154090 con 0x7f9d6c141650 2024-03-18T16:28:55.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.203+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d6c1ef440 con 0x7f9d6c141650 2024-03-18T16:28:55.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.204+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6c074fd0 con 0x7f9d6c141650 2024-03-18T16:28:55.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.224+0000 7f9d63e00640 1 -- 172.21.15.102:0/2257149179 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9d5801a070 con 0x7f9d6c141650 2024-03-18T16:28:55.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.225+0000 7f9d63e00640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d4c06bc20 0x7f9d4c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:55.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.226+0000 7f9d6be00640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d4c06bc20 0x7f9d4c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:55.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.227+0000 7f9d6be00640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d4c06bc20 0x7f9d4c06e0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c036000 tx=0x7f9d5c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:55.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.465+0000 7f9d63e00640 1 -- 172.21.15.102:0/2257149179 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9d58057500 con 0x7f9d6c141650 2024-03-18T16:28:55.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.476+0000 7f9d63e00640 1 -- 172.21.15.102:0/2257149179 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9d6c1584b0 con 0x7f9d6c141650 2024-03-18T16:28:55.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.730+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d6c0b7f00 con 0x7f9d6c141650 2024-03-18T16:28:55.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.753+0000 7f9d63e00640 1 -- 172.21.15.102:0/2257149179 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f9d58059460 con 0x7f9d6c141650 2024-03-18T16:28:55.755 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:55.755 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:55.755 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.755+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d4c06bc20 msgr2=0x7f9d4c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d4c06bc20 0x7f9d4c06e0e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c036000 tx=0x7f9d5c008030 comp rx=0 tx=0).stop 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 msgr2=0x7f9d6c13e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c13e3e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800e8a0 tx=0x7f9d5800ed60 comp rx=0 tx=0).stop 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 shutdown_connections 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d4c06bc20 0x7f9d4c06e0e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d6c15c3a0 0x7f9d6c1ebed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d6c141650 0x7f9d6c13e3e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 --2- 172.21.15.102:0/2257149179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d6c0c4fa0 0x7f9d6c153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 >> 172.21.15.102:0/2257149179 conn(0x7f9d6c0cb890 msgr2=0x7f9d6c0ca1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 shutdown_connections 2024-03-18T16:28:55.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:55.756+0000 7f9d71800640 1 -- 172.21.15.102:0/2257149179 wait complete. 2024-03-18T16:28:56.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:56.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:57.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:57.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.560+0000 7f9ed7e00640 1 Processor -- start 2024-03-18T16:28:57.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.560+0000 7f9ed7e00640 1 -- start start 2024-03-18T16:28:57.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.562+0000 7f9ed7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8071700 0x7f9ed8072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:57.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.562+0000 7f9ed7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ed8161d30 0x7f9ed8072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:57.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.562+0000 7f9ed7e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ed80c4190 con 0x7f9ed8158c00 2024-03-18T16:28:57.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.562+0000 7f9ed7e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9ed8158fe0 con 0x7f9ed8161d30 2024-03-18T16:28:57.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.562+0000 7f9ed7e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed806f3b0 con 0x7f9ed8071700 2024-03-18T16:28:57.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.563+0000 7f9ed6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8071700 0x7f9ed8072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:57.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.564+0000 7f9ed7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ed8161d30 0x7f9ed8072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:57.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.564+0000 7f9ed7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ed8161d30 0x7f9ed8072fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39302/0 (socket says 172.21.15.102:39302) 2024-03-18T16:28:57.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.564+0000 7f9ed7400640 1 -- 172.21.15.102:0/4218888236 learned_addr learned my addr 172.21.15.102:0/4218888236 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:57.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.567+0000 7f9ed7400640 1 -- 172.21.15.102:0/4218888236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8071700 msgr2=0x7f9ed8072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:57.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.567+0000 7f9ed7400640 1 --2- 172.21.15.102:0/4218888236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8071700 0x7f9ed8072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:57.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.567+0000 7f9ed7400640 1 -- 172.21.15.102:0/4218888236 >> v1:172.21.15.102:6789/0 conn(0x7f9ed8158c00 legacy=0x7f9ed806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:28:57.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.567+0000 7f9ed7400640 1 -- 172.21.15.102:0/4218888236 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed806f060 con 0x7f9ed8161d30 2024-03-18T16:28:57.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.571+0000 7f9ed7400640 1 --2- 172.21.15.102:0/4218888236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed8161d30 0x7f9ed8072fb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009e50 tx=0x7f9ed002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8a3a8c19eda3c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:57.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.577+0000 7f9ed5600640 1 -- 172.21.15.102:0/4218888236 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed002ca80 con 0x7f9ed8161d30 2024-03-18T16:28:57.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.577+0000 7f9ed5600640 1 -- 172.21.15.102:0/4218888236 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed002cc20 con 0x7f9ed8161d30 2024-03-18T16:28:57.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.577+0000 7f9ed5600640 1 -- 172.21.15.102:0/4218888236 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00343f0 con 0x7f9ed8161d30 2024-03-18T16:28:57.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.578+0000 7f9ed7e00640 1 -- 172.21.15.102:0/4218888236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed8161d30 msgr2=0x7f9ed8072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:57.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.578+0000 7f9ed7e00640 1 --2- 172.21.15.102:0/4218888236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed8161d30 0x7f9ed8072fb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009e50 tx=0x7f9ed002bee0 comp rx=0 tx=0).stop 2024-03-18T16:28:57.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.578+0000 7f9ed7e00640 1 -- 172.21.15.102:0/4218888236 shutdown_connections 2024-03-18T16:28:57.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.578+0000 7f9ed7e00640 1 --2- 172.21.15.102:0/4218888236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed8161d30 0x7f9ed8072fb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:57.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.578+0000 7f9ed7e00640 1 --2- 172.21.15.102:0/4218888236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8071700 0x7f9ed8072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:57.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.578+0000 7f9ed7e00640 1 -- 172.21.15.102:0/4218888236 >> 172.21.15.102:0/4218888236 conn(0x7f9ed80cb890 msgr2=0x7f9ed80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:57.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.578+0000 7f9ed7e00640 1 -- 172.21.15.102:0/4218888236 shutdown_connections 2024-03-18T16:28:57.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.578+0000 7f9ed7e00640 1 -- 172.21.15.102:0/4218888236 wait complete. 2024-03-18T16:28:57.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.580+0000 7f9ed7e00640 1 Processor -- start 2024-03-18T16:28:57.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.581+0000 7f9ed7e00640 1 -- start start 2024-03-18T16:28:57.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.582+0000 7f9ed7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed8071700 0x7f9ed81f4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:57.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.582+0000 7f9ed7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8158c00 0x7f9ed8155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:57.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.582+0000 7f9ed7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed8161d30 0x7f9ed815ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:57.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.582+0000 7f9ed7e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed8072270 con 0x7f9ed8161d30 2024-03-18T16:28:57.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.582+0000 7f9ed7e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed8071f90 con 0x7f9ed8071700 2024-03-18T16:28:57.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.582+0000 7f9ed7e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed8072550 con 0x7f9ed8158c00 2024-03-18T16:28:57.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.583+0000 7f9ed6a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed8071700 0x7f9ed81f4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:57.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.583+0000 7f9ed7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed8161d30 0x7f9ed815ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:57.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.583+0000 7f9ed6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8158c00 0x7f9ed8155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:57.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.583+0000 7f9ed6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8158c00 0x7f9ed8155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58802/0 (socket says 172.21.15.102:58802) 2024-03-18T16:28:57.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.583+0000 7f9ed6000640 1 -- 172.21.15.102:0/3341391871 learned_addr learned my addr 172.21.15.102:0/3341391871 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:57.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.588+0000 7f9ed6000640 1 -- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed8071700 msgr2=0x7f9ed81f4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:57.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.588+0000 7f9ed6000640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed8071700 0x7f9ed81f4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:57.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.588+0000 7f9ed6000640 1 -- 172.21.15.102:0/3341391871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed8161d30 msgr2=0x7f9ed815ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:57.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.588+0000 7f9ed6000640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed8161d30 0x7f9ed815ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:57.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.588+0000 7f9ed6000640 1 -- 172.21.15.102:0/3341391871 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed815a3c0 con 0x7f9ed8158c00 2024-03-18T16:28:57.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.591+0000 7f9ed6000640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8158c00 0x7f9ed8155a10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400e8a0 tx=0x7f9ec400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.595+0000 7f9ecea00640 1 -- 172.21.15.102:0/3341391871 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4017070 con 0x7f9ed8158c00 2024-03-18T16:28:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.595+0000 7f9ecea00640 1 -- 172.21.15.102:0/3341391871 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ec400c940 con 0x7f9ed8158c00 2024-03-18T16:28:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.595+0000 7f9ecea00640 1 -- 172.21.15.102:0/3341391871 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec400cd30 con 0x7f9ed8158c00 2024-03-18T16:28:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.595+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed8159300 con 0x7f9ed8158c00 2024-03-18T16:28:57.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.595+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed815a960 con 0x7f9ed8158c00 2024-03-18T16:28:57.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.596+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ed8074fd0 con 0x7f9ed8158c00 2024-03-18T16:28:57.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.860+0000 7f9ecea00640 1 -- 172.21.15.102:0/3341391871 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9ec401a070 con 0x7f9ed8158c00 2024-03-18T16:28:57.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.862+0000 7f9ecea00640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9eb806bc20 0x7f9eb806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:57.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.863+0000 7f9ed6a00640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9eb806bc20 0x7f9eb806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:57.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.863+0000 7f9ed6a00640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9eb806bc20 0x7f9eb806e0e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80017f0 tx=0x7f9ec8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.864+0000 7f9ecea00640 1 -- 172.21.15.102:0/3341391871 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9ec4057590 con 0x7f9ed8158c00 2024-03-18T16:28:57.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:57.875+0000 7f9ecea00640 1 -- 172.21.15.102:0/3341391871 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9ec40594f0 con 0x7f9ed8158c00 2024-03-18T16:28:58.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.122+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ed8072850 con 0x7f9ed8158c00 2024-03-18T16:28:58.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.149+0000 7f9ecea00640 1 -- 172.21.15.102:0/3341391871 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f9ec405d590 con 0x7f9ed8158c00 2024-03-18T16:28:58.151 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:28:58.152 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:28:58.152 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:28:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9eb806bc20 msgr2=0x7f9eb806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9eb806bc20 0x7f9eb806e0e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9ec80017f0 tx=0x7f9ec8009040 comp rx=0 tx=0).stop 2024-03-18T16:28:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8158c00 msgr2=0x7f9ed8155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8158c00 0x7f9ed8155a10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400e8a0 tx=0x7f9ec400ed60 comp rx=0 tx=0).stop 2024-03-18T16:28:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 shutdown_connections 2024-03-18T16:28:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9eb806bc20 0x7f9eb806e0e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ed8161d30 0x7f9ed815ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ed8158c00 0x7f9ed8155a10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 --2- 172.21.15.102:0/3341391871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ed8071700 0x7f9ed81f4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 >> 172.21.15.102:0/3341391871 conn(0x7f9ed80cb890 msgr2=0x7f9ed81621a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 shutdown_connections 2024-03-18T16:28:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:58.153+0000 7f9ed7e00640 1 -- 172.21.15.102:0/3341391871 wait complete. 2024-03-18T16:28:58.775 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:28:58.775 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:28:59.776 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:28:59.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.946+0000 7fdb8ac00640 1 Processor -- start 2024-03-18T16:28:59.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.946+0000 7fdb8ac00640 1 -- start start 2024-03-18T16:28:59.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.947+0000 7fdb8ac00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdb84141650 0x7fdb84158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:59.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.947+0000 7fdb8ac00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb840c72d0 con 0x7fdb8415c3a0 2024-03-18T16:28:59.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.947+0000 7fdb8ac00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb84158ad0 con 0x7fdb840c4fa0 2024-03-18T16:28:59.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.947+0000 7fdb8ac00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdb84159370 con 0x7fdb84141650 2024-03-18T16:28:59.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.948+0000 7fdb89800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdb84141650 0x7fdb84158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:59.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.948+0000 7fdb88e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdb8415c3a0 0x7fdb840c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:39710/0 (socket says 172.21.15.102:39710) 2024-03-18T16:28:59.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.948+0000 7fdb88e00640 1 -- 172.21.15.102:0/3944095168 learned_addr learned my addr 172.21.15.102:0/3944095168 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:28:59.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.951+0000 7fdb89800640 1 -- 172.21.15.102:0/3944095168 >> v1:172.21.15.154:6789/0 conn(0x7fdb840c4fa0 legacy=0x7fdb84162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:59.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.951+0000 7fdb89800640 1 -- 172.21.15.102:0/3944095168 >> v1:172.21.15.102:6789/0 conn(0x7fdb8415c3a0 legacy=0x7fdb840c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:59.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.951+0000 7fdb89800640 1 -- 172.21.15.102:0/3944095168 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb84158db0 con 0x7fdb84141650 2024-03-18T16:28:59.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.954+0000 7fdb89800640 1 --2- 172.21.15.102:0/3944095168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 0x7fdb84158420 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdb780097a0 tx=0x7fdb7802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=442da6fb91a643bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:59.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.958+0000 7fdb83e00640 1 -- 172.21.15.102:0/3944095168 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78034070 con 0x7fdb84141650 2024-03-18T16:28:59.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.958+0000 7fdb83e00640 1 -- 172.21.15.102:0/3944095168 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb78030610 con 0x7fdb84141650 2024-03-18T16:28:59.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.958+0000 7fdb83e00640 1 -- 172.21.15.102:0/3944095168 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb780309e0 con 0x7fdb84141650 2024-03-18T16:28:59.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.959+0000 7fdb8ac00640 1 -- 172.21.15.102:0/3944095168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 msgr2=0x7fdb84158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:59.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.959+0000 7fdb8ac00640 1 --2- 172.21.15.102:0/3944095168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 0x7fdb84158420 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdb780097a0 tx=0x7fdb7802b960 comp rx=0 tx=0).stop 2024-03-18T16:28:59.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.959+0000 7fdb8ac00640 1 -- 172.21.15.102:0/3944095168 shutdown_connections 2024-03-18T16:28:59.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.959+0000 7fdb8ac00640 1 --2- 172.21.15.102:0/3944095168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 0x7fdb84158420 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:59.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.959+0000 7fdb8ac00640 1 -- 172.21.15.102:0/3944095168 >> 172.21.15.102:0/3944095168 conn(0x7fdb840cb890 msgr2=0x7fdb840cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:28:59.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.959+0000 7fdb8ac00640 1 -- 172.21.15.102:0/3944095168 shutdown_connections 2024-03-18T16:28:59.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.959+0000 7fdb8ac00640 1 -- 172.21.15.102:0/3944095168 wait complete. 2024-03-18T16:28:59.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.961+0000 7fdb8ac00640 1 Processor -- start 2024-03-18T16:28:59.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.962+0000 7fdb8ac00640 1 -- start start 2024-03-18T16:28:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.963+0000 7fdb8ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb840c4fa0 0x7fdb84152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.963+0000 7fdb8ac00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 0x7fdb84156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.963+0000 7fdb8ac00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb8415c3a0 0x7fdb841e5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:28:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.963+0000 7fdb8ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdb840c66a0 con 0x7fdb840c4fa0 2024-03-18T16:28:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.963+0000 7fdb8ac00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb840c63c0 con 0x7fdb84141650 2024-03-18T16:28:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.963+0000 7fdb8ac00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdb8415c7c0 con 0x7fdb8415c3a0 2024-03-18T16:28:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.964+0000 7fdb88e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 0x7fdb84156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.964+0000 7fdb8a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb8415c3a0 0x7fdb841e5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.964+0000 7fdb89800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb840c4fa0 0x7fdb84152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:28:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.964+0000 7fdb88e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 0x7fdb84156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54858/0 (socket says 172.21.15.102:54858) 2024-03-18T16:28:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.964+0000 7fdb88e00640 1 -- 172.21.15.102:0/2495574462 learned_addr learned my addr 172.21.15.102:0/2495574462 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:28:59.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.969+0000 7fdb8a200640 1 -- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 msgr2=0x7fdb84156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:59.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.969+0000 7fdb8a200640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 0x7fdb84156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:59.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.969+0000 7fdb8a200640 1 -- 172.21.15.102:0/2495574462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb840c4fa0 msgr2=0x7fdb84152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:28:59.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.969+0000 7fdb8a200640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb840c4fa0 0x7fdb84152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:28:59.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.969+0000 7fdb8a200640 1 -- 172.21.15.102:0/2495574462 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb841e4400 con 0x7fdb8415c3a0 2024-03-18T16:28:59.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.970+0000 7fdb8a200640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb8415c3a0 0x7fdb841e5de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00c690 tx=0x7fdb7c00cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:28:59.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.974+0000 7fdb81600640 1 -- 172.21.15.102:0/2495574462 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c003b70 con 0x7fdb8415c3a0 2024-03-18T16:28:59.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.974+0000 7fdb81600640 1 -- 172.21.15.102:0/2495574462 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb7c003d10 con 0x7fdb8415c3a0 2024-03-18T16:28:59.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.974+0000 7fdb81600640 1 -- 172.21.15.102:0/2495574462 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c005310 con 0x7fdb8415c3a0 2024-03-18T16:28:59.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.974+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb841f0320 con 0x7fdb8415c3a0 2024-03-18T16:28:59.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.974+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb841f05e0 con 0x7fdb8415c3a0 2024-03-18T16:28:59.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:28:59.976+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb84074fd0 con 0x7fdb8415c3a0 2024-03-18T16:29:00.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.235+0000 7fdb81600640 1 -- 172.21.15.102:0/2495574462 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdb7c0054b0 con 0x7fdb8415c3a0 2024-03-18T16:29:00.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.237+0000 7fdb81600640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb6406bc50 0x7fdb6406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:00.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.237+0000 7fdb81600640 1 -- 172.21.15.102:0/2495574462 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fdb7c08b4e0 con 0x7fdb8415c3a0 2024-03-18T16:29:00.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.237+0000 7fdb89800640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb6406bc50 0x7fdb6406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:00.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.238+0000 7fdb89800640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb6406bc50 0x7fdb6406e110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdb78036000 tx=0x7fdb7803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:00.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.248+0000 7fdb81600640 1 -- 172.21.15.102:0/2495574462 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdb7c059690 con 0x7fdb8415c3a0 2024-03-18T16:29:00.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.480+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb840b7f00 con 0x7fdb8415c3a0 2024-03-18T16:29:00.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.492+0000 7fdb81600640 1 -- 172.21.15.102:0/2495574462 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fdb7c05d730 con 0x7fdb8415c3a0 2024-03-18T16:29:00.494 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:00.494 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:00.495 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:00.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb6406bc50 msgr2=0x7fdb6406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:00.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb6406bc50 0x7fdb6406e110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdb78036000 tx=0x7fdb7803b040 comp rx=0 tx=0).stop 2024-03-18T16:29:00.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb8415c3a0 msgr2=0x7fdb841e5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:00.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb8415c3a0 0x7fdb841e5de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00c690 tx=0x7fdb7c00cb50 comp rx=0 tx=0).stop 2024-03-18T16:29:00.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 shutdown_connections 2024-03-18T16:29:00.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdb6406bc50 0x7fdb6406e110 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:00.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdb8415c3a0 0x7fdb841e5de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:00.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb84141650 0x7fdb84156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:00.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 --2- 172.21.15.102:0/2495574462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb840c4fa0 0x7fdb84152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:00.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 >> 172.21.15.102:0/2495574462 conn(0x7fdb840cb890 msgr2=0x7fdb840cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:00.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 shutdown_connections 2024-03-18T16:29:00.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:00.495+0000 7fdb8ac00640 1 -- 172.21.15.102:0/2495574462 wait complete. 2024-03-18T16:29:01.125 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:01.125 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:02.126 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.295+0000 7fe773200640 1 Processor -- start 2024-03-18T16:29:02.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.295+0000 7fe773200640 1 -- start start 2024-03-18T16:29:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.296+0000 7fe773200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7740ca960 0x7fe77406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.296+0000 7fe773200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7740cad40 0x7fe77415e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.296+0000 7fe773200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe77415a1a0 0x7fe7740705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.296+0000 7fe773200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe7740c8680 con 0x7fe7740ca960 2024-03-18T16:29:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.296+0000 7fe773200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe77415e670 con 0x7fe77415a1a0 2024-03-18T16:29:02.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.296+0000 7fe773200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe77406c9c0 con 0x7fe7740cad40 2024-03-18T16:29:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.297+0000 7fe771e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7740ca960 0x7fe77406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.297+0000 7fe772800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe77415a1a0 0x7fe7740705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.297+0000 7fe771400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7740cad40 0x7fe77415e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.297+0000 7fe771e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7740ca960 0x7fe77406c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53498/0 (socket says 172.21.15.102:53498) 2024-03-18T16:29:02.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.297+0000 7fe771e00640 1 -- 172.21.15.102:0/2987624863 learned_addr learned my addr 172.21.15.102:0/2987624863 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:02.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.300+0000 7fe772800640 1 -- 172.21.15.102:0/2987624863 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7740cad40 msgr2=0x7fe77415e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.300+0000 7fe772800640 1 --2- 172.21.15.102:0/2987624863 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7740cad40 0x7fe77415e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.300+0000 7fe772800640 1 -- 172.21.15.102:0/2987624863 >> v2:172.21.15.102:3300/0 conn(0x7fe7740ca960 msgr2=0x7fe77406c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.300+0000 7fe772800640 1 --2- 172.21.15.102:0/2987624863 >> v2:172.21.15.102:3300/0 conn(0x7fe7740ca960 0x7fe77406c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.300+0000 7fe772800640 1 -- 172.21.15.102:0/2987624863 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77406cca0 con 0x7fe77415a1a0 2024-03-18T16:29:02.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.302+0000 7fe772800640 1 --2- 172.21.15.102:0/2987624863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe77415a1a0 0x7fe7740705a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe77406e180 tx=0x7fe76c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91c9d6e46d516ca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.306+0000 7fe770a00640 1 -- 172.21.15.102:0/2987624863 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c039070 con 0x7fe77415a1a0 2024-03-18T16:29:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.306+0000 7fe770a00640 1 -- 172.21.15.102:0/2987624863 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe76c02cc60 con 0x7fe77415a1a0 2024-03-18T16:29:02.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.306+0000 7fe770a00640 1 -- 172.21.15.102:0/2987624863 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c034660 con 0x7fe77415a1a0 2024-03-18T16:29:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.307+0000 7fe773200640 1 -- 172.21.15.102:0/2987624863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe77415a1a0 msgr2=0x7fe7740705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.307+0000 7fe773200640 1 --2- 172.21.15.102:0/2987624863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe77415a1a0 0x7fe7740705a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe77406e180 tx=0x7fe76c02beb0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.307+0000 7fe773200640 1 -- 172.21.15.102:0/2987624863 shutdown_connections 2024-03-18T16:29:02.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.307+0000 7fe773200640 1 --2- 172.21.15.102:0/2987624863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe77415a1a0 0x7fe7740705a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.307+0000 7fe773200640 1 --2- 172.21.15.102:0/2987624863 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7740cad40 0x7fe77415e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.307+0000 7fe773200640 1 --2- 172.21.15.102:0/2987624863 >> v2:172.21.15.102:3300/0 conn(0x7fe7740ca960 0x7fe77406c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.307+0000 7fe773200640 1 -- 172.21.15.102:0/2987624863 >> 172.21.15.102:0/2987624863 conn(0x7fe774067190 msgr2=0x7fe7740675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.308+0000 7fe773200640 1 -- 172.21.15.102:0/2987624863 shutdown_connections 2024-03-18T16:29:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.308+0000 7fe773200640 1 -- 172.21.15.102:0/2987624863 wait complete. 2024-03-18T16:29:02.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.310+0000 7fe773200640 1 Processor -- start 2024-03-18T16:29:02.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.310+0000 7fe773200640 1 -- start start 2024-03-18T16:29:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.312+0000 7fe773200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7740ca960 0x7fe7741e47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.312+0000 7fe773200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7740cad40 0x7fe7741f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.312+0000 7fe773200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe77415a1a0 0x7fe7741f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.312+0000 7fe773200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe77415dc10 con 0x7fe7740ca960 2024-03-18T16:29:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.312+0000 7fe773200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe77415d930 con 0x7fe7740cad40 2024-03-18T16:29:02.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.312+0000 7fe773200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe774070ae0 con 0x7fe77415a1a0 2024-03-18T16:29:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.312+0000 7fe771400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7740cad40 0x7fe7741f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.313+0000 7fe771e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7740ca960 0x7fe7741e47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.313+0000 7fe771400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7740cad40 0x7fe7741f33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34634/0 (socket says 172.21.15.102:34634) 2024-03-18T16:29:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.313+0000 7fe771400640 1 -- 172.21.15.102:0/2632126088 learned_addr learned my addr 172.21.15.102:0/2632126088 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.313+0000 7fe772800640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe77415a1a0 0x7fe7741f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:02.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.316+0000 7fe771400640 1 -- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe77415a1a0 msgr2=0x7fe7741f3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:02.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.316+0000 7fe771400640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe77415a1a0 0x7fe7741f3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.316+0000 7fe771400640 1 -- 172.21.15.102:0/2632126088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7740ca960 msgr2=0x7fe7741e47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:02.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.316+0000 7fe771400640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7740ca960 0x7fe7741e47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.316+0000 7fe771400640 1 -- 172.21.15.102:0/2632126088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7741e66a0 con 0x7fe7740cad40 2024-03-18T16:29:02.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.319+0000 7fe771400640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7740cad40 0x7fe7741f33e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe76000e9e0 tx=0x7fe76000eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:02.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.322+0000 7fe76a000640 1 -- 172.21.15.102:0/2632126088 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760017070 con 0x7fe7740cad40 2024-03-18T16:29:02.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.322+0000 7fe76a000640 1 -- 172.21.15.102:0/2632126088 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe76000c9d0 con 0x7fe7740cad40 2024-03-18T16:29:02.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.322+0000 7fe76a000640 1 -- 172.21.15.102:0/2632126088 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76000cda0 con 0x7fe7740cad40 2024-03-18T16:29:02.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.323+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe774156c20 con 0x7fe7740cad40 2024-03-18T16:29:02.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.323+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe774156ec0 con 0x7fe7740cad40 2024-03-18T16:29:02.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.324+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7740c83c0 con 0x7fe7740cad40 2024-03-18T16:29:02.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.344+0000 7fe76a000640 1 -- 172.21.15.102:0/2632126088 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe76001a070 con 0x7fe7740cad40 2024-03-18T16:29:02.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.345+0000 7fe76a000640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe75406bc20 0x7fe75406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:02.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.346+0000 7fe771e00640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe75406bc20 0x7fe75406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:02.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.346+0000 7fe76a000640 1 -- 172.21.15.102:0/2632126088 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe760057770 con 0x7fe7740cad40 2024-03-18T16:29:02.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.346+0000 7fe771e00640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe75406bc20 0x7fe75406e0e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe764006cc0 tx=0x7fe764009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:02.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.603+0000 7fe76a000640 1 -- 172.21.15.102:0/2632126088 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe7740c83c0 con 0x7fe7740cad40 2024-03-18T16:29:02.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.834+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe774074fd0 con 0x7fe7740cad40 2024-03-18T16:29:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.849+0000 7fe76a000640 1 -- 172.21.15.102:0/2632126088 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe7600596d0 con 0x7fe7740cad40 2024-03-18T16:29:02.852 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:02.852 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:02.852 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe75406bc20 msgr2=0x7fe75406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe75406bc20 0x7fe75406e0e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe764006cc0 tx=0x7fe764009240 comp rx=0 tx=0).stop 2024-03-18T16:29:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7740cad40 msgr2=0x7fe7741f33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7740cad40 0x7fe7741f33e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe76000e9e0 tx=0x7fe76000eea0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 shutdown_connections 2024-03-18T16:29:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe75406bc20 0x7fe75406e0e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe77415a1a0 0x7fe7741f3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7740cad40 0x7fe7741f33e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 --2- 172.21.15.102:0/2632126088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7740ca960 0x7fe7741e47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 >> 172.21.15.102:0/2632126088 conn(0x7fe774067190 msgr2=0x7fe774071100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 shutdown_connections 2024-03-18T16:29:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:02.852+0000 7fe773200640 1 -- 172.21.15.102:0/2632126088 wait complete. 2024-03-18T16:29:03.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:03.476 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:04.477 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:04.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.654+0000 7f8b54a00640 1 Processor -- start 2024-03-18T16:29:04.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.655+0000 7f8b54a00640 1 -- start start 2024-03-18T16:29:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.656+0000 7f8b54a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b50158c00 0x7f8b5006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.656+0000 7f8b54a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b500c4190 con 0x7f8b5006fd80 2024-03-18T16:29:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.656+0000 7f8b54a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b5006f060 con 0x7f8b50071700 2024-03-18T16:29:04.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.656+0000 7f8b54a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8b50158fe0 con 0x7f8b50158c00 2024-03-18T16:29:04.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.657+0000 7f8b4ea00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8b5006fd80 0x7f8b50072a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59942/0 (socket says 172.21.15.102:59942) 2024-03-18T16:29:04.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.657+0000 7f8b4ea00640 1 -- 172.21.15.102:0/2574347015 learned_addr learned my addr 172.21.15.102:0/2574347015 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:04.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.657+0000 7f8b4f400640 1 --2- 172.21.15.102:0/2574347015 >> v2:172.21.15.154:3300/0 conn(0x7f8b50158c00 0x7f8b5006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.660+0000 7f8b4f400640 1 -- 172.21.15.102:0/2574347015 >> v1:172.21.15.154:6789/0 conn(0x7f8b50071700 legacy=0x7f8b50169110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.660+0000 7f8b4f400640 1 -- 172.21.15.102:0/2574347015 >> v1:172.21.15.102:6789/0 conn(0x7f8b5006fd80 legacy=0x7f8b50072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:04.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.660+0000 7f8b4f400640 1 -- 172.21.15.102:0/2574347015 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b50071cb0 con 0x7f8b50158c00 2024-03-18T16:29:04.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.665+0000 7f8b4f400640 1 --2- 172.21.15.102:0/2574347015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50158c00 0x7f8b5006f840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8b400097a0 tx=0x7f8b4002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23f763fdfdae899d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:04.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.669+0000 7f8b4e000640 1 -- 172.21.15.102:0/2574347015 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40034070 con 0x7f8b50158c00 2024-03-18T16:29:04.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.670+0000 7f8b4e000640 1 -- 172.21.15.102:0/2574347015 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b40030610 con 0x7f8b50158c00 2024-03-18T16:29:04.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.670+0000 7f8b4e000640 1 -- 172.21.15.102:0/2574347015 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b400309e0 con 0x7f8b50158c00 2024-03-18T16:29:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.670+0000 7f8b54a00640 1 -- 172.21.15.102:0/2574347015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50158c00 msgr2=0x7f8b5006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.670+0000 7f8b54a00640 1 --2- 172.21.15.102:0/2574347015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50158c00 0x7f8b5006f840 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8b400097a0 tx=0x7f8b4002b960 comp rx=0 tx=0).stop 2024-03-18T16:29:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.671+0000 7f8b54a00640 1 -- 172.21.15.102:0/2574347015 shutdown_connections 2024-03-18T16:29:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.671+0000 7f8b54a00640 1 --2- 172.21.15.102:0/2574347015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50158c00 0x7f8b5006f840 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.671+0000 7f8b54a00640 1 -- 172.21.15.102:0/2574347015 >> 172.21.15.102:0/2574347015 conn(0x7f8b500cb890 msgr2=0x7f8b500cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.671+0000 7f8b54a00640 1 -- 172.21.15.102:0/2574347015 shutdown_connections 2024-03-18T16:29:04.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.671+0000 7f8b54a00640 1 -- 172.21.15.102:0/2574347015 wait complete. 2024-03-18T16:29:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.673+0000 7f8b54a00640 1 Processor -- start 2024-03-18T16:29:04.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.674+0000 7f8b54a00640 1 -- start start 2024-03-18T16:29:04.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.675+0000 7f8b54a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b5006fd80 0x7f8b500ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:04.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.675+0000 7f8b54a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b50071700 0x7f8b5014e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:04.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.675+0000 7f8b54a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b50158c00 0x7f8b501f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:04.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.675+0000 7f8b54a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8b50072270 con 0x7f8b50071700 2024-03-18T16:29:04.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.675+0000 7f8b54a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5006f3b0 con 0x7f8b5006fd80 2024-03-18T16:29:04.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.675+0000 7f8b54a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8b50072550 con 0x7f8b50158c00 2024-03-18T16:29:04.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.676+0000 7f8b4f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b5006fd80 0x7f8b500ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:04.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.676+0000 7f8b4ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b50071700 0x7f8b5014e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:04.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.676+0000 7f8b4fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b50158c00 0x7f8b501f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:04.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.676+0000 7f8b4f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b5006fd80 0x7f8b500ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34650/0 (socket says 172.21.15.102:34650) 2024-03-18T16:29:04.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.676+0000 7f8b4f400640 1 -- 172.21.15.102:0/990794307 learned_addr learned my addr 172.21.15.102:0/990794307 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:04.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.680+0000 7f8b4f400640 1 -- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b50158c00 msgr2=0x7f8b501f7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:04.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.680+0000 7f8b4f400640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b50158c00 0x7f8b501f7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:04.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.680+0000 7f8b4f400640 1 -- 172.21.15.102:0/990794307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b50071700 msgr2=0x7f8b5014e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:04.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.680+0000 7f8b4f400640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b50071700 0x7f8b5014e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:04.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.680+0000 7f8b4f400640 1 -- 172.21.15.102:0/990794307 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b5014d500 con 0x7f8b5006fd80 2024-03-18T16:29:04.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.682+0000 7f8b4f400640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b5006fd80 0x7f8b500ca180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8b40009890 tx=0x7f8b400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:04.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.686+0000 7f8b47400640 1 -- 172.21.15.102:0/990794307 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40034040 con 0x7f8b5006fd80 2024-03-18T16:29:04.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.687+0000 7f8b47400640 1 -- 172.21.15.102:0/990794307 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b4003ad30 con 0x7f8b5006fd80 2024-03-18T16:29:04.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.687+0000 7f8b47400640 1 -- 172.21.15.102:0/990794307 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b40044630 con 0x7f8b5006fd80 2024-03-18T16:29:04.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.688+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b50160350 con 0x7f8b5006fd80 2024-03-18T16:29:04.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.688+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b5014eae0 con 0x7f8b5006fd80 2024-03-18T16:29:04.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.689+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b5014e8d0 con 0x7f8b5006fd80 2024-03-18T16:29:04.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.709+0000 7f8b47400640 1 -- 172.21.15.102:0/990794307 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8b4004b060 con 0x7f8b5006fd80 2024-03-18T16:29:04.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.711+0000 7f8b47400640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b3006bc20 0x7f8b3006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:04.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.711+0000 7f8b4ea00640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b3006bc20 0x7f8b3006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:04.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.712+0000 7f8b4ea00640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b3006bc20 0x7f8b3006e0e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8b5006ead0 tx=0x7f8b3c007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:04.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.712+0000 7f8b47400640 1 -- 172.21.15.102:0/990794307 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8b400305c0 con 0x7f8b5006fd80 2024-03-18T16:29:04.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:04.966+0000 7f8b47400640 1 -- 172.21.15.102:0/990794307 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8b4003f070 con 0x7f8b5006fd80 2024-03-18T16:29:05.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.238+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b50074fd0 con 0x7f8b5006fd80 2024-03-18T16:29:05.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.371+0000 7f8b47400640 1 -- 172.21.15.102:0/990794307 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8b4007afd0 con 0x7f8b5006fd80 2024-03-18T16:29:05.374 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:05.374 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:05.374 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b3006bc20 msgr2=0x7f8b3006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b3006bc20 0x7f8b3006e0e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8b5006ead0 tx=0x7f8b3c007e80 comp rx=0 tx=0).stop 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b5006fd80 msgr2=0x7f8b500ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b5006fd80 0x7f8b500ca180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8b40009890 tx=0x7f8b400046c0 comp rx=0 tx=0).stop 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 shutdown_connections 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8b3006bc20 0x7f8b3006e0e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8b50158c00 0x7f8b501f7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b50071700 0x7f8b5014e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 --2- 172.21.15.102:0/990794307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b5006fd80 0x7f8b500ca180 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 >> 172.21.15.102:0/990794307 conn(0x7f8b500cb890 msgr2=0x7f8b50153a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 shutdown_connections 2024-03-18T16:29:05.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:05.375+0000 7f8b54a00640 1 -- 172.21.15.102:0/990794307 wait complete. 2024-03-18T16:29:05.997 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:05.997 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:06.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:07.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.184+0000 7fb0ee600640 1 Processor -- start 2024-03-18T16:29:07.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.184+0000 7fb0ee600640 1 -- start start 2024-03-18T16:29:07.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.185+0000 7fb0ee600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb0e8141650 0x7fb0e8158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:07.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.185+0000 7fb0ee600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0e80c4fa0 0x7fb0e80c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:07.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.185+0000 7fb0ee600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0e80c72d0 con 0x7fb0e815c3a0 2024-03-18T16:29:07.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.185+0000 7fb0ee600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb0e8159370 con 0x7fb0e80c4fa0 2024-03-18T16:29:07.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.185+0000 7fb0ee600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb0e8158ad0 con 0x7fb0e8141650 2024-03-18T16:29:07.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.186+0000 7fb0ed200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb0e8141650 0x7fb0e8158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:07.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.186+0000 7fb0edc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0e80c4fa0 0x7fb0e80c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:07.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.187+0000 7fb0edc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0e80c4fa0 0x7fb0e80c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53516/0 (socket says 172.21.15.102:53516) 2024-03-18T16:29:07.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.187+0000 7fb0edc00640 1 -- 172.21.15.102:0/2260372688 learned_addr learned my addr 172.21.15.102:0/2260372688 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:07.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.189+0000 7fb0ed200640 1 -- 172.21.15.102:0/2260372688 >> v2:172.21.15.102:3300/0 conn(0x7fb0e80c4fa0 msgr2=0x7fb0e80c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:07.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.189+0000 7fb0ed200640 1 --2- 172.21.15.102:0/2260372688 >> v2:172.21.15.102:3300/0 conn(0x7fb0e80c4fa0 0x7fb0e80c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.189+0000 7fb0ed200640 1 -- 172.21.15.102:0/2260372688 >> v1:172.21.15.102:6789/0 conn(0x7fb0e815c3a0 legacy=0x7fb0e80c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:07.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.189+0000 7fb0ed200640 1 -- 172.21.15.102:0/2260372688 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e8158db0 con 0x7fb0e8141650 2024-03-18T16:29:07.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.191+0000 7fb0ed200640 1 --2- 172.21.15.102:0/2260372688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 0x7fb0e8158420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8068ac0 tx=0x7fb0dc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ea14db605654bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:07.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.193+0000 7fb0e7400640 1 -- 172.21.15.102:0/2260372688 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc0315a0 con 0x7fb0e8141650 2024-03-18T16:29:07.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.195+0000 7fb0e7400640 1 -- 172.21.15.102:0/2260372688 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0dc037070 con 0x7fb0e8141650 2024-03-18T16:29:07.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.195+0000 7fb0e7400640 1 -- 172.21.15.102:0/2260372688 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc031960 con 0x7fb0e8141650 2024-03-18T16:29:07.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.196+0000 7fb0ee600640 1 -- 172.21.15.102:0/2260372688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 msgr2=0x7fb0e8158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:07.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.196+0000 7fb0ee600640 1 --2- 172.21.15.102:0/2260372688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 0x7fb0e8158420 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8068ac0 tx=0x7fb0dc02b840 comp rx=0 tx=0).stop 2024-03-18T16:29:07.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.196+0000 7fb0ee600640 1 -- 172.21.15.102:0/2260372688 shutdown_connections 2024-03-18T16:29:07.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.196+0000 7fb0ee600640 1 --2- 172.21.15.102:0/2260372688 >> v2:172.21.15.102:3300/0 conn(0x7fb0e80c4fa0 0x7fb0e80c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.196+0000 7fb0ee600640 1 --2- 172.21.15.102:0/2260372688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 0x7fb0e8158420 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.196+0000 7fb0ee600640 1 -- 172.21.15.102:0/2260372688 >> 172.21.15.102:0/2260372688 conn(0x7fb0e80cb890 msgr2=0x7fb0e80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:07.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.196+0000 7fb0ee600640 1 -- 172.21.15.102:0/2260372688 shutdown_connections 2024-03-18T16:29:07.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.196+0000 7fb0ee600640 1 -- 172.21.15.102:0/2260372688 wait complete. 2024-03-18T16:29:07.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.199+0000 7fb0ee600640 1 Processor -- start 2024-03-18T16:29:07.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.199+0000 7fb0ee600640 1 -- start start 2024-03-18T16:29:07.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.200+0000 7fb0ee600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0e80c4fa0 0x7fb0e81574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:07.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.200+0000 7fb0ee600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 0x7fb0e813a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:07.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.200+0000 7fb0ee600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e815c3a0 0x7fb0e813a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:07.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.200+0000 7fb0ee600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e80c66a0 con 0x7fb0e815c3a0 2024-03-18T16:29:07.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.200+0000 7fb0ee600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e80c63c0 con 0x7fb0e8141650 2024-03-18T16:29:07.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.200+0000 7fb0ee600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e815c7c0 con 0x7fb0e80c4fa0 2024-03-18T16:29:07.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.201+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 0x7fb0e813a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:07.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.201+0000 7fb0edc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e815c3a0 0x7fb0e813a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:07.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.201+0000 7fb0ed200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0e80c4fa0 0x7fb0e81574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:07.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.201+0000 7fb0e7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 0x7fb0e813a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34674/0 (socket says 172.21.15.102:34674) 2024-03-18T16:29:07.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.201+0000 7fb0e7e00640 1 -- 172.21.15.102:0/1376623134 learned_addr learned my addr 172.21.15.102:0/1376623134 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:07.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.205+0000 7fb0e7e00640 1 -- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0e80c4fa0 msgr2=0x7fb0e81574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:07.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.205+0000 7fb0e7e00640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0e80c4fa0 0x7fb0e81574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.205+0000 7fb0e7e00640 1 -- 172.21.15.102:0/1376623134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e815c3a0 msgr2=0x7fb0e813a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:07.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.205+0000 7fb0e7e00640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e815c3a0 0x7fb0e813a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.205+0000 7fb0e7e00640 1 -- 172.21.15.102:0/1376623134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e8158570 con 0x7fb0e8141650 2024-03-18T16:29:07.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.208+0000 7fb0e7e00640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 0x7fb0e813a0b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400b3e0 tx=0x7fb0d400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.211+0000 7fb0e4c00640 1 -- 172.21.15.102:0/1376623134 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4015020 con 0x7fb0e8141650 2024-03-18T16:29:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.211+0000 7fb0e4c00640 1 -- 172.21.15.102:0/1376623134 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0d4011850 con 0x7fb0e8141650 2024-03-18T16:29:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.211+0000 7fb0e4c00640 1 -- 172.21.15.102:0/1376623134 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4011c40 con 0x7fb0e8141650 2024-03-18T16:29:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.211+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e81579e0 con 0x7fb0e8141650 2024-03-18T16:29:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.212+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e81e29a0 con 0x7fb0e8141650 2024-03-18T16:29:07.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.213+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e8074fd0 con 0x7fb0e8141650 2024-03-18T16:29:07.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.235+0000 7fb0e4c00640 1 -- 172.21.15.102:0/1376623134 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb0d4018020 con 0x7fb0e8141650 2024-03-18T16:29:07.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.236+0000 7fb0e4c00640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0c806bc50 0x7fb0c806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:07.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.236+0000 7fb0ed200640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0c806bc50 0x7fb0c806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:07.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.237+0000 7fb0ed200640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0c806bc50 0x7fb0c806e110 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc036000 tx=0x7fb0dc03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:07.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.237+0000 7fb0e4c00640 1 -- 172.21.15.102:0/1376623134 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb0d4011de0 con 0x7fb0e8141650 2024-03-18T16:29:07.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.489+0000 7fb0e4c00640 1 -- 172.21.15.102:0/1376623134 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb0d401f070 con 0x7fb0e8141650 2024-03-18T16:29:07.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.904+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0e80b7f00 con 0x7fb0e8141650 2024-03-18T16:29:07.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.916+0000 7fb0e4c00640 1 -- 172.21.15.102:0/1376623134 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb0e8158570 con 0x7fb0e8141650 2024-03-18T16:29:07.918 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:07.919 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:07.919 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:07.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0c806bc50 msgr2=0x7fb0c806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:07.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0c806bc50 0x7fb0c806e110 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc036000 tx=0x7fb0dc03d040 comp rx=0 tx=0).stop 2024-03-18T16:29:07.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 msgr2=0x7fb0e813a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:07.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 0x7fb0e813a0b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400b3e0 tx=0x7fb0d400b8a0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 shutdown_connections 2024-03-18T16:29:07.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0c806bc50 0x7fb0c806e110 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e815c3a0 0x7fb0e813a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0e8141650 0x7fb0e813a0b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 --2- 172.21.15.102:0/1376623134 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb0e80c4fa0 0x7fb0e81574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:07.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 >> 172.21.15.102:0/1376623134 conn(0x7fb0e80cb890 msgr2=0x7fb0e813ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:07.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 shutdown_connections 2024-03-18T16:29:07.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:07.919+0000 7fb0ee600640 1 -- 172.21.15.102:0/1376623134 wait complete. 2024-03-18T16:29:08.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:08.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:09.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:09.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.734+0000 7ff34b200640 1 Processor -- start 2024-03-18T16:29:09.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.735+0000 7ff34b200640 1 -- start start 2024-03-18T16:29:09.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.737+0000 7ff34b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3440c8b20 0x7ff344072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:09.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.737+0000 7ff34b200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff344158760 0x7ff34415b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:09.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.737+0000 7ff34b200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff344157ac0 con 0x7ff3440c8740 2024-03-18T16:29:09.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.737+0000 7ff34b200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff34415a860 con 0x7ff344158760 2024-03-18T16:29:09.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.737+0000 7ff34b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff344072840 con 0x7ff3440c8b20 2024-03-18T16:29:09.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.751+0000 7ff349200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff344158760 0x7ff34415b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:09.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.752+0000 7ff349200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff344158760 0x7ff34415b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34688/0 (socket says 172.21.15.102:34688) 2024-03-18T16:29:09.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.752+0000 7ff349200640 1 -- 172.21.15.102:0/3232542865 learned_addr learned my addr 172.21.15.102:0/3232542865 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:09.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.754+0000 7ff349200640 1 -- 172.21.15.102:0/3232542865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3440c8b20 msgr2=0x7ff344072300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:09.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.754+0000 7ff349200640 1 --2- 172.21.15.102:0/3232542865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3440c8b20 0x7ff344072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:09.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.754+0000 7ff349200640 1 -- 172.21.15.102:0/3232542865 >> v1:172.21.15.154:6789/0 conn(0x7ff3440c8740 legacy=0x7ff3441572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:09.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.754+0000 7ff349200640 1 -- 172.21.15.102:0/3232542865 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff344157da0 con 0x7ff344158760 2024-03-18T16:29:09.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.759+0000 7ff349200640 1 --2- 172.21.15.102:0/3232542865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff344158760 0x7ff34415b2b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff33c009e50 tx=0x7ff33c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=128a11df830eddc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:09.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.765+0000 7ff342a00640 1 -- 172.21.15.102:0/3232542865 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c02cb00 con 0x7ff344158760 2024-03-18T16:29:09.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.765+0000 7ff342a00640 1 -- 172.21.15.102:0/3232542865 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff33c02cca0 con 0x7ff344158760 2024-03-18T16:29:09.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.765+0000 7ff342a00640 1 -- 172.21.15.102:0/3232542865 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c034610 con 0x7ff344158760 2024-03-18T16:29:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.766+0000 7ff34b200640 1 -- 172.21.15.102:0/3232542865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff344158760 msgr2=0x7ff34415b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.766+0000 7ff34b200640 1 --2- 172.21.15.102:0/3232542865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff344158760 0x7ff34415b2b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff33c009e50 tx=0x7ff33c02bf60 comp rx=0 tx=0).stop 2024-03-18T16:29:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.766+0000 7ff34b200640 1 -- 172.21.15.102:0/3232542865 shutdown_connections 2024-03-18T16:29:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.766+0000 7ff34b200640 1 --2- 172.21.15.102:0/3232542865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff344158760 0x7ff34415b2b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.766+0000 7ff34b200640 1 --2- 172.21.15.102:0/3232542865 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff3440c8b20 0x7ff344072300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.766+0000 7ff34b200640 1 -- 172.21.15.102:0/3232542865 >> 172.21.15.102:0/3232542865 conn(0x7ff3440700d0 msgr2=0x7ff34406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:09.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.766+0000 7ff34b200640 1 -- 172.21.15.102:0/3232542865 shutdown_connections 2024-03-18T16:29:09.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.766+0000 7ff34b200640 1 -- 172.21.15.102:0/3232542865 wait complete. 2024-03-18T16:29:09.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.768+0000 7ff34b200640 1 Processor -- start 2024-03-18T16:29:09.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.769+0000 7ff34b200640 1 -- start start 2024-03-18T16:29:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.769+0000 7ff34b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3440c8740 0x7ff344148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.769+0000 7ff34b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3440c8b20 0x7ff3440c1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.769+0000 7ff34b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff344158760 0x7ff34414cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.769+0000 7ff34b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff34415aab0 con 0x7ff3440c8b20 2024-03-18T16:29:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.769+0000 7ff34b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff34415b7f0 con 0x7ff3440c8740 2024-03-18T16:29:09.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.769+0000 7ff34b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff34415ad90 con 0x7ff344158760 2024-03-18T16:29:09.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.771+0000 7ff343e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3440c8740 0x7ff344148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.771+0000 7ff343400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3440c8b20 0x7ff3440c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.771+0000 7ff343e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3440c8740 0x7ff344148ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34690/0 (socket says 172.21.15.102:34690) 2024-03-18T16:29:09.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.771+0000 7ff343e00640 1 -- 172.21.15.102:0/4140124380 learned_addr learned my addr 172.21.15.102:0/4140124380 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.775+0000 7ff343e00640 1 -- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff344158760 msgr2=0x7ff34414cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.775+0000 7ff343e00640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff344158760 0x7ff34414cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.775+0000 7ff343e00640 1 -- 172.21.15.102:0/4140124380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3440c8b20 msgr2=0x7ff3440c1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.775+0000 7ff343e00640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3440c8b20 0x7ff3440c1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:09.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.775+0000 7ff343e00640 1 -- 172.21.15.102:0/4140124380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34414b0e0 con 0x7ff3440c8740 2024-03-18T16:29:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.778+0000 7ff343e00640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3440c8740 0x7ff344148ae0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c3c0 tx=0x7ff33400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:09.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.782+0000 7ff33be00640 1 -- 172.21.15.102:0/4140124380 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3340052d0 con 0x7ff3440c8740 2024-03-18T16:29:09.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.783+0000 7ff33be00640 1 -- 172.21.15.102:0/4140124380 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff334005490 con 0x7ff3440c8740 2024-03-18T16:29:09.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.783+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3441f11e0 con 0x7ff3440c8740 2024-03-18T16:29:09.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.783+0000 7ff33be00640 1 -- 172.21.15.102:0/4140124380 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff334007260 con 0x7ff3440c8740 2024-03-18T16:29:09.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.783+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3441f0420 con 0x7ff3440c8740 2024-03-18T16:29:09.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:09.784+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3440c4390 con 0x7ff3440c8740 2024-03-18T16:29:10.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.051+0000 7ff33be00640 1 -- 172.21.15.102:0/4140124380 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff334007400 con 0x7ff3440c8740 2024-03-18T16:29:10.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.053+0000 7ff33be00640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff32406bc20 0x7ff32406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:10.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.053+0000 7ff343400640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff32406bc20 0x7ff32406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:10.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.054+0000 7ff33be00640 1 -- 172.21.15.102:0/4140124380 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff334056850 con 0x7ff3440c8740 2024-03-18T16:29:10.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.054+0000 7ff343400640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff32406bc20 0x7ff32406e0e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff34415a540 tx=0x7ff330009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.073+0000 7ff33be00640 1 -- 172.21.15.102:0/4140124380 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff3440c4390 con 0x7ff3440c8740 2024-03-18T16:29:10.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.316+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff344074fd0 con 0x7ff3440c8740 2024-03-18T16:29:10.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.329+0000 7ff33be00640 1 -- 172.21.15.102:0/4140124380 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff3340587b0 con 0x7ff3440c8740 2024-03-18T16:29:10.331 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:10.331 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:10.331 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:10.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff32406bc20 msgr2=0x7ff32406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:10.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff32406bc20 0x7ff32406e0e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff34415a540 tx=0x7ff330009060 comp rx=0 tx=0).stop 2024-03-18T16:29:10.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3440c8740 msgr2=0x7ff344148ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:10.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3440c8740 0x7ff344148ae0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c3c0 tx=0x7ff33400c880 comp rx=0 tx=0).stop 2024-03-18T16:29:10.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 shutdown_connections 2024-03-18T16:29:10.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff32406bc20 0x7ff32406e0e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff344158760 0x7ff34414cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3440c8b20 0x7ff3440c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 --2- 172.21.15.102:0/4140124380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3440c8740 0x7ff344148ae0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 >> 172.21.15.102:0/4140124380 conn(0x7ff3440700d0 msgr2=0x7ff344158b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 shutdown_connections 2024-03-18T16:29:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:10.331+0000 7ff34b200640 1 -- 172.21.15.102:0/4140124380 wait complete. 2024-03-18T16:29:10.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:10.954 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:11.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:12.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.133+0000 7f7682800640 1 Processor -- start 2024-03-18T16:29:12.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.133+0000 7f7682800640 1 -- start start 2024-03-18T16:29:12.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.134+0000 7f7682800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f767c15a310 0x7f767c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:12.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.134+0000 7f7682800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f767c15a6f0 0x7f767c15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:12.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.134+0000 7f7682800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f767c0ca500 con 0x7f767c15baa0 2024-03-18T16:29:12.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.134+0000 7f7682800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f767c0cd410 con 0x7f767c15a6f0 2024-03-18T16:29:12.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.134+0000 7f7682800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f767c15b080 con 0x7f767c15a310 2024-03-18T16:29:12.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.135+0000 7f7681400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f767c15a310 0x7f767c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:12.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.135+0000 7f7680a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f767c15a6f0 0x7f767c15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:12.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.135+0000 7f7681400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f767c15a310 0x7f767c06d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40774/0 (socket says 172.21.15.102:40774) 2024-03-18T16:29:12.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.135+0000 7f7681400640 1 -- 172.21.15.102:0/3449256936 learned_addr learned my addr 172.21.15.102:0/3449256936 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.139+0000 7f7681400640 1 -- 172.21.15.102:0/3449256936 >> v2:172.21.15.102:3300/0 conn(0x7f767c15a6f0 msgr2=0x7f767c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:12.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.139+0000 7f7681400640 1 --2- 172.21.15.102:0/3449256936 >> v2:172.21.15.102:3300/0 conn(0x7f767c15a6f0 0x7f767c15ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:12.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.139+0000 7f7681400640 1 -- 172.21.15.102:0/3449256936 >> v1:172.21.15.102:6789/0 conn(0x7f767c15baa0 legacy=0x7f767c15bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:12.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.139+0000 7f7681400640 1 -- 172.21.15.102:0/3449256936 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767c0ca7a0 con 0x7f767c15a310 2024-03-18T16:29:12.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.142+0000 7f7681400640 1 --2- 172.21.15.102:0/3449256936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 0x7f767c06d3e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f767c067db0 tx=0x7f767002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc3785dd71906c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.146+0000 7f767be00640 1 -- 172.21.15.102:0/3449256936 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7670035020 con 0x7f767c15a310 2024-03-18T16:29:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.146+0000 7f767be00640 1 -- 172.21.15.102:0/3449256936 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7670031870 con 0x7f767c15a310 2024-03-18T16:29:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.146+0000 7f767be00640 1 -- 172.21.15.102:0/3449256936 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7670031c40 con 0x7f767c15a310 2024-03-18T16:29:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.147+0000 7f7682800640 1 -- 172.21.15.102:0/3449256936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 msgr2=0x7f767c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:12.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.147+0000 7f7682800640 1 --2- 172.21.15.102:0/3449256936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 0x7f767c06d3e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f767c067db0 tx=0x7f767002b910 comp rx=0 tx=0).stop 2024-03-18T16:29:12.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.147+0000 7f7682800640 1 -- 172.21.15.102:0/3449256936 shutdown_connections 2024-03-18T16:29:12.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.147+0000 7f7682800640 1 --2- 172.21.15.102:0/3449256936 >> v2:172.21.15.102:3300/0 conn(0x7f767c15a6f0 0x7f767c15ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:12.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.147+0000 7f7682800640 1 --2- 172.21.15.102:0/3449256936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 0x7f767c06d3e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:12.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.147+0000 7f7682800640 1 -- 172.21.15.102:0/3449256936 >> 172.21.15.102:0/3449256936 conn(0x7f767c08c1a0 msgr2=0x7f767c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:12.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.147+0000 7f7682800640 1 -- 172.21.15.102:0/3449256936 shutdown_connections 2024-03-18T16:29:12.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.147+0000 7f7682800640 1 -- 172.21.15.102:0/3449256936 wait complete. 2024-03-18T16:29:12.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.149+0000 7f7682800640 1 Processor -- start 2024-03-18T16:29:12.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.149+0000 7f7682800640 1 -- start start 2024-03-18T16:29:12.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.150+0000 7f7682800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 0x7f767c0c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:12.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.150+0000 7f7682800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f767c15a6f0 0x7f767c158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:12.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.150+0000 7f7682800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f767c15baa0 0x7f767c0c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:12.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.150+0000 7f7682800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f767c0cd130 con 0x7f767c15a6f0 2024-03-18T16:29:12.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.150+0000 7f7682800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f767c0cce50 con 0x7f767c15a310 2024-03-18T16:29:12.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.150+0000 7f7682800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f767c06d920 con 0x7f767c15baa0 2024-03-18T16:29:12.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.150+0000 7f7681400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 0x7f767c0c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:12.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.150+0000 7f7681e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f767c15baa0 0x7f767c0c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:12.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.151+0000 7f7680a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f767c15a6f0 0x7f767c158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:12.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.151+0000 7f7681400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 0x7f767c0c9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40786/0 (socket says 172.21.15.102:40786) 2024-03-18T16:29:12.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.151+0000 7f7681400640 1 -- 172.21.15.102:0/1266609481 learned_addr learned my addr 172.21.15.102:0/1266609481 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:12.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.155+0000 7f7681e00640 1 -- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 msgr2=0x7f767c0c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:12.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.155+0000 7f7681e00640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 0x7f767c0c9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:12.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.155+0000 7f7681e00640 1 -- 172.21.15.102:0/1266609481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f767c15a6f0 msgr2=0x7f767c158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:12.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.155+0000 7f7681e00640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f767c15a6f0 0x7f767c158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:12.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.155+0000 7f7681e00640 1 -- 172.21.15.102:0/1266609481 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767c1f46c0 con 0x7f767c15baa0 2024-03-18T16:29:12.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.157+0000 7f7681e00640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f767c15baa0 0x7f767c0c7960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f767400be20 tx=0x7f767400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.161+0000 7f7679600640 1 -- 172.21.15.102:0/1266609481 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674019070 con 0x7f767c15baa0 2024-03-18T16:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.161+0000 7f7679600640 1 -- 172.21.15.102:0/1266609481 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7674014440 con 0x7f767c15baa0 2024-03-18T16:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.161+0000 7f7679600640 1 -- 172.21.15.102:0/1266609481 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674014870 con 0x7f767c15baa0 2024-03-18T16:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.161+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f767c1f49a0 con 0x7f767c15baa0 2024-03-18T16:29:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.161+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f767c0c7160 con 0x7f767c15baa0 2024-03-18T16:29:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.163+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f767c0c32b0 con 0x7f767c15baa0 2024-03-18T16:29:12.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.423+0000 7f7679600640 1 -- 172.21.15.102:0/1266609481 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f767401c070 con 0x7f767c15baa0 2024-03-18T16:29:12.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.424+0000 7f7679600640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f765c06bc50 0x7f765c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:12.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.424+0000 7f7679600640 1 -- 172.21.15.102:0/1266609481 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f767408b240 con 0x7f767c15baa0 2024-03-18T16:29:12.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.424+0000 7f7681400640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f765c06bc50 0x7f765c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:12.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.426+0000 7f7681400640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f765c06bc50 0x7f765c06e110 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7670009630 tx=0x7f767003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:12.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.434+0000 7f7679600640 1 -- 172.21.15.102:0/1266609481 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7674023020 con 0x7f767c15baa0 2024-03-18T16:29:12.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.694+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f767c06df20 con 0x7f767c15baa0 2024-03-18T16:29:12.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.708+0000 7f7679600640 1 -- 172.21.15.102:0/1266609481 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f76740592d0 con 0x7f767c15baa0 2024-03-18T16:29:12.710 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:12.710 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:12.710 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f765c06bc50 msgr2=0x7f765c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f765c06bc50 0x7f765c06e110 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7670009630 tx=0x7f767003c040 comp rx=0 tx=0).stop 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f767c15baa0 msgr2=0x7f767c0c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f767c15baa0 0x7f767c0c7960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f767400be20 tx=0x7f767400bf10 comp rx=0 tx=0).stop 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 shutdown_connections 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f765c06bc50 0x7f765c06e110 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f767c15baa0 0x7f767c0c7960 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f767c15a6f0 0x7f767c158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 --2- 172.21.15.102:0/1266609481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f767c15a310 0x7f767c0c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 >> 172.21.15.102:0/1266609481 conn(0x7f767c08c1a0 msgr2=0x7f767c14ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 shutdown_connections 2024-03-18T16:29:12.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:12.712+0000 7f7682800640 1 -- 172.21.15.102:0/1266609481 wait complete. 2024-03-18T16:29:13.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:13.344 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:14.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:14.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.517+0000 7feb0f200640 1 Processor -- start 2024-03-18T16:29:14.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.518+0000 7feb0f200640 1 -- start start 2024-03-18T16:29:14.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.519+0000 7feb0f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cca00 0x7feb1015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:14.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.520+0000 7feb0f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb100cabd0 con 0x7feb100cfe70 2024-03-18T16:29:14.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.520+0000 7feb0f200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb100caea0 con 0x7feb100d0250 2024-03-18T16:29:14.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.520+0000 7feb0f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feb100cb180 con 0x7feb100cca00 2024-03-18T16:29:14.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.520+0000 7feb0de00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7feb100cfe70 0x7feb100c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40504/0 (socket says 172.21.15.102:40504) 2024-03-18T16:29:14.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.520+0000 7feb0de00640 1 -- 172.21.15.102:0/31801178 learned_addr learned my addr 172.21.15.102:0/31801178 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:14.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.521+0000 7feb0e800640 1 --2- 172.21.15.102:0/31801178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cca00 0x7feb1015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.523+0000 7feb0ca00640 1 -- 172.21.15.102:0/31801178 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3053163039 0 0) 0x7feb100caea0 con 0x7feb100d0250 2024-03-18T16:29:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.523+0000 7feb0ca00640 1 -- 172.21.15.102:0/31801178 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaf4003520 con 0x7feb100d0250 2024-03-18T16:29:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.523+0000 7feb0e800640 1 -- 172.21.15.102:0/31801178 >> v1:172.21.15.154:6789/0 conn(0x7feb100d0250 legacy=0x7feb100cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.523+0000 7feb0e800640 1 -- 172.21.15.102:0/31801178 >> v1:172.21.15.102:6789/0 conn(0x7feb100cfe70 legacy=0x7feb100c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:14.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.523+0000 7feb0e800640 1 -- 172.21.15.102:0/31801178 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb100cb460 con 0x7feb100cca00 2024-03-18T16:29:14.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.526+0000 7feb0e800640 1 --2- 172.21.15.102:0/31801178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cca00 0x7feb1015ae40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feb08009dc0 tx=0x7feb0802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e028a2bf970d527 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:14.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.529+0000 7feb0ca00640 1 -- 172.21.15.102:0/31801178 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08039070 con 0x7feb100cca00 2024-03-18T16:29:14.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.529+0000 7feb0ca00640 1 -- 172.21.15.102:0/31801178 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0802cd50 con 0x7feb100cca00 2024-03-18T16:29:14.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.529+0000 7feb0ca00640 1 -- 172.21.15.102:0/31801178 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08034660 con 0x7feb100cca00 2024-03-18T16:29:14.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.530+0000 7feb0f200640 1 -- 172.21.15.102:0/31801178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cca00 msgr2=0x7feb1015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:14.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.530+0000 7feb0f200640 1 --2- 172.21.15.102:0/31801178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cca00 0x7feb1015ae40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feb08009dc0 tx=0x7feb0802bfa0 comp rx=0 tx=0).stop 2024-03-18T16:29:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.530+0000 7feb0f200640 1 -- 172.21.15.102:0/31801178 shutdown_connections 2024-03-18T16:29:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.530+0000 7feb0f200640 1 --2- 172.21.15.102:0/31801178 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cca00 0x7feb1015ae40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.530+0000 7feb0f200640 1 -- 172.21.15.102:0/31801178 >> 172.21.15.102:0/31801178 conn(0x7feb1008c3e0 msgr2=0x7feb1006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.530+0000 7feb0f200640 1 -- 172.21.15.102:0/31801178 shutdown_connections 2024-03-18T16:29:14.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.531+0000 7feb0f200640 1 -- 172.21.15.102:0/31801178 wait complete. 2024-03-18T16:29:14.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.532+0000 7feb0f200640 1 Processor -- start 2024-03-18T16:29:14.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.532+0000 7feb0f200640 1 -- start start 2024-03-18T16:29:14.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.533+0000 7feb0f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb100cca00 0x7feb10159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:14.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.533+0000 7feb0f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cfe70 0x7feb10152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:14.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.533+0000 7feb0f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb100d0250 0x7feb101f0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:14.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.533+0000 7feb0f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feb100cbd00 con 0x7feb100cca00 2024-03-18T16:29:14.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.533+0000 7feb0f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feb100cba20 con 0x7feb100d0250 2024-03-18T16:29:14.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.533+0000 7feb0f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feb100c8330 con 0x7feb100cfe70 2024-03-18T16:29:14.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.534+0000 7feb0e800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb100d0250 0x7feb101f0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:14.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.534+0000 7feb0d400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cfe70 0x7feb10152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:14.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.534+0000 7feb0de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb100cca00 0x7feb10159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:14.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.534+0000 7feb0e800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb100d0250 0x7feb101f0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40802/0 (socket says 172.21.15.102:40802) 2024-03-18T16:29:14.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.534+0000 7feb0e800640 1 -- 172.21.15.102:0/1274029645 learned_addr learned my addr 172.21.15.102:0/1274029645 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:14.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.538+0000 7feb0e800640 1 -- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cfe70 msgr2=0x7feb10152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:14.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.538+0000 7feb0e800640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cfe70 0x7feb10152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:14.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.538+0000 7feb0e800640 1 -- 172.21.15.102:0/1274029645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb100cca00 msgr2=0x7feb10159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:14.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.538+0000 7feb0e800640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb100cca00 0x7feb10159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:14.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.538+0000 7feb0e800640 1 -- 172.21.15.102:0/1274029645 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb101f0a70 con 0x7feb100d0250 2024-03-18T16:29:14.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.541+0000 7feb0e800640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb100d0250 0x7feb101f0530 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7feb08009ee0 tx=0x7feb08004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:14.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.545+0000 7feb06000640 1 -- 172.21.15.102:0/1274029645 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb08039050 con 0x7feb100d0250 2024-03-18T16:29:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.545+0000 7feb06000640 1 -- 172.21.15.102:0/1274029645 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb08045070 con 0x7feb100d0250 2024-03-18T16:29:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.545+0000 7feb06000640 1 -- 172.21.15.102:0/1274029645 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0803e850 con 0x7feb100d0250 2024-03-18T16:29:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.545+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb101e7000 con 0x7feb100d0250 2024-03-18T16:29:14.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.545+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb101550a0 con 0x7feb100d0250 2024-03-18T16:29:14.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.546+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb10074fd0 con 0x7feb100d0250 2024-03-18T16:29:14.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.565+0000 7feb06000640 1 -- 172.21.15.102:0/1274029645 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7feb0803e9f0 con 0x7feb100d0250 2024-03-18T16:29:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.566+0000 7feb06000640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feaf006bc20 0x7feaf006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.566+0000 7feb0de00640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feaf006bc20 0x7feaf006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.567+0000 7feb0de00640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feaf006bc20 0x7feaf006e0e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feb00007fd0 tx=0x7feb00007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:14.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.568+0000 7feb06000640 1 -- 172.21.15.102:0/1274029645 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7feb0807a5a0 con 0x7feb100d0250 2024-03-18T16:29:14.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:14.817+0000 7feb06000640 1 -- 172.21.15.102:0/1274029645 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feb0807c500 con 0x7feb100d0250 2024-03-18T16:29:15.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.112+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb10153a40 con 0x7feb100d0250 2024-03-18T16:29:15.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.134+0000 7feb06000640 1 -- 172.21.15.102:0/1274029645 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7feb080805a0 con 0x7feb100d0250 2024-03-18T16:29:15.136 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:15.137 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:15.137 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.136+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feaf006bc20 msgr2=0x7feaf006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.136+0000 7feb0f200640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feaf006bc20 0x7feaf006e0e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feb00007fd0 tx=0x7feb00007e80 comp rx=0 tx=0).stop 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.136+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb100d0250 msgr2=0x7feb101f0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.136+0000 7feb0f200640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb100d0250 0x7feb101f0530 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7feb08009ee0 tx=0x7feb08004a40 comp rx=0 tx=0).stop 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.137+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 shutdown_connections 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.137+0000 7feb0f200640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feaf006bc20 0x7feaf006e0e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.137+0000 7feb0f200640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb100d0250 0x7feb101f0530 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.137+0000 7feb0f200640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb100cfe70 0x7feb10152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.137+0000 7feb0f200640 1 --2- 172.21.15.102:0/1274029645 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb100cca00 0x7feb10159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.137+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 >> 172.21.15.102:0/1274029645 conn(0x7feb1008c3e0 msgr2=0x7feb100c7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.137+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 shutdown_connections 2024-03-18T16:29:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:15.137+0000 7feb0f200640 1 -- 172.21.15.102:0/1274029645 wait complete. 2024-03-18T16:29:15.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:15.759 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:16.760 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.934+0000 7f8976600640 1 Processor -- start 2024-03-18T16:29:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.934+0000 7f8976600640 1 -- start start 2024-03-18T16:29:16.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.935+0000 7f8976600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f897015c3a0 0x7f8970163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:16.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.935+0000 7f8976600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89700c72d0 con 0x7f8970158420 2024-03-18T16:29:16.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.935+0000 7f8976600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8970158840 con 0x7f8970141650 2024-03-18T16:29:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.935+0000 7f8976600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8970159370 con 0x7f897015c3a0 2024-03-18T16:29:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.936+0000 7f8975200640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8970141650 0x7f89700c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:47278/0 (socket says 172.21.15.102:47278) 2024-03-18T16:29:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.936+0000 7f8975200640 1 -- 172.21.15.102:0/3876386446 learned_addr learned my addr 172.21.15.102:0/3876386446 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.937+0000 7f8975c00640 1 --2- 172.21.15.102:0/3876386446 >> v2:172.21.15.154:3300/0 conn(0x7f897015c3a0 0x7f8970163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:16.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.939+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316694268 0 0) 0x7f8970158840 con 0x7f8970141650 2024-03-18T16:29:16.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.939+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8958003520 con 0x7f8970141650 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 224026265 0 0) 0x7f8958003520 con 0x7f8970141650 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 >> v2:172.21.15.154:3300/0 conn(0x7f897015c3a0 msgr2=0x7f8970163cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 --2- 172.21.15.102:0/3876386446 >> v2:172.21.15.154:3300/0 conn(0x7f897015c3a0 0x7f8970163cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 >> v1:172.21.15.102:6789/0 conn(0x7f8970158420 legacy=0x7f89700c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8970158ad0 con 0x7f8970141650 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f8964001670 con 0x7f8970141650 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 >> v1:172.21.15.154:6789/0 conn(0x7f8970141650 legacy=0x7f89700c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 --2- 172.21.15.102:0/3876386446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8958003a00 0x7f8958003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 --2- 172.21.15.102:0/3876386446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8958004800 0x7f8958004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 --2- 172.21.15.102:0/3876386446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89580051c0 0x7f895800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:16.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8970159370 con 0x7f8958004800 2024-03-18T16:29:16.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f895800c090 con 0x7f89580051c0 2024-03-18T16:29:16.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.944+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f895800c290 con 0x7f8958003a00 2024-03-18T16:29:16.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.945+0000 7f8975200640 1 --2- 172.21.15.102:0/3876386446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8958003a00 0x7f8958003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.949+0000 7f8975200640 1 -- 172.21.15.102:0/3876386446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89580051c0 msgr2=0x7f895800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.949+0000 7f8975200640 1 --2- 172.21.15.102:0/3876386446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89580051c0 0x7f895800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.949+0000 7f8975200640 1 -- 172.21.15.102:0/3876386446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8958004800 msgr2=0x7f8958004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.949+0000 7f8975200640 1 --2- 172.21.15.102:0/3876386446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8958004800 0x7f8958004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.949+0000 7f8975200640 1 -- 172.21.15.102:0/3876386446 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8958003520 con 0x7f8958003a00 2024-03-18T16:29:16.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.949+0000 7f8975c00640 1 -- 172.21.15.102:0/3876386446 reap_dead start 2024-03-18T16:29:16.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.952+0000 7f8975200640 1 --2- 172.21.15.102:0/3876386446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8958003a00 0x7f8958003e00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f896400c190 tx=0x7f8958005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=515c817cdb01bf30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:16.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.957+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964007c60 con 0x7f8958003a00 2024-03-18T16:29:16.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.957+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8964030070 con 0x7f8958003a00 2024-03-18T16:29:16.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.957+0000 7f896f400640 1 -- 172.21.15.102:0/3876386446 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964005d50 con 0x7f8958003a00 2024-03-18T16:29:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.958+0000 7f8976600640 1 -- 172.21.15.102:0/3876386446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8958003a00 msgr2=0x7f8958003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.958+0000 7f8976600640 1 --2- 172.21.15.102:0/3876386446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8958003a00 0x7f8958003e00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f896400c190 tx=0x7f8958005830 comp rx=0 tx=0).stop 2024-03-18T16:29:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.958+0000 7f8976600640 1 -- 172.21.15.102:0/3876386446 shutdown_connections 2024-03-18T16:29:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.958+0000 7f8976600640 1 --2- 172.21.15.102:0/3876386446 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8958003a00 0x7f8958003e00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.958+0000 7f8976600640 1 -- 172.21.15.102:0/3876386446 >> 172.21.15.102:0/3876386446 conn(0x7f89700cb890 msgr2=0x7f89700cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.958+0000 7f8976600640 1 -- 172.21.15.102:0/3876386446 shutdown_connections 2024-03-18T16:29:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.958+0000 7f8976600640 1 -- 172.21.15.102:0/3876386446 wait complete. 2024-03-18T16:29:16.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.961+0000 7f8976600640 1 Processor -- start 2024-03-18T16:29:16.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.961+0000 7f8976600640 1 -- start start 2024-03-18T16:29:16.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.962+0000 7f8976600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8958003a00 0x7f89701e2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.962+0000 7f8976600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8970153760 0x7f8970156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.962+0000 7f8976600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8970152200 0x7f89701efd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.962+0000 7f8976600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f89700c63c0 con 0x7f8958003a00 2024-03-18T16:29:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.962+0000 7f8976600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8970159090 con 0x7f8970152200 2024-03-18T16:29:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.962+0000 7f8976600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f89700c66a0 con 0x7f8970153760 2024-03-18T16:29:16.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.963+0000 7f8975c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8970152200 0x7f89701efd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:16.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.963+0000 7f896fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8970153760 0x7f8970156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:16.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.963+0000 7f8975c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8970152200 0x7f89701efd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:40830/0 (socket says 172.21.15.102:40830) 2024-03-18T16:29:16.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.963+0000 7f8975c00640 1 -- 172.21.15.102:0/1395272758 learned_addr learned my addr 172.21.15.102:0/1395272758 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:16.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.967+0000 7f896fe00640 1 -- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8970152200 msgr2=0x7f89701efd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:16.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.967+0000 7f896fe00640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8970152200 0x7f89701efd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:16.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.967+0000 7f896fe00640 1 -- 172.21.15.102:0/1395272758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8958003a00 msgr2=0x7f89701e2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:16.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.967+0000 7f896fe00640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8958003a00 0x7f89701e2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:16.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.967+0000 7f896fe00640 1 -- 172.21.15.102:0/1395272758 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89701e26d0 con 0x7f8970153760 2024-03-18T16:29:16.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.969+0000 7f896fe00640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8970153760 0x7f8970156500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8970073d10 tx=0x7f895c00c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:16.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.973+0000 7f896cc00640 1 -- 172.21.15.102:0/1395272758 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c017070 con 0x7f8970153760 2024-03-18T16:29:16.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.973+0000 7f896cc00640 1 -- 172.21.15.102:0/1395272758 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f895c00ce10 con 0x7f8970153760 2024-03-18T16:29:16.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.973+0000 7f896cc00640 1 -- 172.21.15.102:0/1395272758 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c003830 con 0x7f8970153760 2024-03-18T16:29:16.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.974+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89701f3440 con 0x7f8970153760 2024-03-18T16:29:16.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.974+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89700c4b40 con 0x7f8970153760 2024-03-18T16:29:16.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.975+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8970074fd0 con 0x7f8970153760 2024-03-18T16:29:16.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.996+0000 7f896cc00640 1 -- 172.21.15.102:0/1395272758 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f895c0039d0 con 0x7f8970153760 2024-03-18T16:29:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.997+0000 7f896cc00640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f895006bc00 0x7f895006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.997+0000 7f8975200640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f895006bc00 0x7f895006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.998+0000 7f8975200640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f895006bc00 0x7f895006e0c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8964034000 tx=0x7f8964007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:16.998+0000 7f896cc00640 1 -- 172.21.15.102:0/1395272758 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f895c056040 con 0x7f8970153760 2024-03-18T16:29:17.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.263+0000 7f896cc00640 1 -- 172.21.15.102:0/1395272758 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f895c0586a0 con 0x7f8970153760 2024-03-18T16:29:17.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.621+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89700b7f00 con 0x7f8970153760 2024-03-18T16:29:17.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.635+0000 7f896cc00640 1 -- 172.21.15.102:0/1395272758 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f895c057fa0 con 0x7f8970153760 2024-03-18T16:29:17.637 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:17.637 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:17.637 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:17.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f895006bc00 msgr2=0x7f895006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f895006bc00 0x7f895006e0c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8964034000 tx=0x7f8964007000 comp rx=0 tx=0).stop 2024-03-18T16:29:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8970153760 msgr2=0x7f8970156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8970153760 0x7f8970156500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8970073d10 tx=0x7f895c00c760 comp rx=0 tx=0).stop 2024-03-18T16:29:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 shutdown_connections 2024-03-18T16:29:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f895006bc00 0x7f895006e0c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8970152200 0x7f89701efd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8970153760 0x7f8970156500 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 --2- 172.21.15.102:0/1395272758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8958003a00 0x7f89701e2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:17.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 >> 172.21.15.102:0/1395272758 conn(0x7f89700cb890 msgr2=0x7f8970157c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:17.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.638+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 shutdown_connections 2024-03-18T16:29:17.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:17.639+0000 7f8976600640 1 -- 172.21.15.102:0/1395272758 wait complete. 2024-03-18T16:29:18.262 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:18.262 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:19.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.445+0000 7feb24800640 1 Processor -- start 2024-03-18T16:29:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.445+0000 7feb24800640 1 -- start start 2024-03-18T16:29:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.447+0000 7feb24800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb1c1524c0 0x7feb1c072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.447+0000 7feb24800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb1c15c200 con 0x7feb1c15b8e0 2024-03-18T16:29:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.447+0000 7feb24800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb1c15c4e0 con 0x7feb1c158ee0 2024-03-18T16:29:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.447+0000 7feb24800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7feb1c159300 con 0x7feb1c1524c0 2024-03-18T16:29:19.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.448+0000 7feb21400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7feb1c158ee0 0x7feb1c06e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:47284/0 (socket says 172.21.15.102:47284) 2024-03-18T16:29:19.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.448+0000 7feb21400640 1 -- 172.21.15.102:0/2995485813 learned_addr learned my addr 172.21.15.102:0/2995485813 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.497+0000 7feb21e00640 1 --2- 172.21.15.102:0/2995485813 >> v2:172.21.15.102:3300/0 conn(0x7feb1c1524c0 0x7feb1c072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:19.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.501+0000 7feb21e00640 1 -- 172.21.15.102:0/2995485813 >> v1:172.21.15.154:6789/0 conn(0x7feb1c158ee0 legacy=0x7feb1c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:19.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.501+0000 7feb21e00640 1 -- 172.21.15.102:0/2995485813 >> v1:172.21.15.102:6789/0 conn(0x7feb1c15b8e0 legacy=0x7feb1c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:19.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.501+0000 7feb21e00640 1 -- 172.21.15.102:0/2995485813 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1c15c7c0 con 0x7feb1c1524c0 2024-03-18T16:29:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.506+0000 7feb21e00640 1 --2- 172.21.15.102:0/2995485813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c1524c0 0x7feb1c072700 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb100097a0 tx=0x7feb1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f37b15105f03e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:19.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.511+0000 7feb20a00640 1 -- 172.21.15.102:0/2995485813 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10035020 con 0x7feb1c1524c0 2024-03-18T16:29:19.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.511+0000 7feb20a00640 1 -- 172.21.15.102:0/2995485813 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb10031830 con 0x7feb1c1524c0 2024-03-18T16:29:19.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.511+0000 7feb20a00640 1 -- 172.21.15.102:0/2995485813 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10031c00 con 0x7feb1c1524c0 2024-03-18T16:29:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.512+0000 7feb24800640 1 -- 172.21.15.102:0/2995485813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c1524c0 msgr2=0x7feb1c072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.512+0000 7feb24800640 1 --2- 172.21.15.102:0/2995485813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c1524c0 0x7feb1c072700 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb100097a0 tx=0x7feb1002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:29:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.512+0000 7feb24800640 1 -- 172.21.15.102:0/2995485813 shutdown_connections 2024-03-18T16:29:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.512+0000 7feb24800640 1 --2- 172.21.15.102:0/2995485813 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c1524c0 0x7feb1c072700 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.512+0000 7feb24800640 1 -- 172.21.15.102:0/2995485813 >> 172.21.15.102:0/2995485813 conn(0x7feb1c0700d0 msgr2=0x7feb1c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.512+0000 7feb24800640 1 -- 172.21.15.102:0/2995485813 shutdown_connections 2024-03-18T16:29:19.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.512+0000 7feb24800640 1 -- 172.21.15.102:0/2995485813 wait complete. 2024-03-18T16:29:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.515+0000 7feb24800640 1 Processor -- start 2024-03-18T16:29:19.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.515+0000 7feb24800640 1 -- start start 2024-03-18T16:29:19.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.517+0000 7feb24800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb1c1524c0 0x7feb1c151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.517+0000 7feb24800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c158ee0 0x7feb1c1f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.517+0000 7feb24800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb1c15b8e0 0x7feb1c1f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.517+0000 7feb24800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c1589c0 con 0x7feb1c158ee0 2024-03-18T16:29:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.517+0000 7feb24800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c1586e0 con 0x7feb1c1524c0 2024-03-18T16:29:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.517+0000 7feb24800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feb1c1528a0 con 0x7feb1c15b8e0 2024-03-18T16:29:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.517+0000 7feb21400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c158ee0 0x7feb1c1f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.518+0000 7feb21400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c158ee0 0x7feb1c1f14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54556/0 (socket says 172.21.15.102:54556) 2024-03-18T16:29:19.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.518+0000 7feb21400640 1 -- 172.21.15.102:0/2757458216 learned_addr learned my addr 172.21.15.102:0/2757458216 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.521+0000 7feb21400640 1 -- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb1c15b8e0 msgr2=0x7feb1c1f35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.521+0000 7feb21400640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb1c15b8e0 0x7feb1c1f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.521+0000 7feb21400640 1 -- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb1c1524c0 msgr2=0x7feb1c151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.521+0000 7feb21400640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb1c1524c0 0x7feb1c151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.521+0000 7feb21400640 1 -- 172.21.15.102:0/2757458216 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1c0adf30 con 0x7feb1c158ee0 2024-03-18T16:29:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.524+0000 7feb21400640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c158ee0 0x7feb1c1f14b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00c400 tx=0x7feb0c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.528+0000 7feb16000640 1 -- 172.21.15.102:0/2757458216 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c017070 con 0x7feb1c158ee0 2024-03-18T16:29:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.529+0000 7feb16000640 1 -- 172.21.15.102:0/2757458216 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0c005ce0 con 0x7feb1c158ee0 2024-03-18T16:29:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.529+0000 7feb16000640 1 -- 172.21.15.102:0/2757458216 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0072f0 con 0x7feb1c158ee0 2024-03-18T16:29:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.530+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1c0ae530 con 0x7feb1c158ee0 2024-03-18T16:29:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.530+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb1c0cb120 con 0x7feb1c158ee0 2024-03-18T16:29:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.531+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb1c072700 con 0x7feb1c158ee0 2024-03-18T16:29:19.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.810+0000 7feb16000640 1 -- 172.21.15.102:0/2757458216 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7feb0c01a070 con 0x7feb1c158ee0 2024-03-18T16:29:19.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.811+0000 7feb16000640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feb0006bc20 0x7feb0006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:19.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.811+0000 7feb21e00640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feb0006bc20 0x7feb0006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:19.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.812+0000 7feb21e00640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feb0006bc20 0x7feb0006e0e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feb10036000 tx=0x7feb10030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:19.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.834+0000 7feb16000640 1 -- 172.21.15.102:0/2757458216 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7feb0c0567d0 con 0x7feb1c158ee0 2024-03-18T16:29:19.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:19.834+0000 7feb16000640 1 -- 172.21.15.102:0/2757458216 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feb1c072700 con 0x7feb1c158ee0 2024-03-18T16:29:20.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.142+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb1c06ee80 con 0x7feb1c158ee0 2024-03-18T16:29:20.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.163+0000 7feb16000640 1 -- 172.21.15.102:0/2757458216 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7feb1c06ee80 con 0x7feb1c158ee0 2024-03-18T16:29:20.165 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:20.166 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:20.166 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:20.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feb0006bc20 msgr2=0x7feb0006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:20.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feb0006bc20 0x7feb0006e0e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feb10036000 tx=0x7feb10030000 comp rx=0 tx=0).stop 2024-03-18T16:29:20.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c158ee0 msgr2=0x7feb1c1f14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c158ee0 0x7feb1c1f14b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00c400 tx=0x7feb0c00c8c0 comp rx=0 tx=0).stop 2024-03-18T16:29:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 shutdown_connections 2024-03-18T16:29:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feb0006bc20 0x7feb0006e0e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feb1c15b8e0 0x7feb1c1f35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb1c158ee0 0x7feb1c1f14b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 --2- 172.21.15.102:0/2757458216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb1c1524c0 0x7feb1c151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 >> 172.21.15.102:0/2757458216 conn(0x7feb1c0700d0 msgr2=0x7feb1c1579f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.167+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 shutdown_connections 2024-03-18T16:29:20.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:20.168+0000 7feb24800640 1 -- 172.21.15.102:0/2757458216 wait complete. 2024-03-18T16:29:20.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:20.790 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:21.791 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:21.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.967+0000 7f6272400640 1 Processor -- start 2024-03-18T16:29:21.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.967+0000 7f6272400640 1 -- start start 2024-03-18T16:29:21.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.968+0000 7f6272400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f626c141650 0x7f626c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:21.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.968+0000 7f6272400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f626c0c72d0 con 0x7f626c15c3a0 2024-03-18T16:29:21.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.968+0000 7f6272400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f626c158ad0 con 0x7f626c0c4fa0 2024-03-18T16:29:21.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.968+0000 7f6272400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f626c159370 con 0x7f626c141650 2024-03-18T16:29:21.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.969+0000 7f6271000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f626c141650 0x7f626c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:21.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.970+0000 7f6271000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f626c141650 0x7f626c158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34902/0 (socket says 172.21.15.102:34902) 2024-03-18T16:29:21.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.970+0000 7f6271000640 1 -- 172.21.15.102:0/3717971859 learned_addr learned my addr 172.21.15.102:0/3717971859 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:21.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.974+0000 7f6271000640 1 -- 172.21.15.102:0/3717971859 >> v1:172.21.15.154:6789/0 conn(0x7f626c0c4fa0 legacy=0x7f626c162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:21.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.974+0000 7f6271000640 1 -- 172.21.15.102:0/3717971859 >> v1:172.21.15.102:6789/0 conn(0x7f626c15c3a0 legacy=0x7f626c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:21.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.974+0000 7f6271000640 1 -- 172.21.15.102:0/3717971859 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626c158db0 con 0x7f626c141650 2024-03-18T16:29:21.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.984+0000 7f6271000640 1 --2- 172.21.15.102:0/3717971859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c141650 0x7f626c158420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6260009870 tx=0x7f626002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3ffb8c5a37be389 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:21.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.990+0000 7f626b400640 1 -- 172.21.15.102:0/3717971859 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260034070 con 0x7f626c141650 2024-03-18T16:29:21.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.990+0000 7f626b400640 1 -- 172.21.15.102:0/3717971859 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6260030860 con 0x7f626c141650 2024-03-18T16:29:21.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.990+0000 7f626b400640 1 -- 172.21.15.102:0/3717971859 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260030c30 con 0x7f626c141650 2024-03-18T16:29:21.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.991+0000 7f6272400640 1 -- 172.21.15.102:0/3717971859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c141650 msgr2=0x7f626c158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:21.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.991+0000 7f6272400640 1 --2- 172.21.15.102:0/3717971859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c141650 0x7f626c158420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6260009870 tx=0x7f626002b9a0 comp rx=0 tx=0).stop 2024-03-18T16:29:21.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.991+0000 7f6272400640 1 -- 172.21.15.102:0/3717971859 shutdown_connections 2024-03-18T16:29:21.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.991+0000 7f6272400640 1 --2- 172.21.15.102:0/3717971859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c141650 0x7f626c158420 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:21.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.991+0000 7f6272400640 1 -- 172.21.15.102:0/3717971859 >> 172.21.15.102:0/3717971859 conn(0x7f626c0cb890 msgr2=0x7f626c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:21.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.991+0000 7f6272400640 1 -- 172.21.15.102:0/3717971859 shutdown_connections 2024-03-18T16:29:21.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.991+0000 7f6272400640 1 -- 172.21.15.102:0/3717971859 wait complete. 2024-03-18T16:29:21.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.994+0000 7f6272400640 1 Processor -- start 2024-03-18T16:29:21.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.994+0000 7f6272400640 1 -- start start 2024-03-18T16:29:21.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.995+0000 7f6272400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c0c4fa0 0x7f626c156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:21.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.995+0000 7f6272400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f626c141650 0x7f626c13c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:21.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.996+0000 7f6272400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15c3a0 0x7f626c137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:21.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.996+0000 7f6272400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f626c0c66a0 con 0x7f626c141650 2024-03-18T16:29:21.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.996+0000 7f6272400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f626c0c63c0 con 0x7f626c0c4fa0 2024-03-18T16:29:21.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:21.996+0000 7f6272400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f626c15c7c0 con 0x7f626c15c3a0 2024-03-18T16:29:22.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.045+0000 7f626be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f626c141650 0x7f626c13c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:22.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.045+0000 7f626be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f626c141650 0x7f626c13c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59198/0 (socket says 172.21.15.102:59198) 2024-03-18T16:29:22.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.045+0000 7f626be00640 1 -- 172.21.15.102:0/2978386038 learned_addr learned my addr 172.21.15.102:0/2978386038 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:22.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.050+0000 7f626be00640 1 -- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15c3a0 msgr2=0x7f626c137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:22.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.050+0000 7f626be00640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15c3a0 0x7f626c137f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:22.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.050+0000 7f626be00640 1 -- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c0c4fa0 msgr2=0x7f626c156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:22.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.050+0000 7f626be00640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c0c4fa0 0x7f626c156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:22.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.050+0000 7f626be00640 1 -- 172.21.15.102:0/2978386038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626c1e5750 con 0x7f626c141650 2024-03-18T16:29:22.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.054+0000 7f626be00640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f626c141650 0x7f626c13c9e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f625800b410 tx=0x7f625800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:22.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.059+0000 7f6268c00640 1 -- 172.21.15.102:0/2978386038 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258015020 con 0x7f626c141650 2024-03-18T16:29:22.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.059+0000 7f6268c00640 1 -- 172.21.15.102:0/2978386038 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6258011830 con 0x7f626c141650 2024-03-18T16:29:22.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.059+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626c1f0220 con 0x7f626c141650 2024-03-18T16:29:22.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.059+0000 7f6268c00640 1 -- 172.21.15.102:0/2978386038 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258011c20 con 0x7f626c141650 2024-03-18T16:29:22.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.060+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f626c1f07e0 con 0x7f626c141650 2024-03-18T16:29:22.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.061+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f626c074fd0 con 0x7f626c141650 2024-03-18T16:29:22.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.344+0000 7f6268c00640 1 -- 172.21.15.102:0/2978386038 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6258018020 con 0x7f626c141650 2024-03-18T16:29:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.345+0000 7f6268c00640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f624c06bc20 0x7f624c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:22.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.346+0000 7f6271000640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f624c06bc20 0x7f624c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:22.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.346+0000 7f6271000640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f624c06bc20 0x7f624c06e0e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6260036000 tx=0x7f626003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:22.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.360+0000 7f6268c00640 1 -- 172.21.15.102:0/2978386038 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6258057470 con 0x7f626c141650 2024-03-18T16:29:22.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.360+0000 7f6268c00640 1 -- 172.21.15.102:0/2978386038 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f625801f020 con 0x7f626c141650 2024-03-18T16:29:22.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:29:22.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:29:22.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.659+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f626c0b7f00 con 0x7f626c141650 2024-03-18T16:29:22.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.681+0000 7f6268c00640 1 -- 172.21.15.102:0/2978386038 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f625800f2e0 con 0x7f626c141650 2024-03-18T16:29:22.685 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:22.686 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:22.686 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:22.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f624c06bc20 msgr2=0x7f624c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:22.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f624c06bc20 0x7f624c06e0e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6260036000 tx=0x7f626003b040 comp rx=0 tx=0).stop 2024-03-18T16:29:22.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f626c141650 msgr2=0x7f626c13c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:22.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f626c141650 0x7f626c13c9e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f625800b410 tx=0x7f625800b8d0 comp rx=0 tx=0).stop 2024-03-18T16:29:22.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 shutdown_connections 2024-03-18T16:29:22.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f624c06bc20 0x7f624c06e0e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:22.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f626c15c3a0 0x7f626c137f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:22.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f626c141650 0x7f626c13c9e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:22.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 --2- 172.21.15.102:0/2978386038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f626c0c4fa0 0x7f626c156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:22.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 >> 172.21.15.102:0/2978386038 conn(0x7f626c0cb890 msgr2=0x7f626c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:22.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 shutdown_connections 2024-03-18T16:29:22.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:22.685+0000 7f6272400640 1 -- 172.21.15.102:0/2978386038 wait complete. 2024-03-18T16:29:23.310 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:23.310 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:24.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.490+0000 7f1228c00640 1 Processor -- start 2024-03-18T16:29:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.490+0000 7f1228c00640 1 -- start start 2024-03-18T16:29:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.491+0000 7f1228c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 0x7f122406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.491+0000 7f1228c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f122406fd80 0x7f1224072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.492+0000 7f1228c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12240c4190 con 0x7f1224071700 2024-03-18T16:29:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.492+0000 7f1228c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1224158fe0 con 0x7f122406fd80 2024-03-18T16:29:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.492+0000 7f1228c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f122406f3b0 con 0x7f1224158c00 2024-03-18T16:29:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.493+0000 7f1222a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f122406fd80 0x7f1224072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.493+0000 7f1222a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f122406fd80 0x7f1224072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59204/0 (socket says 172.21.15.102:59204) 2024-03-18T16:29:24.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.493+0000 7f1222a00640 1 -- 172.21.15.102:0/1587547030 learned_addr learned my addr 172.21.15.102:0/1587547030 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:24.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.493+0000 7f1223400640 1 --2- 172.21.15.102:0/1587547030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 0x7f122406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:24.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.495+0000 7f1222a00640 1 -- 172.21.15.102:0/1587547030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 msgr2=0x7f122406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:24.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.495+0000 7f1222a00640 1 --2- 172.21.15.102:0/1587547030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 0x7f122406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.495+0000 7f1222a00640 1 -- 172.21.15.102:0/1587547030 >> v1:172.21.15.102:6789/0 conn(0x7f1224071700 legacy=0x7f122415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.495+0000 7f1222a00640 1 -- 172.21.15.102:0/1587547030 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122406f060 con 0x7f122406fd80 2024-03-18T16:29:24.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.499+0000 7f1222a00640 1 --2- 172.21.15.102:0/1587547030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f122406fd80 0x7f1224072a70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1210009870 tx=0x7f121002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81572a7afdf188c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:24.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.504+0000 7f1222000640 1 -- 172.21.15.102:0/1587547030 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210035020 con 0x7f122406fd80 2024-03-18T16:29:24.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.504+0000 7f1222000640 1 -- 172.21.15.102:0/1587547030 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1210031640 con 0x7f122406fd80 2024-03-18T16:29:24.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.504+0000 7f1222000640 1 -- 172.21.15.102:0/1587547030 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121002d040 con 0x7f122406fd80 2024-03-18T16:29:24.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.505+0000 7f1228c00640 1 -- 172.21.15.102:0/1587547030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f122406fd80 msgr2=0x7f1224072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:24.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.505+0000 7f1228c00640 1 --2- 172.21.15.102:0/1587547030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f122406fd80 0x7f1224072a70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1210009870 tx=0x7f121002b8f0 comp rx=0 tx=0).stop 2024-03-18T16:29:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.506+0000 7f1228c00640 1 -- 172.21.15.102:0/1587547030 shutdown_connections 2024-03-18T16:29:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.506+0000 7f1228c00640 1 --2- 172.21.15.102:0/1587547030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f122406fd80 0x7f1224072a70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.506+0000 7f1228c00640 1 --2- 172.21.15.102:0/1587547030 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 0x7f122406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.506+0000 7f1228c00640 1 -- 172.21.15.102:0/1587547030 >> 172.21.15.102:0/1587547030 conn(0x7f12240cb890 msgr2=0x7f12240cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.506+0000 7f1228c00640 1 -- 172.21.15.102:0/1587547030 shutdown_connections 2024-03-18T16:29:24.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.506+0000 7f1228c00640 1 -- 172.21.15.102:0/1587547030 wait complete. 2024-03-18T16:29:24.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.508+0000 7f1228c00640 1 Processor -- start 2024-03-18T16:29:24.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.508+0000 7f1228c00640 1 -- start start 2024-03-18T16:29:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.509+0000 7f1228c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122406fd80 0x7f1224147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.509+0000 7f1228c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1224071700 0x7f12240ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.509+0000 7f1228c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 0x7f122414a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.509+0000 7f1228c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1224072270 con 0x7f1224071700 2024-03-18T16:29:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.509+0000 7f1228c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1224071f90 con 0x7f122406fd80 2024-03-18T16:29:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.509+0000 7f1228c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1224072550 con 0x7f1224158c00 2024-03-18T16:29:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.510+0000 7f1223400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122406fd80 0x7f1224147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.510+0000 7f1222a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1224071700 0x7f12240ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.510+0000 7f1223e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 0x7f122414a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.510+0000 7f1223e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 0x7f122414a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43108/0 (socket says 172.21.15.102:43108) 2024-03-18T16:29:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.510+0000 7f1223e00640 1 -- 172.21.15.102:0/1651067573 learned_addr learned my addr 172.21.15.102:0/1651067573 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:24.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.514+0000 7f1223400640 1 -- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 msgr2=0x7f122414a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:24.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.514+0000 7f1223400640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 0x7f122414a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:24.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.514+0000 7f1223400640 1 -- 172.21.15.102:0/1651067573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1224071700 msgr2=0x7f12240ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:24.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.514+0000 7f1223400640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1224071700 0x7f12240ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:24.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.514+0000 7f1223400640 1 -- 172.21.15.102:0/1651067573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122415f990 con 0x7f122406fd80 2024-03-18T16:29:24.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.516+0000 7f1223400640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122406fd80 0x7f1224147730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f121400e8a0 tx=0x7f121400ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:24.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.519+0000 7f121b400640 1 -- 172.21.15.102:0/1651067573 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214017070 con 0x7f122406fd80 2024-03-18T16:29:24.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.519+0000 7f121b400640 1 -- 172.21.15.102:0/1651067573 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f121400c940 con 0x7f122406fd80 2024-03-18T16:29:24.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.519+0000 7f121b400640 1 -- 172.21.15.102:0/1651067573 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214009e60 con 0x7f122406fd80 2024-03-18T16:29:24.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.519+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1224148ba0 con 0x7f122406fd80 2024-03-18T16:29:24.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.520+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f122415f6b0 con 0x7f122406fd80 2024-03-18T16:29:24.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.521+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12241541b0 con 0x7f122406fd80 2024-03-18T16:29:24.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.541+0000 7f121b400640 1 -- 172.21.15.102:0/1651067573 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f121401a070 con 0x7f122406fd80 2024-03-18T16:29:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.542+0000 7f121b400640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f120406bc00 0x7f120406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.543+0000 7f1222a00640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f120406bc00 0x7f120406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:24.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.544+0000 7f1222a00640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f120406bc00 0x7f120406e0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1210036000 tx=0x7f121003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:24.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.544+0000 7f121b400640 1 -- 172.21.15.102:0/1651067573 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1214057510 con 0x7f122406fd80 2024-03-18T16:29:24.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:24.795+0000 7f121b400640 1 -- 172.21.15.102:0/1651067573 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f12241541b0 con 0x7f122406fd80 2024-03-18T16:29:25.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.068+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1224153240 con 0x7f122406fd80 2024-03-18T16:29:25.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.124+0000 7f121b400640 1 -- 172.21.15.102:0/1651067573 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1224153240 con 0x7f122406fd80 2024-03-18T16:29:25.128 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:25.128 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:25.128 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:25.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f120406bc00 msgr2=0x7f120406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:25.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f120406bc00 0x7f120406e0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1210036000 tx=0x7f121003c040 comp rx=0 tx=0).stop 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122406fd80 msgr2=0x7f1224147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122406fd80 0x7f1224147730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f121400e8a0 tx=0x7f121400ed60 comp rx=0 tx=0).stop 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 shutdown_connections 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f120406bc00 0x7f120406e0c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1224158c00 0x7f122414a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1224071700 0x7f12240ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 --2- 172.21.15.102:0/1651067573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f122406fd80 0x7f1224147730 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.128+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 >> 172.21.15.102:0/1651067573 conn(0x7f12240cb890 msgr2=0x7f12240c4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.129+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 shutdown_connections 2024-03-18T16:29:25.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:25.129+0000 7f1228c00640 1 -- 172.21.15.102:0/1651067573 wait complete. 2024-03-18T16:29:25.760 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:25.761 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:26.762 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:26.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.933+0000 7fe526e00640 1 Processor -- start 2024-03-18T16:29:26.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.933+0000 7fe526e00640 1 -- start start 2024-03-18T16:29:26.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.934+0000 7fe526e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 0x7fe52015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:26.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.934+0000 7fe526e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe52015baa0 0x7fe52015bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:26.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.934+0000 7fe526e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe52015c470 con 0x7fe52015a310 2024-03-18T16:29:26.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.934+0000 7fe526e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe5200cd410 con 0x7fe52015baa0 2024-03-18T16:29:26.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.934+0000 7fe526e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe5200ca500 con 0x7fe52015a6f0 2024-03-18T16:29:26.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.935+0000 7fe525a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe52015a310 0x7fe52006d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:56436/0 (socket says 172.21.15.102:56436) 2024-03-18T16:29:26.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.935+0000 7fe525a00640 1 -- 172.21.15.102:0/2576383349 learned_addr learned my addr 172.21.15.102:0/2576383349 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:26.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.935+0000 7fe525000640 1 --2- 172.21.15.102:0/2576383349 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 0x7fe52015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:26.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.938+0000 7fe525000640 1 -- 172.21.15.102:0/2576383349 >> v2:172.21.15.102:3300/0 conn(0x7fe52015baa0 msgr2=0x7fe52015bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:26.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.938+0000 7fe525000640 1 --2- 172.21.15.102:0/2576383349 >> v2:172.21.15.102:3300/0 conn(0x7fe52015baa0 0x7fe52015bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:26.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.938+0000 7fe525000640 1 -- 172.21.15.102:0/2576383349 >> v1:172.21.15.154:6789/0 conn(0x7fe52015a310 legacy=0x7fe52006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:26.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.938+0000 7fe525000640 1 -- 172.21.15.102:0/2576383349 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52015b050 con 0x7fe52015a6f0 2024-03-18T16:29:26.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.940+0000 7fe525000640 1 --2- 172.21.15.102:0/2576383349 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 0x7fe52015ab10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe50c0097a0 tx=0x7fe50c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fe9ba817400e261 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:26.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.943+0000 7fe51fe00640 1 -- 172.21.15.102:0/2576383349 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c035020 con 0x7fe52015a6f0 2024-03-18T16:29:26.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.943+0000 7fe51fe00640 1 -- 172.21.15.102:0/2576383349 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe50c031810 con 0x7fe52015a6f0 2024-03-18T16:29:26.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.943+0000 7fe51fe00640 1 -- 172.21.15.102:0/2576383349 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c031be0 con 0x7fe52015a6f0 2024-03-18T16:29:26.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.944+0000 7fe526e00640 1 -- 172.21.15.102:0/2576383349 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 msgr2=0x7fe52015ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:26.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.944+0000 7fe526e00640 1 --2- 172.21.15.102:0/2576383349 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 0x7fe52015ab10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe50c0097a0 tx=0x7fe50c02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:29:26.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.944+0000 7fe526e00640 1 -- 172.21.15.102:0/2576383349 shutdown_connections 2024-03-18T16:29:26.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.944+0000 7fe526e00640 1 --2- 172.21.15.102:0/2576383349 >> v2:172.21.15.102:3300/0 conn(0x7fe52015baa0 0x7fe52015bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:26.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.944+0000 7fe526e00640 1 --2- 172.21.15.102:0/2576383349 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 0x7fe52015ab10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:26.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.944+0000 7fe526e00640 1 -- 172.21.15.102:0/2576383349 >> 172.21.15.102:0/2576383349 conn(0x7fe52008c1a0 msgr2=0x7fe520066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:26.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.944+0000 7fe526e00640 1 -- 172.21.15.102:0/2576383349 shutdown_connections 2024-03-18T16:29:26.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.945+0000 7fe526e00640 1 -- 172.21.15.102:0/2576383349 wait complete. 2024-03-18T16:29:26.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.946+0000 7fe526e00640 1 Processor -- start 2024-03-18T16:29:26.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.946+0000 7fe526e00640 1 -- start start 2024-03-18T16:29:26.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.947+0000 7fe526e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe52015a310 0x7fe5200c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:26.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.947+0000 7fe526e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 0x7fe5200c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:26.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.947+0000 7fe526e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe52015baa0 0x7fe520158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:26.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.947+0000 7fe526e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe5200cce50 con 0x7fe52015baa0 2024-03-18T16:29:26.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.947+0000 7fe526e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe5200caa80 con 0x7fe52015a310 2024-03-18T16:29:26.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.947+0000 7fe526e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe5200cd130 con 0x7fe52015a6f0 2024-03-18T16:29:26.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.948+0000 7fe525a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe52015a310 0x7fe5200c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:26.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.948+0000 7fe525000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 0x7fe5200c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:26.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.948+0000 7fe525a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe52015a310 0x7fe5200c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34924/0 (socket says 172.21.15.102:34924) 2024-03-18T16:29:26.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.948+0000 7fe525a00640 1 -- 172.21.15.102:0/2544733559 learned_addr learned my addr 172.21.15.102:0/2544733559 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:26.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.952+0000 7fe525a00640 1 -- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 msgr2=0x7fe5200c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:26.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.952+0000 7fe525a00640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 0x7fe5200c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:26.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.952+0000 7fe525a00640 1 -- 172.21.15.102:0/2544733559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe52015baa0 msgr2=0x7fe520158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:26.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.952+0000 7fe525a00640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe52015baa0 0x7fe520158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:26.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.952+0000 7fe525a00640 1 -- 172.21.15.102:0/2544733559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5200c9de0 con 0x7fe52015a310 2024-03-18T16:29:26.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.957+0000 7fe525a00640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe52015a310 0x7fe5200c4fa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe51400c3c0 tx=0x7fe51400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:26.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.960+0000 7fe51d600640 1 -- 172.21.15.102:0/2544733559 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe514017070 con 0x7fe52015a310 2024-03-18T16:29:26.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.960+0000 7fe51d600640 1 -- 172.21.15.102:0/2544733559 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe514005ce0 con 0x7fe52015a310 2024-03-18T16:29:26.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.960+0000 7fe51d600640 1 -- 172.21.15.102:0/2544733559 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5140072f0 con 0x7fe52015a310 2024-03-18T16:29:26.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.960+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5201f0540 con 0x7fe52015a310 2024-03-18T16:29:26.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.961+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5201f0b00 con 0x7fe52015a310 2024-03-18T16:29:26.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:26.962+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe520074fd0 con 0x7fe52015a310 2024-03-18T16:29:27.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.237+0000 7fe51d600640 1 -- 172.21.15.102:0/2544733559 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe51401a070 con 0x7fe52015a310 2024-03-18T16:29:27.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.239+0000 7fe51d600640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe50006bc20 0x7fe50006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:27.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.239+0000 7fe525000640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe50006bc20 0x7fe50006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:27.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.240+0000 7fe51d600640 1 -- 172.21.15.102:0/2544733559 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe514056770 con 0x7fe52015a310 2024-03-18T16:29:27.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.241+0000 7fe525000640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe50006bc20 0x7fe50006e0e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe50c036000 tx=0x7fe50c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:27.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.252+0000 7fe51d600640 1 -- 172.21.15.102:0/2544733559 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe514021020 con 0x7fe52015a310 2024-03-18T16:29:27.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.498+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5200af370 con 0x7fe52015a310 2024-03-18T16:29:27.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.627+0000 7fe51d600640 1 -- 172.21.15.102:0/2544733559 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe5140586d0 con 0x7fe52015a310 2024-03-18T16:29:27.630 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:27.630 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:27.630 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:27.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe50006bc20 msgr2=0x7fe50006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:27.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe50006bc20 0x7fe50006e0e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe50c036000 tx=0x7fe50c03c040 comp rx=0 tx=0).stop 2024-03-18T16:29:27.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe52015a310 msgr2=0x7fe5200c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:27.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe52015a310 0x7fe5200c4fa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe51400c3c0 tx=0x7fe51400c880 comp rx=0 tx=0).stop 2024-03-18T16:29:27.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 shutdown_connections 2024-03-18T16:29:27.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe50006bc20 0x7fe50006e0e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:27.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe52015baa0 0x7fe520158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:27.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe52015a6f0 0x7fe5200c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:27.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 --2- 172.21.15.102:0/2544733559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe52015a310 0x7fe5200c4fa0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:27.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 >> 172.21.15.102:0/2544733559 conn(0x7fe52008c1a0 msgr2=0x7fe520153ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:27.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 shutdown_connections 2024-03-18T16:29:27.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:27.630+0000 7fe526e00640 1 -- 172.21.15.102:0/2544733559 wait complete. 2024-03-18T16:29:28.261 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:28.261 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:29.262 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:29.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.450+0000 7f04d1a00640 1 Processor -- start 2024-03-18T16:29:29.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.451+0000 7f04d1a00640 1 -- start start 2024-03-18T16:29:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.452+0000 7f04d1a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f04cc158c00 0x7f04cc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.452+0000 7f04d1a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04cc0c4190 con 0x7f04cc06fd80 2024-03-18T16:29:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.452+0000 7f04d1a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04cc06f060 con 0x7f04cc071700 2024-03-18T16:29:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.452+0000 7f04d1a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f04cc158fe0 con 0x7f04cc158c00 2024-03-18T16:29:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.453+0000 7f04cbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f04cc158c00 0x7f04cc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.453+0000 7f04cb400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f04cc06fd80 0x7f04cc072a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40156/0 (socket says 172.21.15.102:40156) 2024-03-18T16:29:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.453+0000 7f04cb400640 1 -- 172.21.15.102:0/2165915329 learned_addr learned my addr 172.21.15.102:0/2165915329 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:29.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.456+0000 7f04caa00640 1 -- 172.21.15.102:0/2165915329 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 977674210 0 0) 0x7f04cc0c4190 con 0x7f04cc06fd80 2024-03-18T16:29:29.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.456+0000 7f04caa00640 1 -- 172.21.15.102:0/2165915329 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04b0003520 con 0x7f04cc06fd80 2024-03-18T16:29:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.457+0000 7f04cbe00640 1 -- 172.21.15.102:0/2165915329 >> v1:172.21.15.154:6789/0 conn(0x7f04cc071700 legacy=0x7f04cc169110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.457+0000 7f04cbe00640 1 -- 172.21.15.102:0/2165915329 >> v1:172.21.15.102:6789/0 conn(0x7f04cc06fd80 legacy=0x7f04cc072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.458+0000 7f04cbe00640 1 -- 172.21.15.102:0/2165915329 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04cc071cb0 con 0x7f04cc158c00 2024-03-18T16:29:29.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.463+0000 7f04cbe00640 1 --2- 172.21.15.102:0/2165915329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04cc158c00 0x7f04cc06f840 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f04bc0097a0 tx=0x7f04bc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b592ff4435209ebe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:29.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.467+0000 7f04caa00640 1 -- 172.21.15.102:0/2165915329 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc034070 con 0x7f04cc158c00 2024-03-18T16:29:29.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.467+0000 7f04caa00640 1 -- 172.21.15.102:0/2165915329 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04bc030610 con 0x7f04cc158c00 2024-03-18T16:29:29.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.467+0000 7f04caa00640 1 -- 172.21.15.102:0/2165915329 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc0309e0 con 0x7f04cc158c00 2024-03-18T16:29:29.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.468+0000 7f04d1a00640 1 -- 172.21.15.102:0/2165915329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04cc158c00 msgr2=0x7f04cc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:29.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.468+0000 7f04d1a00640 1 --2- 172.21.15.102:0/2165915329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04cc158c00 0x7f04cc06f840 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f04bc0097a0 tx=0x7f04bc02b960 comp rx=0 tx=0).stop 2024-03-18T16:29:29.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.468+0000 7f04d1a00640 1 -- 172.21.15.102:0/2165915329 shutdown_connections 2024-03-18T16:29:29.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.469+0000 7f04d1a00640 1 --2- 172.21.15.102:0/2165915329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04cc158c00 0x7f04cc06f840 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:29.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.469+0000 7f04d1a00640 1 -- 172.21.15.102:0/2165915329 >> 172.21.15.102:0/2165915329 conn(0x7f04cc0cb890 msgr2=0x7f04cc0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:29.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.469+0000 7f04d1a00640 1 -- 172.21.15.102:0/2165915329 shutdown_connections 2024-03-18T16:29:29.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.469+0000 7f04d1a00640 1 -- 172.21.15.102:0/2165915329 wait complete. 2024-03-18T16:29:29.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.471+0000 7f04d1a00640 1 Processor -- start 2024-03-18T16:29:29.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.471+0000 7f04d1a00640 1 -- start start 2024-03-18T16:29:29.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.472+0000 7f04d1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04cc06fd80 0x7f04cc0c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:29.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.472+0000 7f04d1a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04cc071700 0x7f04cc152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:29.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.472+0000 7f04d1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc158c00 0x7f04cc1f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:29.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.472+0000 7f04d1a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc072270 con 0x7f04cc158c00 2024-03-18T16:29:29.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.472+0000 7f04d1a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04cc06f3b0 con 0x7f04cc071700 2024-03-18T16:29:29.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.472+0000 7f04d1a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f04cc072550 con 0x7f04cc06fd80 2024-03-18T16:29:29.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.474+0000 7f04d1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc158c00 0x7f04cc1f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:29.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.474+0000 7f04d1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc158c00 0x7f04cc1f7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59222/0 (socket says 172.21.15.102:59222) 2024-03-18T16:29:29.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.474+0000 7f04d1000640 1 -- 172.21.15.102:0/3087505713 learned_addr learned my addr 172.21.15.102:0/3087505713 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:29.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.475+0000 7f04cb400640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04cc071700 0x7f04cc152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:29.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.476+0000 7f04cbe00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04cc06fd80 0x7f04cc0c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:29.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.480+0000 7f04d1000640 1 -- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04cc06fd80 msgr2=0x7f04cc0c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.480+0000 7f04d1000640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04cc06fd80 0x7f04cc0c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.480+0000 7f04d1000640 1 -- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04cc071700 msgr2=0x7f04cc152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.480+0000 7f04d1000640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04cc071700 0x7f04cc152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.480+0000 7f04d1000640 1 -- 172.21.15.102:0/3087505713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04cc0c71c0 con 0x7f04cc158c00 2024-03-18T16:29:29.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.480+0000 7f04cbe00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04cc06fd80 0x7f04cc0c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:29:29.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.485+0000 7f04d1000640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc158c00 0x7f04cc1f7a10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f04c400c7a0 tx=0x7f04c400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.492+0000 7f04c3e00640 1 -- 172.21.15.102:0/3087505713 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4003b70 con 0x7f04cc158c00 2024-03-18T16:29:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.492+0000 7f04c3e00640 1 -- 172.21.15.102:0/3087505713 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c4003d30 con 0x7f04cc158c00 2024-03-18T16:29:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.492+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04cc0c74a0 con 0x7f04cc158c00 2024-03-18T16:29:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.492+0000 7f04c3e00640 1 -- 172.21.15.102:0/3087505713 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c400aa90 con 0x7f04cc158c00 2024-03-18T16:29:29.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.493+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04cc0c7a60 con 0x7f04cc158c00 2024-03-18T16:29:29.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.494+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04cc0c8070 con 0x7f04cc158c00 2024-03-18T16:29:29.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.613+0000 7f04c3e00640 1 -- 172.21.15.102:0/3087505713 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f04c400ac30 con 0x7f04cc158c00 2024-03-18T16:29:29.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.614+0000 7f04c3e00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04ac06bc20 0x7f04ac06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:29.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.615+0000 7f04cbe00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04ac06bc20 0x7f04ac06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:29.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.615+0000 7f04cbe00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04ac06bc20 0x7f04ac06e0e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f04bc036000 tx=0x7f04bc03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:29.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.972+0000 7f04c3e00640 1 -- 172.21.15.102:0/3087505713 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f04c4057850 con 0x7f04cc158c00 2024-03-18T16:29:29.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:29.973+0000 7f04c3e00640 1 -- 172.21.15.102:0/3087505713 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f04c40597b0 con 0x7f04cc158c00 2024-03-18T16:29:30.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.204+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04cc074fd0 con 0x7f04cc158c00 2024-03-18T16:29:30.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.226+0000 7f04c3e00640 1 -- 172.21.15.102:0/3087505713 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f04c405d850 con 0x7f04cc158c00 2024-03-18T16:29:30.230 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:30.230 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:30.231 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04ac06bc20 msgr2=0x7f04ac06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04ac06bc20 0x7f04ac06e0e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f04bc036000 tx=0x7f04bc03b040 comp rx=0 tx=0).stop 2024-03-18T16:29:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc158c00 msgr2=0x7f04cc1f7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc158c00 0x7f04cc1f7a10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f04c400c7a0 tx=0x7f04c400cc60 comp rx=0 tx=0).stop 2024-03-18T16:29:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 shutdown_connections 2024-03-18T16:29:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04ac06bc20 0x7f04ac06e0e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04cc158c00 0x7f04cc1f7a10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04cc071700 0x7f04cc152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 --2- 172.21.15.102:0/3087505713 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04cc06fd80 0x7f04cc0c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 >> 172.21.15.102:0/3087505713 conn(0x7f04cc0cb890 msgr2=0x7f04cc14bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 shutdown_connections 2024-03-18T16:29:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:30.229+0000 7f04d1a00640 1 -- 172.21.15.102:0/3087505713 wait complete. 2024-03-18T16:29:30.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:30.861 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:31.862 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:32.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.030+0000 7f615b200640 1 Processor -- start 2024-03-18T16:29:32.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.030+0000 7f615b200640 1 -- start start 2024-03-18T16:29:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.031+0000 7f615b200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f615c154770 0x7f615c15ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.031+0000 7f615b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615c0ca5c0 con 0x7f615c15e290 2024-03-18T16:29:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.031+0000 7f615b200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f615c0ca860 con 0x7f615c0ce2f0 2024-03-18T16:29:32.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.031+0000 7f615b200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f615c15e7f0 con 0x7f615c154770 2024-03-18T16:29:32.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.032+0000 7f6159400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f615c15e290 0x7f615c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:48758/0 (socket says 172.21.15.102:48758) 2024-03-18T16:29:32.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.032+0000 7f6159400640 1 -- 172.21.15.102:0/215705468 learned_addr learned my addr 172.21.15.102:0/215705468 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:32.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.032+0000 7f6159e00640 1 --2- 172.21.15.102:0/215705468 >> v2:172.21.15.154:3300/0 conn(0x7f615c154770 0x7f615c15ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:32.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.036+0000 7f6158a00640 1 -- 172.21.15.102:0/215705468 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 88383824 0 0) 0x7f615c0ca5c0 con 0x7f615c15e290 2024-03-18T16:29:32.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.036+0000 7f6158a00640 1 -- 172.21.15.102:0/215705468 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6140003520 con 0x7f615c15e290 2024-03-18T16:29:32.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.037+0000 7f6159e00640 1 -- 172.21.15.102:0/215705468 >> v1:172.21.15.154:6789/0 conn(0x7f615c0ce2f0 legacy=0x7f615c164bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:32.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.037+0000 7f6159e00640 1 -- 172.21.15.102:0/215705468 >> v1:172.21.15.102:6789/0 conn(0x7f615c15e290 legacy=0x7f615c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:32.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.037+0000 7f6159e00640 1 -- 172.21.15.102:0/215705468 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c15a420 con 0x7f615c154770 2024-03-18T16:29:32.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.042+0000 7f6159e00640 1 --2- 172.21.15.102:0/215705468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c154770 0x7f615c15ac20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f614c0097a0 tx=0x7f614c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7271760d24368464 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:32.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.046+0000 7f6158a00640 1 -- 172.21.15.102:0/215705468 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c035020 con 0x7f615c154770 2024-03-18T16:29:32.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.047+0000 7f6158a00640 1 -- 172.21.15.102:0/215705468 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f614c031830 con 0x7f615c154770 2024-03-18T16:29:32.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.047+0000 7f6158a00640 1 -- 172.21.15.102:0/215705468 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c031c00 con 0x7f615c154770 2024-03-18T16:29:32.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.047+0000 7f615b200640 1 -- 172.21.15.102:0/215705468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c154770 msgr2=0x7f615c15ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:32.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.047+0000 7f615b200640 1 --2- 172.21.15.102:0/215705468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c154770 0x7f615c15ac20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f614c0097a0 tx=0x7f614c02b8d0 comp rx=0 tx=0).stop 2024-03-18T16:29:32.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.048+0000 7f615b200640 1 -- 172.21.15.102:0/215705468 shutdown_connections 2024-03-18T16:29:32.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.048+0000 7f615b200640 1 --2- 172.21.15.102:0/215705468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c154770 0x7f615c15ac20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:32.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.048+0000 7f615b200640 1 -- 172.21.15.102:0/215705468 >> 172.21.15.102:0/215705468 conn(0x7f615c066f80 msgr2=0x7f615c067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:32.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.048+0000 7f615b200640 1 -- 172.21.15.102:0/215705468 shutdown_connections 2024-03-18T16:29:32.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.048+0000 7f615b200640 1 -- 172.21.15.102:0/215705468 wait complete. 2024-03-18T16:29:32.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.050+0000 7f615b200640 1 Processor -- start 2024-03-18T16:29:32.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.050+0000 7f615b200640 1 -- start start 2024-03-18T16:29:32.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.051+0000 7f615b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0ce2f0 0x7f615c1f0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:32.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.051+0000 7f615b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f615c154770 0x7f615c1f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:32.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.051+0000 7f615b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f615c15e290 0x7f615c1572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.051+0000 7f615b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f615c0cdf60 con 0x7f615c15e290 2024-03-18T16:29:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.051+0000 7f615b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f615c15a790 con 0x7f615c0ce2f0 2024-03-18T16:29:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.051+0000 7f615b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f615c0ced90 con 0x7f615c154770 2024-03-18T16:29:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.051+0000 7f615a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f615c15e290 0x7f615c1572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.051+0000 7f6159e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0ce2f0 0x7f615c1f0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.052+0000 7f6159400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f615c154770 0x7f615c1f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.052+0000 7f6159e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0ce2f0 0x7f615c1f0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:43782/0 (socket says 172.21.15.102:43782) 2024-03-18T16:29:32.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.052+0000 7f6159e00640 1 -- 172.21.15.102:0/2391735218 learned_addr learned my addr 172.21.15.102:0/2391735218 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:32.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.055+0000 7f6159e00640 1 -- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f615c154770 msgr2=0x7f615c1f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:32.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.055+0000 7f6159e00640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f615c154770 0x7f615c1f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:32.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.055+0000 7f6159e00640 1 -- 172.21.15.102:0/2391735218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f615c15e290 msgr2=0x7f615c1572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:32.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.055+0000 7f6159e00640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f615c15e290 0x7f615c1572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:32.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.055+0000 7f6159e00640 1 -- 172.21.15.102:0/2391735218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c15b970 con 0x7f615c0ce2f0 2024-03-18T16:29:32.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.057+0000 7f6159e00640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0ce2f0 0x7f615c1f0d20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f614c02bee0 tx=0x7f614c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:32.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.062+0000 7f6152000640 1 -- 172.21.15.102:0/2391735218 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c035040 con 0x7f615c0ce2f0 2024-03-18T16:29:32.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.062+0000 7f6152000640 1 -- 172.21.15.102:0/2391735218 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f614c03b920 con 0x7f615c0ce2f0 2024-03-18T16:29:32.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.062+0000 7f6152000640 1 -- 172.21.15.102:0/2391735218 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c03bd10 con 0x7f615c0ce2f0 2024-03-18T16:29:32.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.062+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615c15c740 con 0x7f615c0ce2f0 2024-03-18T16:29:32.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.062+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f615c1f0520 con 0x7f615c0ce2f0 2024-03-18T16:29:32.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.063+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f615c15aec0 con 0x7f615c0ce2f0 2024-03-18T16:29:32.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.099+0000 7f6152000640 1 -- 172.21.15.102:0/2391735218 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f614c038020 con 0x7f615c0ce2f0 2024-03-18T16:29:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.100+0000 7f6152000640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f613c06bc20 0x7f613c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.101+0000 7f6159400640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f613c06bc20 0x7f613c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.101+0000 7f6159400640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f613c06bc20 0x7f613c06e0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f615c15a1d0 tx=0x7f6148007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.101+0000 7f6152000640 1 -- 172.21.15.102:0/2391735218 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f614c0792f0 con 0x7f615c0ce2f0 2024-03-18T16:29:32.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.360+0000 7f6152000640 1 -- 172.21.15.102:0/2391735218 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f615c15aec0 con 0x7f615c0ce2f0 2024-03-18T16:29:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.599+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f615c074fd0 con 0x7f615c0ce2f0 2024-03-18T16:29:32.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.620+0000 7f6152000640 1 -- 172.21.15.102:0/2391735218 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f614c031800 con 0x7f615c0ce2f0 2024-03-18T16:29:32.622 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:32.622 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:32.622 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:32.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.623+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f613c06bc20 msgr2=0x7f613c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.623+0000 7f615b200640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f613c06bc20 0x7f613c06e0e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f615c15a1d0 tx=0x7f6148007d30 comp rx=0 tx=0).stop 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.623+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0ce2f0 msgr2=0x7f615c1f0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.623+0000 7f615b200640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0ce2f0 0x7f615c1f0d20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f614c02bee0 tx=0x7f614c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.624+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 shutdown_connections 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.624+0000 7f615b200640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f613c06bc20 0x7f613c06e0e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.624+0000 7f615b200640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f615c15e290 0x7f615c1572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.624+0000 7f615b200640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f615c154770 0x7f615c1f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.624+0000 7f615b200640 1 --2- 172.21.15.102:0/2391735218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f615c0ce2f0 0x7f615c1f0d20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.624+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 >> 172.21.15.102:0/2391735218 conn(0x7f615c066f80 msgr2=0x7f615c153610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.624+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 shutdown_connections 2024-03-18T16:29:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:32.624+0000 7f615b200640 1 -- 172.21.15.102:0/2391735218 wait complete. 2024-03-18T16:29:33.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:33.246 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:34.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:34.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.421+0000 7fedc7200640 1 Processor -- start 2024-03-18T16:29:34.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.422+0000 7fedc7200640 1 -- start start 2024-03-18T16:29:34.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.423+0000 7fedc7200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fedc80d0250 0x7fedc815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:34.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.423+0000 7fedc7200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fedc80cc2f0 0x7fedc80cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:34.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.423+0000 7fedc7200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc80ccc80 con 0x7fedc80cfe70 2024-03-18T16:29:34.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.423+0000 7fedc7200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fedc80cabd0 con 0x7fedc80cc2f0 2024-03-18T16:29:34.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.423+0000 7fedc7200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fedc80caea0 con 0x7fedc80d0250 2024-03-18T16:29:34.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.423+0000 7fedc5400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fedc80d0250 0x7fedc815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:34.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.424+0000 7fedc6800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fedc80cc2f0 0x7fedc80cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:34.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.424+0000 7fedc6800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fedc80cc2f0 0x7fedc80cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49784/0 (socket says 172.21.15.102:49784) 2024-03-18T16:29:34.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.424+0000 7fedc6800640 1 -- 172.21.15.102:0/3533433592 learned_addr learned my addr 172.21.15.102:0/3533433592 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:34.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.426+0000 7fedc6800640 1 -- 172.21.15.102:0/3533433592 >> v2:172.21.15.154:3300/0 conn(0x7fedc80d0250 msgr2=0x7fedc815ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:34.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.426+0000 7fedc6800640 1 --2- 172.21.15.102:0/3533433592 >> v2:172.21.15.154:3300/0 conn(0x7fedc80d0250 0x7fedc815ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:34.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.426+0000 7fedc6800640 1 -- 172.21.15.102:0/3533433592 >> v1:172.21.15.102:6789/0 conn(0x7fedc80cfe70 legacy=0x7fedc80c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:34.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.426+0000 7fedc6800640 1 -- 172.21.15.102:0/3533433592 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc80cb180 con 0x7fedc80cc2f0 2024-03-18T16:29:34.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.429+0000 7fedc6800640 1 --2- 172.21.15.102:0/3533433592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 0x7fedc80cc740 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fedc80c6370 tx=0x7fedc002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b6e06df3b7ed47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:34.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.435+0000 7fedc4a00640 1 -- 172.21.15.102:0/3533433592 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc002cb20 con 0x7fedc80cc2f0 2024-03-18T16:29:34.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.435+0000 7fedc4a00640 1 -- 172.21.15.102:0/3533433592 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedc002ccc0 con 0x7fedc80cc2f0 2024-03-18T16:29:34.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.435+0000 7fedc4a00640 1 -- 172.21.15.102:0/3533433592 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc0034610 con 0x7fedc80cc2f0 2024-03-18T16:29:34.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.436+0000 7fedc7200640 1 -- 172.21.15.102:0/3533433592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 msgr2=0x7fedc80cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:34.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.436+0000 7fedc7200640 1 --2- 172.21.15.102:0/3533433592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 0x7fedc80cc740 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fedc80c6370 tx=0x7fedc002bf80 comp rx=0 tx=0).stop 2024-03-18T16:29:34.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.436+0000 7fedc7200640 1 -- 172.21.15.102:0/3533433592 shutdown_connections 2024-03-18T16:29:34.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.436+0000 7fedc7200640 1 --2- 172.21.15.102:0/3533433592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 0x7fedc80cc740 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:34.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.436+0000 7fedc7200640 1 --2- 172.21.15.102:0/3533433592 >> v2:172.21.15.154:3300/0 conn(0x7fedc80d0250 0x7fedc815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:34.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.436+0000 7fedc7200640 1 -- 172.21.15.102:0/3533433592 >> 172.21.15.102:0/3533433592 conn(0x7fedc808c3e0 msgr2=0x7fedc806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:34.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.436+0000 7fedc7200640 1 -- 172.21.15.102:0/3533433592 shutdown_connections 2024-03-18T16:29:34.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.436+0000 7fedc7200640 1 -- 172.21.15.102:0/3533433592 wait complete. 2024-03-18T16:29:34.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.438+0000 7fedc7200640 1 Processor -- start 2024-03-18T16:29:34.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.438+0000 7fedc7200640 1 -- start start 2024-03-18T16:29:34.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.439+0000 7fedc7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 0x7fedc81510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:34.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.439+0000 7fedc7200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc80cfe70 0x7fedc8157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:34.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.439+0000 7fedc7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedc80d0250 0x7fedc8155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:34.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.439+0000 7fedc7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fedc80cba20 con 0x7fedc80cc2f0 2024-03-18T16:29:34.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.439+0000 7fedc7200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fedc80cb740 con 0x7fedc80cfe70 2024-03-18T16:29:34.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.439+0000 7fedc7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fedc80cbd00 con 0x7fedc80d0250 2024-03-18T16:29:34.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.440+0000 7fedc5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 0x7fedc81510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:34.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.440+0000 7fedc5400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc80cfe70 0x7fedc8157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:34.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.440+0000 7fedc6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedc80d0250 0x7fedc8155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:34.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.440+0000 7fedc5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 0x7fedc81510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49794/0 (socket says 172.21.15.102:49794) 2024-03-18T16:29:34.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.440+0000 7fedc5e00640 1 -- 172.21.15.102:0/3111008734 learned_addr learned my addr 172.21.15.102:0/3111008734 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:34.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.444+0000 7fedc6800640 1 -- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc80cfe70 msgr2=0x7fedc8157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:34.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.444+0000 7fedc6800640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc80cfe70 0x7fedc8157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:34.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.444+0000 7fedc6800640 1 -- 172.21.15.102:0/3111008734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 msgr2=0x7fedc81510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:34.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.444+0000 7fedc6800640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 0x7fedc81510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:34.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.444+0000 7fedc6800640 1 -- 172.21.15.102:0/3111008734 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc81e16b0 con 0x7fedc80d0250 2024-03-18T16:29:34.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.446+0000 7fedc6800640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedc80d0250 0x7fedc8155260 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fedc0032000 tx=0x7fedc0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:34.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.451+0000 7fedbe000640 1 -- 172.21.15.102:0/3111008734 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc0043020 con 0x7fedc80d0250 2024-03-18T16:29:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.451+0000 7fedbe000640 1 -- 172.21.15.102:0/3111008734 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedc003e450 con 0x7fedc80d0250 2024-03-18T16:29:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.451+0000 7fedbe000640 1 -- 172.21.15.102:0/3111008734 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc0009510 con 0x7fedc80d0250 2024-03-18T16:29:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.451+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc81e6260 con 0x7fedc80d0250 2024-03-18T16:29:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.451+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc81e5ca0 con 0x7fedc80d0250 2024-03-18T16:29:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.452+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc8153f40 con 0x7fedc80d0250 2024-03-18T16:29:34.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.484+0000 7fedbe000640 1 -- 172.21.15.102:0/3111008734 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fedc0046020 con 0x7fedc80d0250 2024-03-18T16:29:34.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.485+0000 7fedbe000640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feda806bc20 0x7feda806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:34.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.485+0000 7fedc5e00640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feda806bc20 0x7feda806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:34.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.486+0000 7fedc5e00640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feda806bc20 0x7feda806e0e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fedb80018b0 tx=0x7fedb8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:34.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.490+0000 7fedbe000640 1 -- 172.21.15.102:0/3111008734 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fedc0079d10 con 0x7fedc80d0250 2024-03-18T16:29:34.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.749+0000 7fedbe000640 1 -- 172.21.15.102:0/3111008734 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fedc8153f40 con 0x7fedc80d0250 2024-03-18T16:29:34.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:34.992+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedc8074fd0 con 0x7fedc80d0250 2024-03-18T16:29:35.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.007+0000 7fedbe000640 1 -- 172.21.15.102:0/3111008734 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fedc004d030 con 0x7fedc80d0250 2024-03-18T16:29:35.009 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:35.009 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:35.009 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feda806bc20 msgr2=0x7feda806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feda806bc20 0x7feda806e0e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fedb80018b0 tx=0x7fedb8009040 comp rx=0 tx=0).stop 2024-03-18T16:29:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedc80d0250 msgr2=0x7fedc8155260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedc80d0250 0x7fedc8155260 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fedc0032000 tx=0x7fedc0004a40 comp rx=0 tx=0).stop 2024-03-18T16:29:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 shutdown_connections 2024-03-18T16:29:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feda806bc20 0x7feda806e0e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fedc80d0250 0x7fedc8155260 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:35.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc80cfe70 0x7fedc8157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:35.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 --2- 172.21.15.102:0/3111008734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fedc80cc2f0 0x7fedc81510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:35.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 >> 172.21.15.102:0/3111008734 conn(0x7fedc808c3e0 msgr2=0x7fedc80cdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:35.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 shutdown_connections 2024-03-18T16:29:35.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:35.009+0000 7fedc7200640 1 -- 172.21.15.102:0/3111008734 wait complete. 2024-03-18T16:29:35.632 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:35.632 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:36.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:36.680 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:02.940 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:29:36.786 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:03.131 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:29:36.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:36.817+0000 7fd943000640 1 Processor -- start 2024-03-18T16:29:36.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:36.817+0000 7fd943000640 1 -- start start 2024-03-18T16:29:36.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:36.818+0000 7fd943000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd93c158c00 0x7fd93c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:36.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:36.818+0000 7fd943000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd93c071700 0x7fd93c169110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:36.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:36.818+0000 7fd943000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd93c0c4190 con 0x7fd93c06fd80 2024-03-18T16:29:36.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:36.818+0000 7fd943000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd93c158fe0 con 0x7fd93c158c00 2024-03-18T16:29:36.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:36.818+0000 7fd943000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd93c06f3b0 con 0x7fd93c071700 2024-03-18T16:29:36.860 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:03.119 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:29:36.862 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:03.207 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:29:36.947 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:09:03.292 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:29:37.005 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:09:03.350 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:29:37.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.072+0000 7fd941c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd93c158c00 0x7fd93c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:37.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.073+0000 7fd941c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd93c158c00 0x7fd93c06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49806/0 (socket says 172.21.15.102:49806) 2024-03-18T16:29:37.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.073+0000 7fd941c00640 1 -- 172.21.15.102:0/4215010886 learned_addr learned my addr 172.21.15.102:0/4215010886 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:37.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.078+0000 7fd941c00640 1 -- 172.21.15.102:0/4215010886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd93c071700 msgr2=0x7fd93c169110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:37.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.078+0000 7fd941c00640 1 --2- 172.21.15.102:0/4215010886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd93c071700 0x7fd93c169110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:37.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.078+0000 7fd941c00640 1 -- 172.21.15.102:0/4215010886 >> v1:172.21.15.102:6789/0 conn(0x7fd93c06fd80 legacy=0x7fd93c072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:37.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.078+0000 7fd941c00640 1 -- 172.21.15.102:0/4215010886 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93c06f060 con 0x7fd93c158c00 2024-03-18T16:29:37.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.108+0000 7fd941c00640 1 --2- 172.21.15.102:0/4215010886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd93c158c00 0x7fd93c06f840 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd930009870 tx=0x7fd93002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1ceeede2a3381d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:37.313 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:03.658 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:29:37.333 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:03.592 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:29:37.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.362+0000 7fd93be00640 1 -- 172.21.15.102:0/4215010886 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd930035020 con 0x7fd93c158c00 2024-03-18T16:29:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.362+0000 7fd93be00640 1 -- 172.21.15.102:0/4215010886 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd930031640 con 0x7fd93c158c00 2024-03-18T16:29:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.362+0000 7fd93be00640 1 -- 172.21.15.102:0/4215010886 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93002d040 con 0x7fd93c158c00 2024-03-18T16:29:37.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.363+0000 7fd943000640 1 -- 172.21.15.102:0/4215010886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd93c158c00 msgr2=0x7fd93c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:37.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.363+0000 7fd943000640 1 --2- 172.21.15.102:0/4215010886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd93c158c00 0x7fd93c06f840 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd930009870 tx=0x7fd93002b8f0 comp rx=0 tx=0).stop 2024-03-18T16:29:37.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.363+0000 7fd943000640 1 -- 172.21.15.102:0/4215010886 shutdown_connections 2024-03-18T16:29:37.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.363+0000 7fd943000640 1 --2- 172.21.15.102:0/4215010886 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd93c071700 0x7fd93c169110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:37.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.363+0000 7fd943000640 1 --2- 172.21.15.102:0/4215010886 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd93c158c00 0x7fd93c06f840 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:37.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.363+0000 7fd943000640 1 -- 172.21.15.102:0/4215010886 >> 172.21.15.102:0/4215010886 conn(0x7fd93c0cb890 msgr2=0x7fd93c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:37.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.364+0000 7fd943000640 1 -- 172.21.15.102:0/4215010886 shutdown_connections 2024-03-18T16:29:37.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.364+0000 7fd943000640 1 -- 172.21.15.102:0/4215010886 wait complete. 2024-03-18T16:29:37.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.366+0000 7fd943000640 1 Processor -- start 2024-03-18T16:29:37.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.366+0000 7fd943000640 1 -- start start 2024-03-18T16:29:37.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.368+0000 7fd943000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd93c06fd80 0x7fd93c0c73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:37.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.368+0000 7fd943000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd93c071700 0x7fd93c161de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:37.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.368+0000 7fd943000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd93c158c00 0x7fd93c1f7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:37.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.368+0000 7fd943000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c072270 con 0x7fd93c158c00 2024-03-18T16:29:37.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.368+0000 7fd943000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd93c071f90 con 0x7fd93c06fd80 2024-03-18T16:29:37.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.368+0000 7fd943000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd93c072550 con 0x7fd93c071700 2024-03-18T16:29:37.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.388+0000 7fd941c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd93c06fd80 0x7fd93c0c73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:37.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.392+0000 7fd941c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd93c06fd80 0x7fd93c0c73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:43812/0 (socket says 172.21.15.102:43812) 2024-03-18T16:29:37.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.392+0000 7fd941c00640 1 -- 172.21.15.102:0/1872957794 learned_addr learned my addr 172.21.15.102:0/1872957794 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:37.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.408+0000 7fd941c00640 1 -- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd93c071700 msgr2=0x7fd93c161de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:37.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.408+0000 7fd941c00640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd93c071700 0x7fd93c161de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:37.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.408+0000 7fd941c00640 1 -- 172.21.15.102:0/1872957794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd93c158c00 msgr2=0x7fd93c1f7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:37.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.408+0000 7fd941c00640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd93c158c00 0x7fd93c1f7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:37.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.408+0000 7fd941c00640 1 -- 172.21.15.102:0/1872957794 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93c1f5810 con 0x7fd93c06fd80 2024-03-18T16:29:37.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.418+0000 7fd941c00640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd93c06fd80 0x7fd93c0c73b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd930000c00 tx=0x7fd9300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.435+0000 7fd939600640 1 -- 172.21.15.102:0/1872957794 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd930040070 con 0x7fd93c06fd80 2024-03-18T16:29:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.435+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd93c15f350 con 0x7fd93c06fd80 2024-03-18T16:29:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.435+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd93c151ef0 con 0x7fd93c06fd80 2024-03-18T16:29:37.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.436+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd93c06c910 con 0x7fd93c06fd80 2024-03-18T16:29:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.440+0000 7fd939600640 1 -- 172.21.15.102:0/1872957794 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9300315d0 con 0x7fd93c06fd80 2024-03-18T16:29:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.440+0000 7fd939600640 1 -- 172.21.15.102:0/1872957794 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd930035040 con 0x7fd93c06fd80 2024-03-18T16:29:37.450 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:09:03.795 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:29:37.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.710+0000 7fd939600640 1 -- 172.21.15.102:0/1872957794 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd93004b060 con 0x7fd93c06fd80 2024-03-18T16:29:37.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.711+0000 7fd939600640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd91c06bbd0 0x7fd91c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:37.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.712+0000 7fd941200640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd91c06bbd0 0x7fd91c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:37.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.713+0000 7fd941200640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd91c06bbd0 0x7fd91c06e090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd93c162900 tx=0x7fd928007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:37.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.715+0000 7fd939600640 1 -- 172.21.15.102:0/1872957794 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd930039e10 con 0x7fd93c06fd80 2024-03-18T16:29:37.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:37.727+0000 7fd939600640 1 -- 172.21.15.102:0/1872957794 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd93c06c910 con 0x7fd93c06fd80 2024-03-18T16:29:38.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.016+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd93c074fd0 con 0x7fd93c06fd80 2024-03-18T16:29:38.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.230+0000 7fd939600640 1 -- 172.21.15.102:0/1872957794 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd93002f300 con 0x7fd93c06fd80 2024-03-18T16:29:38.232 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:38.232 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:38.232 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd91c06bbd0 msgr2=0x7fd91c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd91c06bbd0 0x7fd91c06e090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd93c162900 tx=0x7fd928007990 comp rx=0 tx=0).stop 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd93c06fd80 msgr2=0x7fd93c0c73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd93c06fd80 0x7fd93c0c73b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd930000c00 tx=0x7fd9300046c0 comp rx=0 tx=0).stop 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 shutdown_connections 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd91c06bbd0 0x7fd91c06e090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd93c158c00 0x7fd93c1f7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd93c071700 0x7fd93c161de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 --2- 172.21.15.102:0/1872957794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd93c06fd80 0x7fd93c0c73b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 >> 172.21.15.102:0/1872957794 conn(0x7fd93c0cb890 msgr2=0x7fd93c157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:38.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 shutdown_connections 2024-03-18T16:29:38.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:38.233+0000 7fd943000640 1 -- 172.21.15.102:0/1872957794 wait complete. 2024-03-18T16:29:38.856 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:38.856 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:39.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:40.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.028+0000 7f455f200640 1 Processor -- start 2024-03-18T16:29:40.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.028+0000 7f455f200640 1 -- start start 2024-03-18T16:29:40.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.029+0000 7f455f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4560141650 0x7f4560158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.029+0000 7f455f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.029+0000 7f455f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45600c72d0 con 0x7f456015c3a0 2024-03-18T16:29:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.029+0000 7f455f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4560159370 con 0x7f4560141650 2024-03-18T16:29:40.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.029+0000 7f455f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4560158ad0 con 0x7f45600c4fa0 2024-03-18T16:29:40.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.030+0000 7f455e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:40.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.030+0000 7f455de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4560141650 0x7f4560158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:40.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.030+0000 7f455e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:55050/0 (socket says 172.21.15.102:55050) 2024-03-18T16:29:40.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.030+0000 7f455e800640 1 -- 172.21.15.102:0/36655227 learned_addr learned my addr 172.21.15.102:0/36655227 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.033+0000 7f455e800640 1 -- 172.21.15.102:0/36655227 >> v2:172.21.15.102:3300/0 conn(0x7f4560141650 msgr2=0x7f4560158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.033+0000 7f455e800640 1 --2- 172.21.15.102:0/36655227 >> v2:172.21.15.102:3300/0 conn(0x7f4560141650 0x7f4560158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.033+0000 7f455e800640 1 -- 172.21.15.102:0/36655227 >> v1:172.21.15.102:6789/0 conn(0x7f456015c3a0 legacy=0x7f45600c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.033+0000 7f455e800640 1 -- 172.21.15.102:0/36655227 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4560158db0 con 0x7f45600c4fa0 2024-03-18T16:29:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.035+0000 7f455e800640 1 --2- 172.21.15.102:0/36655227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c5430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4558009e50 tx=0x7f455802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=679d99769917b248 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:40.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.038+0000 7f455ca00640 1 -- 172.21.15.102:0/36655227 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455802ca70 con 0x7f45600c4fa0 2024-03-18T16:29:40.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.038+0000 7f455ca00640 1 -- 172.21.15.102:0/36655227 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f455802cc10 con 0x7f45600c4fa0 2024-03-18T16:29:40.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.038+0000 7f455ca00640 1 -- 172.21.15.102:0/36655227 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45580343f0 con 0x7f45600c4fa0 2024-03-18T16:29:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.039+0000 7f455f200640 1 -- 172.21.15.102:0/36655227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 msgr2=0x7f45600c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.039+0000 7f455f200640 1 --2- 172.21.15.102:0/36655227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c5430 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4558009e50 tx=0x7f455802bed0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.039+0000 7f455f200640 1 -- 172.21.15.102:0/36655227 shutdown_connections 2024-03-18T16:29:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.039+0000 7f455f200640 1 --2- 172.21.15.102:0/36655227 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c5430 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.039+0000 7f455f200640 1 --2- 172.21.15.102:0/36655227 >> v2:172.21.15.102:3300/0 conn(0x7f4560141650 0x7f4560158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.039+0000 7f455f200640 1 -- 172.21.15.102:0/36655227 >> 172.21.15.102:0/36655227 conn(0x7f45600cb890 msgr2=0x7f45600cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.039+0000 7f455f200640 1 -- 172.21.15.102:0/36655227 shutdown_connections 2024-03-18T16:29:40.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.039+0000 7f455f200640 1 -- 172.21.15.102:0/36655227 wait complete. 2024-03-18T16:29:40.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.041+0000 7f455f200640 1 Processor -- start 2024-03-18T16:29:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.041+0000 7f455f200640 1 -- start start 2024-03-18T16:29:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.042+0000 7f455f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.042+0000 7f455f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4560141650 0x7f45600c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.042+0000 7f455f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f456015c3a0 0x7f4560154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.042+0000 7f455f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f45600c66a0 con 0x7f4560141650 2024-03-18T16:29:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.042+0000 7f455f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45600c63c0 con 0x7f456015c3a0 2024-03-18T16:29:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.042+0000 7f455f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f456015c7c0 con 0x7f45600c4fa0 2024-03-18T16:29:40.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.043+0000 7f455de00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:40.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.043+0000 7f455d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4560141650 0x7f45600c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:40.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.043+0000 7f455e800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f456015c3a0 0x7f4560154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:40.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.043+0000 7f455de00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:55062/0 (socket says 172.21.15.102:55062) 2024-03-18T16:29:40.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.043+0000 7f455de00640 1 -- 172.21.15.102:0/3832394807 learned_addr learned my addr 172.21.15.102:0/3832394807 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:40.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.047+0000 7f455de00640 1 -- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f456015c3a0 msgr2=0x7f4560154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:40.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.047+0000 7f455de00640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f456015c3a0 0x7f4560154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.047+0000 7f455de00640 1 -- 172.21.15.102:0/3832394807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4560141650 msgr2=0x7f45600c4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:40.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.047+0000 7f455de00640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4560141650 0x7f45600c4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.047+0000 7f455de00640 1 -- 172.21.15.102:0/3832394807 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f456015b490 con 0x7f45600c4fa0 2024-03-18T16:29:40.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.050+0000 7f455de00640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c3e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f455000e9e0 tx=0x7f455000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:40.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.053+0000 7f4556000640 1 -- 172.21.15.102:0/3832394807 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550017070 con 0x7f45600c4fa0 2024-03-18T16:29:40.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.053+0000 7f4556000640 1 -- 172.21.15.102:0/3832394807 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f455000c9d0 con 0x7f45600c4fa0 2024-03-18T16:29:40.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.053+0000 7f4556000640 1 -- 172.21.15.102:0/3832394807 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455000cdc0 con 0x7f45600c4fa0 2024-03-18T16:29:40.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.053+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f456015b770 con 0x7f45600c4fa0 2024-03-18T16:29:40.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.054+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f456015bd30 con 0x7f45600c4fa0 2024-03-18T16:29:40.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.055+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4560074fd0 con 0x7f45600c4fa0 2024-03-18T16:29:40.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.075+0000 7f4556000640 1 -- 172.21.15.102:0/3832394807 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f455001a070 con 0x7f45600c4fa0 2024-03-18T16:29:40.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.077+0000 7f4556000640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f454006bc50 0x7f454006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:40.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.077+0000 7f455d400640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f454006bc50 0x7f454006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:40.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.078+0000 7f455d400640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f454006bc50 0x7f454006e110 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4560156020 tx=0x7f454c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:40.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.309+0000 7f4556000640 1 -- 172.21.15.102:0/3832394807 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f4550057660 con 0x7f45600c4fa0 2024-03-18T16:29:40.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.320+0000 7f4556000640 1 -- 172.21.15.102:0/3832394807 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f45500595c0 con 0x7f45600c4fa0 2024-03-18T16:29:40.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.555+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45600b7f00 con 0x7f45600c4fa0 2024-03-18T16:29:40.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.568+0000 7f4556000640 1 -- 172.21.15.102:0/3832394807 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f455005d660 con 0x7f45600c4fa0 2024-03-18T16:29:40.570 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:40.570 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:40.570 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f454006bc50 msgr2=0x7f454006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f454006bc50 0x7f454006e110 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4560156020 tx=0x7f454c009040 comp rx=0 tx=0).stop 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 msgr2=0x7f45600c3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c3e10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f455000e9e0 tx=0x7f455000eea0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 shutdown_connections 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f454006bc50 0x7f454006e110 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f456015c3a0 0x7f4560154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4560141650 0x7f45600c4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 --2- 172.21.15.102:0/3832394807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f45600c4fa0 0x7f45600c3e10 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f455000e9e0 tx=0x7f455000eea0 comp rx=0 tx=0).stop 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.570+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 >> 172.21.15.102:0/3832394807 conn(0x7f45600cb890 msgr2=0x7f45600c83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.571+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 shutdown_connections 2024-03-18T16:29:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:40.571+0000 7f455f200640 1 -- 172.21.15.102:0/3832394807 wait complete. 2024-03-18T16:29:41.192 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:41.192 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:42.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.361+0000 7fb525a00640 1 Processor -- start 2024-03-18T16:29:42.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.361+0000 7fb525a00640 1 -- start start 2024-03-18T16:29:42.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.362+0000 7fb525a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb5200cfe70 0x7fb52015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:42.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.362+0000 7fb525a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5200cabd0 con 0x7fb5200cc2f0 2024-03-18T16:29:42.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.363+0000 7fb525a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5200caea0 con 0x7fb5200d0250 2024-03-18T16:29:42.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.363+0000 7fb525a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb5200cb180 con 0x7fb5200cfe70 2024-03-18T16:29:42.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.363+0000 7fb51fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb5200cfe70 0x7fb52015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:42.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.364+0000 7fb51f400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb5200d0250 0x7fb5200c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:51526/0 (socket says 172.21.15.102:51526) 2024-03-18T16:29:42.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.364+0000 7fb51f400640 1 -- 172.21.15.102:0/800173768 learned_addr learned my addr 172.21.15.102:0/800173768 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.367+0000 7fb51ea00640 1 -- 172.21.15.102:0/800173768 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 47314191 0 0) 0x7fb5200caea0 con 0x7fb5200d0250 2024-03-18T16:29:42.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.367+0000 7fb51ea00640 1 -- 172.21.15.102:0/800173768 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb504003520 con 0x7fb5200d0250 2024-03-18T16:29:42.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.367+0000 7fb51fe00640 1 -- 172.21.15.102:0/800173768 >> v1:172.21.15.154:6789/0 conn(0x7fb5200d0250 legacy=0x7fb5200c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:42.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.367+0000 7fb51fe00640 1 -- 172.21.15.102:0/800173768 >> v1:172.21.15.102:6789/0 conn(0x7fb5200cc2f0 legacy=0x7fb5200cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:42.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.367+0000 7fb51fe00640 1 -- 172.21.15.102:0/800173768 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5200cb460 con 0x7fb5200cfe70 2024-03-18T16:29:42.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.371+0000 7fb51fe00640 1 --2- 172.21.15.102:0/800173768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5200cfe70 0x7fb52015ae40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb5100097a0 tx=0x7fb51002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57012ad6bf789aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:42.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.376+0000 7fb51ea00640 1 -- 172.21.15.102:0/800173768 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510034070 con 0x7fb5200cfe70 2024-03-18T16:29:42.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.377+0000 7fb51ea00640 1 -- 172.21.15.102:0/800173768 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb510030610 con 0x7fb5200cfe70 2024-03-18T16:29:42.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.377+0000 7fb51ea00640 1 -- 172.21.15.102:0/800173768 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5100309e0 con 0x7fb5200cfe70 2024-03-18T16:29:42.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.377+0000 7fb525a00640 1 -- 172.21.15.102:0/800173768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5200cfe70 msgr2=0x7fb52015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:42.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.377+0000 7fb525a00640 1 --2- 172.21.15.102:0/800173768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5200cfe70 0x7fb52015ae40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb5100097a0 tx=0x7fb51002b960 comp rx=0 tx=0).stop 2024-03-18T16:29:42.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.377+0000 7fb525a00640 1 -- 172.21.15.102:0/800173768 shutdown_connections 2024-03-18T16:29:42.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.377+0000 7fb525a00640 1 --2- 172.21.15.102:0/800173768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5200cfe70 0x7fb52015ae40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:42.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.377+0000 7fb525a00640 1 -- 172.21.15.102:0/800173768 >> 172.21.15.102:0/800173768 conn(0x7fb52008c3e0 msgr2=0x7fb52006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:42.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.378+0000 7fb525a00640 1 -- 172.21.15.102:0/800173768 shutdown_connections 2024-03-18T16:29:42.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.378+0000 7fb525a00640 1 -- 172.21.15.102:0/800173768 wait complete. 2024-03-18T16:29:42.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.379+0000 7fb525a00640 1 Processor -- start 2024-03-18T16:29:42.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.380+0000 7fb525a00640 1 -- start start 2024-03-18T16:29:42.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.381+0000 7fb525a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5200cc2f0 0x7fb520150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:42.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.381+0000 7fb525a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5200cfe70 0x7fb520158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:42.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.381+0000 7fb525a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5200d0250 0x7fb520159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:42.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.381+0000 7fb525a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb5200cbd00 con 0x7fb5200cc2f0 2024-03-18T16:29:42.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.381+0000 7fb525a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb5200cba20 con 0x7fb5200cfe70 2024-03-18T16:29:42.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.381+0000 7fb525a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb5200c8330 con 0x7fb5200d0250 2024-03-18T16:29:42.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.382+0000 7fb51f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5200cfe70 0x7fb520158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:42.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.382+0000 7fb51fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5200cc2f0 0x7fb520150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:42.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.382+0000 7fb525000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5200d0250 0x7fb520159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:42.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.382+0000 7fb51f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5200cfe70 0x7fb520158b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36104/0 (socket says 172.21.15.102:36104) 2024-03-18T16:29:42.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.382+0000 7fb51f400640 1 -- 172.21.15.102:0/436324139 learned_addr learned my addr 172.21.15.102:0/436324139 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:42.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.386+0000 7fb525000640 1 -- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5200cfe70 msgr2=0x7fb520158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:42.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.386+0000 7fb525000640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5200cfe70 0x7fb520158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:42.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.386+0000 7fb525000640 1 -- 172.21.15.102:0/436324139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5200cc2f0 msgr2=0x7fb520150ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:42.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.386+0000 7fb525000640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5200cc2f0 0x7fb520150ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:42.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.386+0000 7fb525000640 1 -- 172.21.15.102:0/436324139 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5201580a0 con 0x7fb5200d0250 2024-03-18T16:29:42.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.388+0000 7fb525000640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5200d0250 0x7fb520159040 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb51800c780 tx=0x7fb51800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:42.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.390+0000 7fb517e00640 1 -- 172.21.15.102:0/436324139 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518017070 con 0x7fb5200d0250 2024-03-18T16:29:42.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.391+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5201f0810 con 0x7fb5200d0250 2024-03-18T16:29:42.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.391+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5201f0af0 con 0x7fb5200d0250 2024-03-18T16:29:42.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.392+0000 7fb517e00640 1 -- 172.21.15.102:0/436324139 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb518003b70 con 0x7fb5200d0250 2024-03-18T16:29:42.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.392+0000 7fb517e00640 1 -- 172.21.15.102:0/436324139 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518021060 con 0x7fb5200d0250 2024-03-18T16:29:42.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.393+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f8005290 con 0x7fb5200d0250 2024-03-18T16:29:42.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.414+0000 7fb517e00640 1 -- 172.21.15.102:0/436324139 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb51801a070 con 0x7fb5200d0250 2024-03-18T16:29:42.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.415+0000 7fb517e00640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb50006bc00 0x7fb50006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:42.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.415+0000 7fb51fe00640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb50006bc00 0x7fb50006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:42.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.416+0000 7fb51fe00640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb50006bc00 0x7fb50006e0c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb510036000 tx=0x7fb51003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:42.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.659+0000 7fb517e00640 1 -- 172.21.15.102:0/436324139 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb518056a20 con 0x7fb5200d0250 2024-03-18T16:29:42.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.670+0000 7fb517e00640 1 -- 172.21.15.102:0/436324139 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb518058980 con 0x7fb5200d0250 2024-03-18T16:29:42.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.925+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4f8005580 con 0x7fb5200d0250 2024-03-18T16:29:42.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.940+0000 7fb517e00640 1 -- 172.21.15.102:0/436324139 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb51805ca20 con 0x7fb5200d0250 2024-03-18T16:29:42.942 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:42.943 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:42.943 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:42.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb50006bc00 msgr2=0x7fb50006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:42.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb50006bc00 0x7fb50006e0c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb510036000 tx=0x7fb51003b040 comp rx=0 tx=0).stop 2024-03-18T16:29:42.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5200d0250 msgr2=0x7fb520159040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:42.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5200d0250 0x7fb520159040 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb51800c780 tx=0x7fb51800cc40 comp rx=0 tx=0).stop 2024-03-18T16:29:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 shutdown_connections 2024-03-18T16:29:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb50006bc00 0x7fb50006e0c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5200d0250 0x7fb520159040 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5200cfe70 0x7fb520158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 --2- 172.21.15.102:0/436324139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5200cc2f0 0x7fb520150ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.944+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 >> 172.21.15.102:0/436324139 conn(0x7fb52008c3e0 msgr2=0x7fb520159f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.945+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 shutdown_connections 2024-03-18T16:29:42.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:42.945+0000 7fb525a00640 1 -- 172.21.15.102:0/436324139 wait complete. 2024-03-18T16:29:43.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:43.576 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:44.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:44.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.749+0000 7fea83200640 1 Processor -- start 2024-03-18T16:29:44.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.749+0000 7fea83200640 1 -- start start 2024-03-18T16:29:44.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.750+0000 7fea83200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c0c3890 0x7fea7c0c8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:44.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.750+0000 7fea83200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fea7c0c3cd0 0x7fea7c0c9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:44.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.750+0000 7fea83200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea7c15a170 con 0x7fea7c154520 2024-03-18T16:29:44.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.750+0000 7fea83200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fea7c154900 con 0x7fea7c0c3cd0 2024-03-18T16:29:44.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.750+0000 7fea83200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fea7c0c4160 con 0x7fea7c0c3890 2024-03-18T16:29:44.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.751+0000 7fea81e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fea7c154520 0x7fea7c0c9eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43900/0 (socket says 172.21.15.102:43900) 2024-03-18T16:29:44.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.751+0000 7fea81e00640 1 -- 172.21.15.102:0/1733405494 learned_addr learned my addr 172.21.15.102:0/1733405494 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:44.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.751+0000 7fea81400640 1 --2- 172.21.15.102:0/1733405494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c0c3890 0x7fea7c0c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:44.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.754+0000 7fea81400640 1 -- 172.21.15.102:0/1733405494 >> v2:172.21.15.102:3300/0 conn(0x7fea7c0c3cd0 msgr2=0x7fea7c0c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:44.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.754+0000 7fea81400640 1 --2- 172.21.15.102:0/1733405494 >> v2:172.21.15.102:3300/0 conn(0x7fea7c0c3cd0 0x7fea7c0c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:44.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.754+0000 7fea81400640 1 -- 172.21.15.102:0/1733405494 >> v1:172.21.15.102:6789/0 conn(0x7fea7c154520 legacy=0x7fea7c0c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:44.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.754+0000 7fea81400640 1 -- 172.21.15.102:0/1733405494 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7c0c4440 con 0x7fea7c0c3890 2024-03-18T16:29:44.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.756+0000 7fea81400640 1 --2- 172.21.15.102:0/1733405494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c0c3890 0x7fea7c0c8920 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0097a0 tx=0x7fea6c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53aaeceda6a76192 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:44.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.760+0000 7fea80a00640 1 -- 172.21.15.102:0/1733405494 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c035020 con 0x7fea7c0c3890 2024-03-18T16:29:44.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.760+0000 7fea80a00640 1 -- 172.21.15.102:0/1733405494 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea6c031810 con 0x7fea7c0c3890 2024-03-18T16:29:44.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.760+0000 7fea80a00640 1 -- 172.21.15.102:0/1733405494 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c031be0 con 0x7fea7c0c3890 2024-03-18T16:29:44.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.760+0000 7fea83200640 1 -- 172.21.15.102:0/1733405494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c0c3890 msgr2=0x7fea7c0c8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:44.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.760+0000 7fea83200640 1 --2- 172.21.15.102:0/1733405494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c0c3890 0x7fea7c0c8920 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0097a0 tx=0x7fea6c02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:29:44.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.760+0000 7fea83200640 1 -- 172.21.15.102:0/1733405494 shutdown_connections 2024-03-18T16:29:44.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.760+0000 7fea83200640 1 --2- 172.21.15.102:0/1733405494 >> v2:172.21.15.102:3300/0 conn(0x7fea7c0c3cd0 0x7fea7c0c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:44.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.760+0000 7fea83200640 1 --2- 172.21.15.102:0/1733405494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c0c3890 0x7fea7c0c8920 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:44.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.761+0000 7fea83200640 1 -- 172.21.15.102:0/1733405494 >> 172.21.15.102:0/1733405494 conn(0x7fea7c066f80 msgr2=0x7fea7c067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:44.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.761+0000 7fea83200640 1 -- 172.21.15.102:0/1733405494 shutdown_connections 2024-03-18T16:29:44.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.761+0000 7fea83200640 1 -- 172.21.15.102:0/1733405494 wait complete. 2024-03-18T16:29:44.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.762+0000 7fea83200640 1 Processor -- start 2024-03-18T16:29:44.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.762+0000 7fea83200640 1 -- start start 2024-03-18T16:29:44.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.763+0000 7fea83200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c0c3890 0x7fea7c1e5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:44.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.763+0000 7fea83200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c154520 0x7fea7c14bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:44.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.763+0000 7fea83200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea7c14cac0 0x7fea7c1595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:44.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.763+0000 7fea83200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c15a9f0 con 0x7fea7c0c3890 2024-03-18T16:29:44.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.763+0000 7fea83200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea7c15a710 con 0x7fea7c14cac0 2024-03-18T16:29:44.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.763+0000 7fea83200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fea7c0c8e60 con 0x7fea7c154520 2024-03-18T16:29:44.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.764+0000 7fea81400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c154520 0x7fea7c14bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:44.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.764+0000 7fea81e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c0c3890 0x7fea7c1e5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:44.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.764+0000 7fea82800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea7c14cac0 0x7fea7c1595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:44.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.764+0000 7fea81400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c154520 0x7fea7c14bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57294/0 (socket says 172.21.15.102:57294) 2024-03-18T16:29:44.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.764+0000 7fea81400640 1 -- 172.21.15.102:0/547348066 learned_addr learned my addr 172.21.15.102:0/547348066 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:44.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.769+0000 7fea81400640 1 -- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea7c14cac0 msgr2=0x7fea7c1595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:44.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.769+0000 7fea81400640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea7c14cac0 0x7fea7c1595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:44.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.769+0000 7fea81400640 1 -- 172.21.15.102:0/547348066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c0c3890 msgr2=0x7fea7c1e5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:44.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.769+0000 7fea81400640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c0c3890 0x7fea7c1e5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:44.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.769+0000 7fea81400640 1 -- 172.21.15.102:0/547348066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea7c14c1d0 con 0x7fea7c154520 2024-03-18T16:29:44.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.772+0000 7fea81400640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c154520 0x7fea7c14bc90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0098c0 tx=0x7fea6c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:44.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.774+0000 7fea76000640 1 -- 172.21.15.102:0/547348066 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c0453f0 con 0x7fea7c154520 2024-03-18T16:29:44.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.775+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7c1e5c40 con 0x7fea7c154520 2024-03-18T16:29:44.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.775+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7c1f8bc0 con 0x7fea7c154520 2024-03-18T16:29:44.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.776+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea7c1540d0 con 0x7fea7c154520 2024-03-18T16:29:44.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.776+0000 7fea76000640 1 -- 172.21.15.102:0/547348066 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea6c0316d0 con 0x7fea7c154520 2024-03-18T16:29:44.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:44.776+0000 7fea76000640 1 -- 172.21.15.102:0/547348066 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c053060 con 0x7fea7c154520 2024-03-18T16:29:45.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.040+0000 7fea76000640 1 -- 172.21.15.102:0/547348066 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fea6c038020 con 0x7fea7c154520 2024-03-18T16:29:45.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.043+0000 7fea76000640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea6006bbd0 0x7fea6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:45.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.043+0000 7fea81e00640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea6006bbd0 0x7fea6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:45.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.043+0000 7fea76000640 1 -- 172.21.15.102:0/547348066 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fea6c079870 con 0x7fea7c154520 2024-03-18T16:29:45.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.044+0000 7fea81e00640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea6006bbd0 0x7fea6006e090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fea70007fd0 tx=0x7fea70007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:45.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.054+0000 7fea76000640 1 -- 172.21.15.102:0/547348066 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fea7c1540d0 con 0x7fea7c154520 2024-03-18T16:29:45.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.296+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea7c1584f0 con 0x7fea7c154520 2024-03-18T16:29:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.329+0000 7fea76000640 1 -- 172.21.15.102:0/547348066 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fea7c1584f0 con 0x7fea7c154520 2024-03-18T16:29:45.331 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:45.331 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:45.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea6006bbd0 msgr2=0x7fea6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:45.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea6006bbd0 0x7fea6006e090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fea70007fd0 tx=0x7fea70007e80 comp rx=0 tx=0).stop 2024-03-18T16:29:45.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c154520 msgr2=0x7fea7c14bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:45.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c154520 0x7fea7c14bc90 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0098c0 tx=0x7fea6c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:29:45.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 shutdown_connections 2024-03-18T16:29:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fea6006bbd0 0x7fea6006e090 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea7c14cac0 0x7fea7c1595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fea7c154520 0x7fea7c14bc90 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 --2- 172.21.15.102:0/547348066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea7c0c3890 0x7fea7c1e5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 >> 172.21.15.102:0/547348066 conn(0x7fea7c066f80 msgr2=0x7fea7c06d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 shutdown_connections 2024-03-18T16:29:45.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:45.332+0000 7fea83200640 1 -- 172.21.15.102:0/547348066 wait complete. 2024-03-18T16:29:45.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:45.960 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:46.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:47.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.139+0000 7fa1d5c00640 1 Processor -- start 2024-03-18T16:29:47.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.139+0000 7fa1d5c00640 1 -- start start 2024-03-18T16:29:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.140+0000 7fa1d5c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa1d0141650 0x7fa1d0158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.140+0000 7fa1d5c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1d00c72d0 con 0x7fa1d015c3a0 2024-03-18T16:29:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.140+0000 7fa1d5c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1d0158ad0 con 0x7fa1d00c4fa0 2024-03-18T16:29:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.140+0000 7fa1d5c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa1d0159370 con 0x7fa1d0141650 2024-03-18T16:29:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.141+0000 7fa1d5200640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa1d00c4fa0 0x7fa1d0162cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:51534/0 (socket says 172.21.15.102:51534) 2024-03-18T16:29:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.141+0000 7fa1d5200640 1 -- 172.21.15.102:0/2475514798 learned_addr learned my addr 172.21.15.102:0/2475514798 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:47.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.142+0000 7fa1cfe00640 1 --2- 172.21.15.102:0/2475514798 >> v2:172.21.15.102:3300/0 conn(0x7fa1d0141650 0x7fa1d0158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.144+0000 7fa1cea00640 1 -- 172.21.15.102:0/2475514798 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2653973925 0 0) 0x7fa1d0158ad0 con 0x7fa1d00c4fa0 2024-03-18T16:29:47.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.144+0000 7fa1cea00640 1 -- 172.21.15.102:0/2475514798 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1b4003520 con 0x7fa1d00c4fa0 2024-03-18T16:29:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.146+0000 7fa1cfe00640 1 -- 172.21.15.102:0/2475514798 >> v1:172.21.15.154:6789/0 conn(0x7fa1d00c4fa0 legacy=0x7fa1d0162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.146+0000 7fa1cfe00640 1 -- 172.21.15.102:0/2475514798 >> v1:172.21.15.102:6789/0 conn(0x7fa1d015c3a0 legacy=0x7fa1d00c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:47.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.146+0000 7fa1cfe00640 1 -- 172.21.15.102:0/2475514798 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d0158db0 con 0x7fa1d0141650 2024-03-18T16:29:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.152+0000 7fa1cfe00640 1 --2- 172.21.15.102:0/2475514798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d0141650 0x7fa1d0158420 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00097a0 tx=0x7fa1c002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9df7cb4576d14c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:47.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.157+0000 7fa1cea00640 1 -- 172.21.15.102:0/2475514798 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0034070 con 0x7fa1d0141650 2024-03-18T16:29:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.157+0000 7fa1cea00640 1 -- 172.21.15.102:0/2475514798 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1c0030610 con 0x7fa1d0141650 2024-03-18T16:29:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.157+0000 7fa1cea00640 1 -- 172.21.15.102:0/2475514798 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c00309e0 con 0x7fa1d0141650 2024-03-18T16:29:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.158+0000 7fa1d5c00640 1 -- 172.21.15.102:0/2475514798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d0141650 msgr2=0x7fa1d0158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.158+0000 7fa1d5c00640 1 --2- 172.21.15.102:0/2475514798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d0141650 0x7fa1d0158420 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00097a0 tx=0x7fa1c002b960 comp rx=0 tx=0).stop 2024-03-18T16:29:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.158+0000 7fa1d5c00640 1 -- 172.21.15.102:0/2475514798 shutdown_connections 2024-03-18T16:29:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.158+0000 7fa1d5c00640 1 --2- 172.21.15.102:0/2475514798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d0141650 0x7fa1d0158420 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.158+0000 7fa1d5c00640 1 -- 172.21.15.102:0/2475514798 >> 172.21.15.102:0/2475514798 conn(0x7fa1d00cb890 msgr2=0x7fa1d00cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.158+0000 7fa1d5c00640 1 -- 172.21.15.102:0/2475514798 shutdown_connections 2024-03-18T16:29:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.158+0000 7fa1d5c00640 1 -- 172.21.15.102:0/2475514798 wait complete. 2024-03-18T16:29:47.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.161+0000 7fa1d5c00640 1 Processor -- start 2024-03-18T16:29:47.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.161+0000 7fa1d5c00640 1 -- start start 2024-03-18T16:29:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.162+0000 7fa1d5c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d00c4fa0 0x7fa1d01e3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.162+0000 7fa1d5c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1d0141650 0x7fa1d01f3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.162+0000 7fa1d5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d015c3a0 0x7fa1d013e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.162+0000 7fa1d5c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d00c66a0 con 0x7fa1d015c3a0 2024-03-18T16:29:47.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.162+0000 7fa1d5c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d00c63c0 con 0x7fa1d00c4fa0 2024-03-18T16:29:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.162+0000 7fa1d5c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa1d015c7c0 con 0x7fa1d0141650 2024-03-18T16:29:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.163+0000 7fa1d5200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d015c3a0 0x7fa1d013e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.163+0000 7fa1cf400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1d0141650 0x7fa1d01f3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.163+0000 7fa1cfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d00c4fa0 0x7fa1d01e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.163+0000 7fa1cf400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1d0141650 0x7fa1d01f3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57298/0 (socket says 172.21.15.102:57298) 2024-03-18T16:29:47.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.163+0000 7fa1cf400640 1 -- 172.21.15.102:0/3531886491 learned_addr learned my addr 172.21.15.102:0/3531886491 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:47.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.163+0000 7fa1d5200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d015c3a0 0x7fa1d013e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35566/0 (socket says 172.21.15.102:35566) 2024-03-18T16:29:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.167+0000 7fa1cf400640 1 -- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d00c4fa0 msgr2=0x7fa1d01e3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.167+0000 7fa1cf400640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d00c4fa0 0x7fa1d01e3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.167+0000 7fa1cf400640 1 -- 172.21.15.102:0/3531886491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d015c3a0 msgr2=0x7fa1d013e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.167+0000 7fa1cf400640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d015c3a0 0x7fa1d013e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:47.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.167+0000 7fa1cf400640 1 -- 172.21.15.102:0/3531886491 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d0153090 con 0x7fa1d0141650 2024-03-18T16:29:47.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.169+0000 7fa1cf400640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1d0141650 0x7fa1d01f3df0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00c3c0 tx=0x7fa1bc00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.172+0000 7fa1c7e00640 1 -- 172.21.15.102:0/3531886491 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc0052d0 con 0x7fa1d0141650 2024-03-18T16:29:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.173+0000 7fa1c7e00640 1 -- 172.21.15.102:0/3531886491 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1bc005490 con 0x7fa1d0141650 2024-03-18T16:29:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.173+0000 7fa1c7e00640 1 -- 172.21.15.102:0/3531886491 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc007260 con 0x7fa1d0141650 2024-03-18T16:29:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.173+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d0153370 con 0x7fa1d0141650 2024-03-18T16:29:47.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.173+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d0153930 con 0x7fa1d0141650 2024-03-18T16:29:47.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.174+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d0074fd0 con 0x7fa1d0141650 2024-03-18T16:29:47.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.194+0000 7fa1c7e00640 1 -- 172.21.15.102:0/3531886491 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa1bc007400 con 0x7fa1d0141650 2024-03-18T16:29:47.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.195+0000 7fa1c7e00640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1b006bc20 0x7fa1b006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:47.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.196+0000 7fa1cfe00640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1b006bc20 0x7fa1b006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:47.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.196+0000 7fa1cfe00640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1b006bc20 0x7fa1b006e0e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0036000 tx=0x7fa1c003b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:47.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.198+0000 7fa1c7e00640 1 -- 172.21.15.102:0/3531886491 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa1bc056890 con 0x7fa1d0141650 2024-03-18T16:29:47.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.451+0000 7fa1c7e00640 1 -- 172.21.15.102:0/3531886491 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa1bc0587f0 con 0x7fa1d0141650 2024-03-18T16:29:47.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.779+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1d00b7f00 con 0x7fa1d0141650 2024-03-18T16:29:47.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.822+0000 7fa1c7e00640 1 -- 172.21.15.102:0/3531886491 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa1bc05c890 con 0x7fa1d0141650 2024-03-18T16:29:47.824 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:47.824 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:47.824 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.825+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1b006bc20 msgr2=0x7fa1b006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1b006bc20 0x7fa1b006e0e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0036000 tx=0x7fa1c003b040 comp rx=0 tx=0).stop 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1d0141650 msgr2=0x7fa1d01f3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1d0141650 0x7fa1d01f3df0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00c3c0 tx=0x7fa1bc00c880 comp rx=0 tx=0).stop 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 shutdown_connections 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1b006bc20 0x7fa1b006e0e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1d015c3a0 0x7fa1d013e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa1d0141650 0x7fa1d01f3df0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 --2- 172.21.15.102:0/3531886491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1d00c4fa0 0x7fa1d01e3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:47.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 >> 172.21.15.102:0/3531886491 conn(0x7fa1d00cb890 msgr2=0x7fa1d0071b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:47.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 shutdown_connections 2024-03-18T16:29:47.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:47.826+0000 7fa1d5c00640 1 -- 172.21.15.102:0/3531886491 wait complete. 2024-03-18T16:29:48.449 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:48.449 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:49.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:49.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.621+0000 7f0a64c00640 1 Processor -- start 2024-03-18T16:29:49.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.621+0000 7f0a64c00640 1 -- start start 2024-03-18T16:29:49.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.622+0000 7f0a64c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a60141650 0x7f0a60158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:49.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.622+0000 7f0a64c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0a6015c3a0 0x7f0a600c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:49.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.622+0000 7f0a64c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a60158ad0 con 0x7f0a600c4fa0 2024-03-18T16:29:49.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.622+0000 7f0a64c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0a60159370 con 0x7f0a6015c3a0 2024-03-18T16:29:49.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.622+0000 7f0a64c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0a60158db0 con 0x7f0a60141650 2024-03-18T16:29:49.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.623+0000 7f0a5fe00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0a600c4fa0 0x7f0a6006f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:51538/0 (socket says 172.21.15.102:51538) 2024-03-18T16:29:49.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.623+0000 7f0a5fe00640 1 -- 172.21.15.102:0/4186122290 learned_addr learned my addr 172.21.15.102:0/4186122290 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:49.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.623+0000 7f0a5f400640 1 --2- 172.21.15.102:0/4186122290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a60141650 0x7f0a60158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:49.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.623+0000 7f0a5ea00640 1 --2- 172.21.15.102:0/4186122290 >> v2:172.21.15.102:3300/0 conn(0x7f0a6015c3a0 0x7f0a600c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:49.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.625+0000 7f0a5e000640 1 -- 172.21.15.102:0/4186122290 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 158889294 0 0) 0x7f0a60158ad0 con 0x7f0a600c4fa0 2024-03-18T16:29:49.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.625+0000 7f0a5e000640 1 -- 172.21.15.102:0/4186122290 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a44003520 con 0x7f0a600c4fa0 2024-03-18T16:29:49.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.626+0000 7f0a5f400640 1 -- 172.21.15.102:0/4186122290 >> v2:172.21.15.102:3300/0 conn(0x7f0a6015c3a0 msgr2=0x7f0a600c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:49.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.626+0000 7f0a5f400640 1 --2- 172.21.15.102:0/4186122290 >> v2:172.21.15.102:3300/0 conn(0x7f0a6015c3a0 0x7f0a600c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:49.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.626+0000 7f0a5f400640 1 -- 172.21.15.102:0/4186122290 >> v1:172.21.15.154:6789/0 conn(0x7f0a600c4fa0 legacy=0x7f0a6006f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:49.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.626+0000 7f0a5f400640 1 -- 172.21.15.102:0/4186122290 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a60159090 con 0x7f0a60141650 2024-03-18T16:29:49.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.629+0000 7f0a5f400640 1 --2- 172.21.15.102:0/4186122290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a60141650 0x7f0a60158420 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0a500097a0 tx=0x7f0a5002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86af96f2c9020f4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:49.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.633+0000 7f0a5e000640 1 -- 172.21.15.102:0/4186122290 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50035020 con 0x7f0a60141650 2024-03-18T16:29:49.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.633+0000 7f0a5e000640 1 -- 172.21.15.102:0/4186122290 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a50031830 con 0x7f0a60141650 2024-03-18T16:29:49.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.633+0000 7f0a5e000640 1 -- 172.21.15.102:0/4186122290 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a50031c00 con 0x7f0a60141650 2024-03-18T16:29:49.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.634+0000 7f0a64c00640 1 -- 172.21.15.102:0/4186122290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a60141650 msgr2=0x7f0a60158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:49.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.634+0000 7f0a64c00640 1 --2- 172.21.15.102:0/4186122290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a60141650 0x7f0a60158420 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0a500097a0 tx=0x7f0a5002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:29:49.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.634+0000 7f0a64c00640 1 -- 172.21.15.102:0/4186122290 shutdown_connections 2024-03-18T16:29:49.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.634+0000 7f0a64c00640 1 --2- 172.21.15.102:0/4186122290 >> v2:172.21.15.102:3300/0 conn(0x7f0a6015c3a0 0x7f0a600c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:49.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.634+0000 7f0a64c00640 1 --2- 172.21.15.102:0/4186122290 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a60141650 0x7f0a60158420 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:49.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.634+0000 7f0a64c00640 1 -- 172.21.15.102:0/4186122290 >> 172.21.15.102:0/4186122290 conn(0x7f0a600cb890 msgr2=0x7f0a600cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:49.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.634+0000 7f0a64c00640 1 -- 172.21.15.102:0/4186122290 shutdown_connections 2024-03-18T16:29:49.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.634+0000 7f0a64c00640 1 -- 172.21.15.102:0/4186122290 wait complete. 2024-03-18T16:29:49.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.636+0000 7f0a64c00640 1 Processor -- start 2024-03-18T16:29:49.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.636+0000 7f0a64c00640 1 -- start start 2024-03-18T16:29:49.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.637+0000 7f0a64c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a600c4fa0 0x7f0a600c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:49.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.637+0000 7f0a64c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a60141650 0x7f0a601ef4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:49.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.637+0000 7f0a64c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a6015c3a0 0x7f0a601ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:49.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.637+0000 7f0a64c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6015c7c0 con 0x7f0a6015c3a0 2024-03-18T16:29:49.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.637+0000 7f0a64c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a600c66a0 con 0x7f0a60141650 2024-03-18T16:29:49.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.637+0000 7f0a64c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0a60141a30 con 0x7f0a600c4fa0 2024-03-18T16:29:49.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.638+0000 7f0a5ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a60141650 0x7f0a601ef4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:49.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.638+0000 7f0a5f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a600c4fa0 0x7f0a600c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:49.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.638+0000 7f0a5fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a6015c3a0 0x7f0a601ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:49.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.638+0000 7f0a5ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a60141650 0x7f0a601ef4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36138/0 (socket says 172.21.15.102:36138) 2024-03-18T16:29:49.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.638+0000 7f0a5ea00640 1 -- 172.21.15.102:0/1370991667 learned_addr learned my addr 172.21.15.102:0/1370991667 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:49.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.641+0000 7f0a5ea00640 1 -- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a600c4fa0 msgr2=0x7f0a600c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:49.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.641+0000 7f0a5ea00640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a600c4fa0 0x7f0a600c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:49.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.641+0000 7f0a5ea00640 1 -- 172.21.15.102:0/1370991667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a6015c3a0 msgr2=0x7f0a601ef9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:49.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.641+0000 7f0a5ea00640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a6015c3a0 0x7f0a601ef9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:49.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.641+0000 7f0a5ea00640 1 -- 172.21.15.102:0/1370991667 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a60158460 con 0x7f0a60141650 2024-03-18T16:29:49.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.644+0000 7f0a5ea00640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a60141650 0x7f0a601ef4a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00e870 tx=0x7f0a4c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:49.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.648+0000 7f0a57400640 1 -- 172.21.15.102:0/1370991667 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c017070 con 0x7f0a60141650 2024-03-18T16:29:49.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.648+0000 7f0a57400640 1 -- 172.21.15.102:0/1370991667 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4c00c940 con 0x7f0a60141650 2024-03-18T16:29:49.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.648+0000 7f0a57400640 1 -- 172.21.15.102:0/1370991667 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c00cd30 con 0x7f0a60141650 2024-03-18T16:29:49.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.648+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a601578e0 con 0x7f0a60141650 2024-03-18T16:29:49.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.648+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a60157ea0 con 0x7f0a60141650 2024-03-18T16:29:49.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.650+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a600b7f00 con 0x7f0a60141650 2024-03-18T16:29:49.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.671+0000 7f0a57400640 1 -- 172.21.15.102:0/1370991667 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0a4c01a070 con 0x7f0a60141650 2024-03-18T16:29:49.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.672+0000 7f0a57400640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a4006bc20 0x7f0a4006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:49.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.673+0000 7f0a5f400640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a4006bc20 0x7f0a4006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:49.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.673+0000 7f0a57400640 1 -- 172.21.15.102:0/1370991667 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f0a4c057170 con 0x7f0a60141650 2024-03-18T16:29:49.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.674+0000 7f0a5f400640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a4006bc20 0x7f0a4006e0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0a50036000 tx=0x7f0a50008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:49.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:49.922+0000 7f0a57400640 1 -- 172.21.15.102:0/1370991667 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0a4c0590d0 con 0x7f0a60141650 2024-03-18T16:29:50.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.286+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a600734f0 con 0x7f0a60141650 2024-03-18T16:29:50.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.300+0000 7f0a57400640 1 -- 172.21.15.102:0/1370991667 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f0a4c05d170 con 0x7f0a60141650 2024-03-18T16:29:50.302 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:50.302 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:50.303 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:50.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.302+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a4006bc20 msgr2=0x7f0a4006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:50.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.302+0000 7f0a64c00640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a4006bc20 0x7f0a4006e0e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0a50036000 tx=0x7f0a50008030 comp rx=0 tx=0).stop 2024-03-18T16:29:50.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.302+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a60141650 msgr2=0x7f0a601ef4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:50.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.302+0000 7f0a64c00640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a60141650 0x7f0a601ef4a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00e870 tx=0x7f0a4c00ed30 comp rx=0 tx=0).stop 2024-03-18T16:29:50.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.303+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 shutdown_connections 2024-03-18T16:29:50.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.303+0000 7f0a64c00640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0a4006bc20 0x7f0a4006e0e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:50.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.303+0000 7f0a64c00640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0a6015c3a0 0x7f0a601ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:50.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.303+0000 7f0a64c00640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a60141650 0x7f0a601ef4a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:50.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.303+0000 7f0a64c00640 1 --2- 172.21.15.102:0/1370991667 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0a600c4fa0 0x7f0a600c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:50.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.303+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 >> 172.21.15.102:0/1370991667 conn(0x7f0a600cb890 msgr2=0x7f0a600726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:50.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.303+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 shutdown_connections 2024-03-18T16:29:50.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:50.303+0000 7f0a64c00640 1 -- 172.21.15.102:0/1370991667 wait complete. 2024-03-18T16:29:50.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:50.927 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:51.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:52.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.100+0000 7f8317000640 1 Processor -- start 2024-03-18T16:29:52.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.101+0000 7f8317000640 1 -- start start 2024-03-18T16:29:52.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.103+0000 7f8317000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f83100c7f30 0x7f8310157d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:52.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.103+0000 7f8317000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83100c2d90 0x7f831015b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:52.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.103+0000 7f8317000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8310073530 con 0x7f8310072810 2024-03-18T16:29:52.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.103+0000 7f8317000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f83100c5d80 con 0x7f83100c7f30 2024-03-18T16:29:52.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.103+0000 7f8317000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8310157900 con 0x7f83100c2d90 2024-03-18T16:29:52.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.104+0000 7f8315200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83100c2d90 0x7f831015b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:52.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.104+0000 7f8315c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f83100c7f30 0x7f8310157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:52.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.104+0000 7f8316600640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8310072810 0x7f831006f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:43454/0 (socket says 172.21.15.102:43454) 2024-03-18T16:29:52.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.104+0000 7f8316600640 1 -- 172.21.15.102:0/3402306635 learned_addr learned my addr 172.21.15.102:0/3402306635 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:52.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.107+0000 7f830fe00640 1 -- 172.21.15.102:0/3402306635 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152155586 0 0) 0x7f8310073530 con 0x7f8310072810 2024-03-18T16:29:52.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.107+0000 7f830fe00640 1 -- 172.21.15.102:0/3402306635 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82f8003520 con 0x7f8310072810 2024-03-18T16:29:52.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.107+0000 7f8315200640 1 -- 172.21.15.102:0/3402306635 >> v2:172.21.15.154:3300/0 conn(0x7f83100c7f30 msgr2=0x7f8310157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:52.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.107+0000 7f8315200640 1 --2- 172.21.15.102:0/3402306635 >> v2:172.21.15.154:3300/0 conn(0x7f83100c7f30 0x7f8310157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:52.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.107+0000 7f8315200640 1 -- 172.21.15.102:0/3402306635 >> v1:172.21.15.154:6789/0 conn(0x7f8310072810 legacy=0x7f831006f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:52.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.107+0000 7f8315200640 1 -- 172.21.15.102:0/3402306635 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83100c57e0 con 0x7f83100c2d90 2024-03-18T16:29:52.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.110+0000 7f8315200640 1 --2- 172.21.15.102:0/3402306635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83100c2d90 0x7f831015b0c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0097a0 tx=0x7f82fc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a8f328c5443938b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:52.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.114+0000 7f830fe00640 1 -- 172.21.15.102:0/3402306635 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc034070 con 0x7f83100c2d90 2024-03-18T16:29:52.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.114+0000 7f830fe00640 1 -- 172.21.15.102:0/3402306635 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82fc030610 con 0x7f83100c2d90 2024-03-18T16:29:52.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.114+0000 7f830fe00640 1 -- 172.21.15.102:0/3402306635 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc0309e0 con 0x7f83100c2d90 2024-03-18T16:29:52.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.115+0000 7f8317000640 1 -- 172.21.15.102:0/3402306635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83100c2d90 msgr2=0x7f831015b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:52.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.115+0000 7f8317000640 1 --2- 172.21.15.102:0/3402306635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83100c2d90 0x7f831015b0c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0097a0 tx=0x7f82fc02b960 comp rx=0 tx=0).stop 2024-03-18T16:29:52.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.115+0000 7f8317000640 1 -- 172.21.15.102:0/3402306635 shutdown_connections 2024-03-18T16:29:52.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.115+0000 7f8317000640 1 --2- 172.21.15.102:0/3402306635 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f83100c2d90 0x7f831015b0c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:52.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.115+0000 7f8317000640 1 --2- 172.21.15.102:0/3402306635 >> v2:172.21.15.154:3300/0 conn(0x7f83100c7f30 0x7f8310157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:52.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.115+0000 7f8317000640 1 -- 172.21.15.102:0/3402306635 >> 172.21.15.102:0/3402306635 conn(0x7f83100cb890 msgr2=0x7f83100cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:52.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.115+0000 7f8317000640 1 -- 172.21.15.102:0/3402306635 shutdown_connections 2024-03-18T16:29:52.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.115+0000 7f8317000640 1 -- 172.21.15.102:0/3402306635 wait complete. 2024-03-18T16:29:52.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.118+0000 7f8317000640 1 Processor -- start 2024-03-18T16:29:52.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.118+0000 7f8317000640 1 -- start start 2024-03-18T16:29:52.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.119+0000 7f8317000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8310072810 0x7f8310151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:52.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.119+0000 7f8317000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83100c2d90 0x7f831014d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:52.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.119+0000 7f8317000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83100c7f30 0x7f831014de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:52.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.119+0000 7f8317000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f83100c5aa0 con 0x7f83100c2d90 2024-03-18T16:29:52.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.120+0000 7f8317000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f831015b600 con 0x7f83100c7f30 2024-03-18T16:29:52.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.120+0000 7f8317000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f83101572b0 con 0x7f8310072810 2024-03-18T16:29:52.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.120+0000 7f8315200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83100c2d90 0x7f831014d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:52.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.120+0000 7f8316600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83100c7f30 0x7f831014de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:52.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.120+0000 7f8315c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8310072810 0x7f8310151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:52.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.121+0000 7f8316600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83100c7f30 0x7f831014de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57142/0 (socket says 172.21.15.102:57142) 2024-03-18T16:29:52.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.121+0000 7f8315c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8310072810 0x7f8310151c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52968/0 (socket says 172.21.15.102:52968) 2024-03-18T16:29:52.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.121+0000 7f8316600640 1 -- 172.21.15.102:0/1886904444 learned_addr learned my addr 172.21.15.102:0/1886904444 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:52.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.124+0000 7f8315c00640 1 -- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83100c7f30 msgr2=0x7f831014de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:52.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.124+0000 7f8315c00640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83100c7f30 0x7f831014de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:52.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.124+0000 7f8315c00640 1 -- 172.21.15.102:0/1886904444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83100c2d90 msgr2=0x7f831014d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:52.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.124+0000 7f8315c00640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83100c2d90 0x7f831014d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:52.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.124+0000 7f8315c00640 1 -- 172.21.15.102:0/1886904444 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83101f4d00 con 0x7f8310072810 2024-03-18T16:29:52.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.126+0000 7f8315c00640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8310072810 0x7f8310151c70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f830400e880 tx=0x7f830400ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:52.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.130+0000 7f830d600640 1 -- 172.21.15.102:0/1886904444 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8304017070 con 0x7f8310072810 2024-03-18T16:29:52.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.130+0000 7f830d600640 1 -- 172.21.15.102:0/1886904444 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f830400c940 con 0x7f8310072810 2024-03-18T16:29:52.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.130+0000 7f830d600640 1 -- 172.21.15.102:0/1886904444 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830400cd30 con 0x7f8310072810 2024-03-18T16:29:52.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.130+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83101f4180 con 0x7f8310072810 2024-03-18T16:29:52.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.131+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83101f4740 con 0x7f8310072810 2024-03-18T16:29:52.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.132+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83101513c0 con 0x7f8310072810 2024-03-18T16:29:52.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.400+0000 7f830d600640 1 -- 172.21.15.102:0/1886904444 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f830401a070 con 0x7f8310072810 2024-03-18T16:29:52.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.401+0000 7f830d600640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f006bbd0 0x7f82f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:52.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.401+0000 7f8315200640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f006bbd0 0x7f82f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:52.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.402+0000 7f8315200640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f006bbd0 0x7f82f006e090 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f82fc036000 tx=0x7f82fc03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:52.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.404+0000 7f830d600640 1 -- 172.21.15.102:0/1886904444 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8304057680 con 0x7f8310072810 2024-03-18T16:29:52.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.423+0000 7f830d600640 1 -- 172.21.15.102:0/1886904444 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f83101513c0 con 0x7f8310072810 2024-03-18T16:29:52.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.678+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8310074970 con 0x7f8310072810 2024-03-18T16:29:52.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.694+0000 7f830d600640 1 -- 172.21.15.102:0/1886904444 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8310074970 con 0x7f8310072810 2024-03-18T16:29:52.696 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:52.696 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:52.696 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f006bbd0 msgr2=0x7f82f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f006bbd0 0x7f82f006e090 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f82fc036000 tx=0x7f82fc03b040 comp rx=0 tx=0).stop 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8310072810 msgr2=0x7f8310151c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8310072810 0x7f8310151c70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f830400e880 tx=0x7f830400ed40 comp rx=0 tx=0).stop 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 shutdown_connections 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f82f006bbd0 0x7f82f006e090 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83100c7f30 0x7f831014de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83100c2d90 0x7f831014d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 --2- 172.21.15.102:0/1886904444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8310072810 0x7f8310151c70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:52.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 >> 172.21.15.102:0/1886904444 conn(0x7f83100cb890 msgr2=0x7f8310150490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:52.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 shutdown_connections 2024-03-18T16:29:52.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:52.697+0000 7f8317000640 1 -- 172.21.15.102:0/1886904444 wait complete. 2024-03-18T16:29:52.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:29:52.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:29:53.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:53.320 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:54.322 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.498+0000 7f2cf3000640 1 Processor -- start 2024-03-18T16:29:54.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.498+0000 7f2cf3000640 1 -- start start 2024-03-18T16:29:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.500+0000 7f2cf3000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec0c7f30 0x7f2cec157d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.500+0000 7f2cf3000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2cec072810 0x7f2cec072c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.500+0000 7f2cf3000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cec073530 con 0x7f2cec0c2d90 2024-03-18T16:29:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.500+0000 7f2cf3000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2cec0c5d80 con 0x7f2cec072810 2024-03-18T16:29:54.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.500+0000 7f2cf3000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2cec157900 con 0x7f2cec0c7f30 2024-03-18T16:29:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.501+0000 7f2cf1c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec0c7f30 0x7f2cec157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.501+0000 7f2cf2600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2cec072810 0x7f2cec072c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.502+0000 7f2cf1c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec0c7f30 0x7f2cec157d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52978/0 (socket says 172.21.15.102:52978) 2024-03-18T16:29:54.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.502+0000 7f2cf1c00640 1 -- 172.21.15.102:0/66147736 learned_addr learned my addr 172.21.15.102:0/66147736 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.504+0000 7f2cf1c00640 1 -- 172.21.15.102:0/66147736 >> v2:172.21.15.102:3300/0 conn(0x7f2cec072810 msgr2=0x7f2cec072c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.504+0000 7f2cf1c00640 1 --2- 172.21.15.102:0/66147736 >> v2:172.21.15.102:3300/0 conn(0x7f2cec072810 0x7f2cec072c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:54.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.504+0000 7f2cf1c00640 1 -- 172.21.15.102:0/66147736 >> v1:172.21.15.102:6789/0 conn(0x7f2cec0c2d90 legacy=0x7f2cec15b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:54.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.504+0000 7f2cf1c00640 1 -- 172.21.15.102:0/66147736 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cec0c57e0 con 0x7f2cec0c7f30 2024-03-18T16:29:54.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.506+0000 7f2cf1c00640 1 --2- 172.21.15.102:0/66147736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec0c7f30 0x7f2cec157d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009870 tx=0x7f2ce002ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48e58cca89696f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.510+0000 7f2cebe00640 1 -- 172.21.15.102:0/66147736 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce002f830 con 0x7f2cec0c7f30 2024-03-18T16:29:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.510+0000 7f2cebe00640 1 -- 172.21.15.102:0/66147736 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce002f9d0 con 0x7f2cec0c7f30 2024-03-18T16:29:54.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.510+0000 7f2cebe00640 1 -- 172.21.15.102:0/66147736 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce002fda0 con 0x7f2cec0c7f30 2024-03-18T16:29:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.511+0000 7f2cf3000640 1 -- 172.21.15.102:0/66147736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec0c7f30 msgr2=0x7f2cec157d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.511+0000 7f2cf3000640 1 --2- 172.21.15.102:0/66147736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec0c7f30 0x7f2cec157d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009870 tx=0x7f2ce002ba10 comp rx=0 tx=0).stop 2024-03-18T16:29:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.511+0000 7f2cf3000640 1 -- 172.21.15.102:0/66147736 shutdown_connections 2024-03-18T16:29:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.511+0000 7f2cf3000640 1 --2- 172.21.15.102:0/66147736 >> v2:172.21.15.102:3300/0 conn(0x7f2cec072810 0x7f2cec072c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.511+0000 7f2cf3000640 1 --2- 172.21.15.102:0/66147736 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec0c7f30 0x7f2cec157d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.511+0000 7f2cf3000640 1 -- 172.21.15.102:0/66147736 >> 172.21.15.102:0/66147736 conn(0x7f2cec0cb890 msgr2=0x7f2cec0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.511+0000 7f2cf3000640 1 -- 172.21.15.102:0/66147736 shutdown_connections 2024-03-18T16:29:54.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.511+0000 7f2cf3000640 1 -- 172.21.15.102:0/66147736 wait complete. 2024-03-18T16:29:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.514+0000 7f2cf3000640 1 Processor -- start 2024-03-18T16:29:54.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.514+0000 7f2cf3000640 1 -- start start 2024-03-18T16:29:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.515+0000 7f2cf3000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec072810 0x7f2cec140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.515+0000 7f2cf3000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cec0c2d90 0x7f2cec14e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.515+0000 7f2cf3000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cec0c7f30 0x7f2cec13fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.515+0000 7f2cf3000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec1572b0 con 0x7f2cec0c2d90 2024-03-18T16:29:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.515+0000 7f2cf3000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec0c5aa0 con 0x7f2cec0c7f30 2024-03-18T16:29:54.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.515+0000 7f2cf3000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2cec157590 con 0x7f2cec072810 2024-03-18T16:29:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.516+0000 7f2cf1c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec072810 0x7f2cec140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.516+0000 7f2cf1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cec0c2d90 0x7f2cec14e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.516+0000 7f2cf2600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cec0c7f30 0x7f2cec13fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:54.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.516+0000 7f2cf1c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec072810 0x7f2cec140cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52984/0 (socket says 172.21.15.102:52984) 2024-03-18T16:29:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.516+0000 7f2cf1c00640 1 -- 172.21.15.102:0/3328348000 learned_addr learned my addr 172.21.15.102:0/3328348000 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.520+0000 7f2cf1c00640 1 -- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cec0c7f30 msgr2=0x7f2cec13fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.520+0000 7f2cf1c00640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cec0c7f30 0x7f2cec13fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.520+0000 7f2cf1c00640 1 -- 172.21.15.102:0/3328348000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cec0c2d90 msgr2=0x7f2cec14e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.520+0000 7f2cf1c00640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cec0c2d90 0x7f2cec14e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:54.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.520+0000 7f2cf1c00640 1 -- 172.21.15.102:0/3328348000 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cec14f750 con 0x7f2cec072810 2024-03-18T16:29:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.522+0000 7f2cf1c00640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec072810 0x7f2cec140cd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2ce002ba50 tx=0x7f2ce0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.525+0000 7f2ce9600640 1 -- 172.21.15.102:0/3328348000 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0047070 con 0x7f2cec072810 2024-03-18T16:29:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.525+0000 7f2ce9600640 1 -- 172.21.15.102:0/3328348000 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce0037c40 con 0x7f2cec072810 2024-03-18T16:29:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.525+0000 7f2ce9600640 1 -- 172.21.15.102:0/3328348000 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0042480 con 0x7f2cec072810 2024-03-18T16:29:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.525+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cec155d30 con 0x7f2cec072810 2024-03-18T16:29:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.526+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cec1562d0 con 0x7f2cec072810 2024-03-18T16:29:54.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.527+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cec074fd0 con 0x7f2cec072810 2024-03-18T16:29:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.547+0000 7f2ce9600640 1 -- 172.21.15.102:0/3328348000 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2ce004a070 con 0x7f2cec072810 2024-03-18T16:29:54.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.548+0000 7f2ce9600640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2ccc06bc20 0x7f2ccc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:54.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.548+0000 7f2cf1200640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2ccc06bc20 0x7f2ccc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:54.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.549+0000 7f2cf1200640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2ccc06bc20 0x7f2ccc06e0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2cec070040 tx=0x7f2cd8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:54.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.783+0000 7f2ce9600640 1 -- 172.21.15.102:0/3328348000 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f2ce00795a0 con 0x7f2cec072810 2024-03-18T16:29:54.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:54.794+0000 7f2ce9600640 1 -- 172.21.15.102:0/3328348000 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2ce0051020 con 0x7f2cec072810 2024-03-18T16:29:55.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.038+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cec154b30 con 0x7f2cec072810 2024-03-18T16:29:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.168+0000 7f2ce9600640 1 -- 172.21.15.102:0/3328348000 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f2ce0033030 con 0x7f2cec072810 2024-03-18T16:29:55.170 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:55.170 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:55.170 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2ccc06bc20 msgr2=0x7f2ccc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2ccc06bc20 0x7f2ccc06e0e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2cec070040 tx=0x7f2cd8009040 comp rx=0 tx=0).stop 2024-03-18T16:29:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec072810 msgr2=0x7f2cec140cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec072810 0x7f2cec140cd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2ce002ba50 tx=0x7f2ce0004730 comp rx=0 tx=0).stop 2024-03-18T16:29:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 shutdown_connections 2024-03-18T16:29:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2ccc06bc20 0x7f2ccc06e0e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cec0c7f30 0x7f2cec13fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cec0c2d90 0x7f2cec14e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 --2- 172.21.15.102:0/3328348000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2cec072810 0x7f2cec140cd0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 >> 172.21.15.102:0/3328348000 conn(0x7f2cec0cb890 msgr2=0x7f2cec145690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 shutdown_connections 2024-03-18T16:29:55.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:55.171+0000 7f2cf3000640 1 -- 172.21.15.102:0/3328348000 wait complete. 2024-03-18T16:29:55.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:55.794 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:56.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:56.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.968+0000 7fd569800640 1 Processor -- start 2024-03-18T16:29:56.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.968+0000 7fd569800640 1 -- start start 2024-03-18T16:29:56.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.970+0000 7fd569800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415a6f0 0x7fd56415ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:56.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.970+0000 7fd569800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd56415baa0 0x7fd56415bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:56.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.970+0000 7fd569800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd56415c470 con 0x7fd56415a310 2024-03-18T16:29:56.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.970+0000 7fd569800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd5640cd410 con 0x7fd56415baa0 2024-03-18T16:29:56.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.970+0000 7fd569800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5640ca500 con 0x7fd56415a6f0 2024-03-18T16:29:56.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.971+0000 7fd563400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415a6f0 0x7fd56415ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:56.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.971+0000 7fd563400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415a6f0 0x7fd56415ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52990/0 (socket says 172.21.15.102:52990) 2024-03-18T16:29:56.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.971+0000 7fd563400640 1 -- 172.21.15.102:0/2042730978 learned_addr learned my addr 172.21.15.102:0/2042730978 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:56.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.975+0000 7fd563400640 1 -- 172.21.15.102:0/2042730978 >> v2:172.21.15.102:3300/0 conn(0x7fd56415baa0 msgr2=0x7fd56415bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:56.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.975+0000 7fd563400640 1 --2- 172.21.15.102:0/2042730978 >> v2:172.21.15.102:3300/0 conn(0x7fd56415baa0 0x7fd56415bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:56.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.975+0000 7fd563400640 1 -- 172.21.15.102:0/2042730978 >> v1:172.21.15.102:6789/0 conn(0x7fd56415a310 legacy=0x7fd56406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:56.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.975+0000 7fd563400640 1 -- 172.21.15.102:0/2042730978 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56415b050 con 0x7fd56415a6f0 2024-03-18T16:29:56.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.978+0000 7fd563400640 1 --2- 172.21.15.102:0/2042730978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415a6f0 0x7fd56415ab10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd550009870 tx=0x7fd55002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=690c8a67c877c04 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:56.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.981+0000 7fd562a00640 1 -- 172.21.15.102:0/2042730978 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd550035020 con 0x7fd56415a6f0 2024-03-18T16:29:56.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.981+0000 7fd562a00640 1 -- 172.21.15.102:0/2042730978 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd550031640 con 0x7fd56415a6f0 2024-03-18T16:29:56.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.981+0000 7fd562a00640 1 -- 172.21.15.102:0/2042730978 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55002d040 con 0x7fd56415a6f0 2024-03-18T16:29:56.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.982+0000 7fd569800640 1 -- 172.21.15.102:0/2042730978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415a6f0 msgr2=0x7fd56415ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:56.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.982+0000 7fd569800640 1 --2- 172.21.15.102:0/2042730978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415a6f0 0x7fd56415ab10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd550009870 tx=0x7fd55002b8f0 comp rx=0 tx=0).stop 2024-03-18T16:29:56.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.983+0000 7fd569800640 1 -- 172.21.15.102:0/2042730978 shutdown_connections 2024-03-18T16:29:56.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.983+0000 7fd569800640 1 --2- 172.21.15.102:0/2042730978 >> v2:172.21.15.102:3300/0 conn(0x7fd56415baa0 0x7fd56415bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:56.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.983+0000 7fd569800640 1 --2- 172.21.15.102:0/2042730978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415a6f0 0x7fd56415ab10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:56.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.983+0000 7fd569800640 1 -- 172.21.15.102:0/2042730978 >> 172.21.15.102:0/2042730978 conn(0x7fd56408c1a0 msgr2=0x7fd564066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:56.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.983+0000 7fd569800640 1 -- 172.21.15.102:0/2042730978 shutdown_connections 2024-03-18T16:29:56.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.983+0000 7fd569800640 1 -- 172.21.15.102:0/2042730978 wait complete. 2024-03-18T16:29:56.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.984+0000 7fd569800640 1 Processor -- start 2024-03-18T16:29:56.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.985+0000 7fd569800640 1 -- start start 2024-03-18T16:29:56.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.985+0000 7fd569800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd56415a310 0x7fd5641f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:56.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.985+0000 7fd569800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd56415a6f0 0x7fd564155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:56.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.985+0000 7fd569800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415baa0 0x7fd5640c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:56.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.985+0000 7fd569800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5640cce50 con 0x7fd56415a6f0 2024-03-18T16:29:56.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.985+0000 7fd569800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5640caa80 con 0x7fd56415a310 2024-03-18T16:29:56.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.985+0000 7fd569800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5640cd130 con 0x7fd56415baa0 2024-03-18T16:29:56.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.986+0000 7fd568e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415baa0 0x7fd5640c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:56.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.986+0000 7fd563e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd56415a310 0x7fd5641f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:56.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.987+0000 7fd563400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd56415a6f0 0x7fd564155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:56.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.987+0000 7fd568e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415baa0 0x7fd5640c94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53002/0 (socket says 172.21.15.102:53002) 2024-03-18T16:29:56.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.987+0000 7fd568e00640 1 -- 172.21.15.102:0/3173565910 learned_addr learned my addr 172.21.15.102:0/3173565910 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:56.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.990+0000 7fd568e00640 1 -- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd56415a310 msgr2=0x7fd5641f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:56.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.990+0000 7fd568e00640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd56415a310 0x7fd5641f0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:56.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.990+0000 7fd568e00640 1 -- 172.21.15.102:0/3173565910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd56415a6f0 msgr2=0x7fd564155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:56.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.990+0000 7fd568e00640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd56415a6f0 0x7fd564155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:56.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.990+0000 7fd568e00640 1 -- 172.21.15.102:0/3173565910 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5641f1140 con 0x7fd56415baa0 2024-03-18T16:29:56.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.992+0000 7fd568e00640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415baa0 0x7fd5640c94e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00b9a0 tx=0x7fd55c00be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:56.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.996+0000 7fd55be00640 1 -- 172.21.15.102:0/3173565910 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c019070 con 0x7fd56415baa0 2024-03-18T16:29:56.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.996+0000 7fd55be00640 1 -- 172.21.15.102:0/3173565910 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55c00cbc0 con 0x7fd56415baa0 2024-03-18T16:29:56.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.996+0000 7fd55be00640 1 -- 172.21.15.102:0/3173565910 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c014440 con 0x7fd56415baa0 2024-03-18T16:29:56.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.996+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5640c7b80 con 0x7fd56415baa0 2024-03-18T16:29:56.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.996+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd564159ed0 con 0x7fd56415baa0 2024-03-18T16:29:56.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:56.997+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd564074fd0 con 0x7fd56415baa0 2024-03-18T16:29:57.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.017+0000 7fd55be00640 1 -- 172.21.15.102:0/3173565910 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd55c01c070 con 0x7fd56415baa0 2024-03-18T16:29:57.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.018+0000 7fd55be00640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd54406bbd0 0x7fd54406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:57.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.019+0000 7fd563e00640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd54406bbd0 0x7fd54406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.019+0000 7fd563e00640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd54406bbd0 0x7fd54406e090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd554006fd0 tx=0x7fd554009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:57.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.260+0000 7fd55be00640 1 -- 172.21.15.102:0/3173565910 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd55c057120 con 0x7fd56415baa0 2024-03-18T16:29:57.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.279+0000 7fd55be00640 1 -- 172.21.15.102:0/3173565910 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd55c023020 con 0x7fd56415baa0 2024-03-18T16:29:57.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.574+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5640caed0 con 0x7fd56415baa0 2024-03-18T16:29:57.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.592+0000 7fd55be00640 1 -- 172.21.15.102:0/3173565910 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd55c059080 con 0x7fd56415baa0 2024-03-18T16:29:57.596 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:29:57.596 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:29:57.596 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:29:57.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd54406bbd0 msgr2=0x7fd54406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:57.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd54406bbd0 0x7fd54406e090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd554006fd0 tx=0x7fd554009060 comp rx=0 tx=0).stop 2024-03-18T16:29:57.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415baa0 msgr2=0x7fd5640c94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:57.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415baa0 0x7fd5640c94e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd55c00b9a0 tx=0x7fd55c00be60 comp rx=0 tx=0).stop 2024-03-18T16:29:57.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 shutdown_connections 2024-03-18T16:29:57.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd54406bbd0 0x7fd54406e090 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:57.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd56415baa0 0x7fd5640c94e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:57.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd56415a6f0 0x7fd564155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:57.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 --2- 172.21.15.102:0/3173565910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd56415a310 0x7fd5641f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:57.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.596+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 >> 172.21.15.102:0/3173565910 conn(0x7fd56408c1a0 msgr2=0x7fd56414f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:57.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.597+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 shutdown_connections 2024-03-18T16:29:57.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:57.597+0000 7fd569800640 1 -- 172.21.15.102:0/3173565910 wait complete. 2024-03-18T16:29:58.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:29:58.228 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:29:59.229 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:29:59.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.404+0000 7f8eee400640 1 Processor -- start 2024-03-18T16:29:59.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.404+0000 7f8eee400640 1 -- start start 2024-03-18T16:29:59.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.405+0000 7f8eee400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 0x7f8ee815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:59.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.405+0000 7f8eee400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ee80c4fa0 0x7f8ee80c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:59.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.405+0000 7f8eee400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ee80c72d0 con 0x7f8ee8141650 2024-03-18T16:29:59.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.405+0000 7f8eee400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8ee815c8e0 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.405+0000 7f8eee400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee8159370 con 0x7f8ee8158420 2024-03-18T16:29:59.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.406+0000 7f8ee7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 0x7f8ee815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:59.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.406+0000 7f8eeda00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ee80c4fa0 0x7f8ee80c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:59.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.406+0000 7f8eed000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8ee8141650 0x7f8ee80c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:43468/0 (socket says 172.21.15.102:43468) 2024-03-18T16:29:59.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.406+0000 7f8eed000640 1 -- 172.21.15.102:0/3355853283 learned_addr learned my addr 172.21.15.102:0/3355853283 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:29:59.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.410+0000 7f8ee7400640 1 -- 172.21.15.102:0/3355853283 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277484870 0 0) 0x7f8ee80c72d0 con 0x7f8ee8141650 2024-03-18T16:29:59.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.410+0000 7f8ee7400640 1 -- 172.21.15.102:0/3355853283 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ed0003520 con 0x7f8ee8141650 2024-03-18T16:29:59.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.410+0000 7f8ee7e00640 1 -- 172.21.15.102:0/3355853283 >> v2:172.21.15.154:3300/0 conn(0x7f8ee80c4fa0 msgr2=0x7f8ee80c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:29:59.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.410+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/3355853283 >> v2:172.21.15.154:3300/0 conn(0x7f8ee80c4fa0 0x7f8ee80c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:59.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.410+0000 7f8ee7e00640 1 -- 172.21.15.102:0/3355853283 >> v1:172.21.15.154:6789/0 conn(0x7f8ee8141650 legacy=0x7f8ee80c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:59.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.410+0000 7f8ee7e00640 1 -- 172.21.15.102:0/3355853283 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee8158840 con 0x7f8ee8158420 2024-03-18T16:29:59.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.413+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/3355853283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 0x7f8ee815c3a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40097a0 tx=0x7f8ed402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd44f2e7fe3dbe65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:59.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.418+0000 7f8ee7400640 1 -- 172.21.15.102:0/3355853283 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4035020 con 0x7f8ee8158420 2024-03-18T16:29:59.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.418+0000 7f8ee7400640 1 -- 172.21.15.102:0/3355853283 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed4031830 con 0x7f8ee8158420 2024-03-18T16:29:59.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.418+0000 7f8ee7400640 1 -- 172.21.15.102:0/3355853283 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4031c00 con 0x7f8ee8158420 2024-03-18T16:29:59.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.419+0000 7f8eee400640 1 -- 172.21.15.102:0/3355853283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 msgr2=0x7f8ee815c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:59.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.419+0000 7f8eee400640 1 --2- 172.21.15.102:0/3355853283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 0x7f8ee815c3a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8ed40097a0 tx=0x7f8ed402b8d0 comp rx=0 tx=0).stop 2024-03-18T16:29:59.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.419+0000 7f8eee400640 1 -- 172.21.15.102:0/3355853283 shutdown_connections 2024-03-18T16:29:59.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.419+0000 7f8eee400640 1 --2- 172.21.15.102:0/3355853283 >> v2:172.21.15.154:3300/0 conn(0x7f8ee80c4fa0 0x7f8ee80c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:59.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.419+0000 7f8eee400640 1 --2- 172.21.15.102:0/3355853283 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 0x7f8ee815c3a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:59.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.419+0000 7f8eee400640 1 -- 172.21.15.102:0/3355853283 >> 172.21.15.102:0/3355853283 conn(0x7f8ee80cb890 msgr2=0x7f8ee80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:29:59.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.419+0000 7f8eee400640 1 -- 172.21.15.102:0/3355853283 shutdown_connections 2024-03-18T16:29:59.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.419+0000 7f8eee400640 1 -- 172.21.15.102:0/3355853283 wait complete. 2024-03-18T16:29:59.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.421+0000 7f8eee400640 1 Processor -- start 2024-03-18T16:29:59.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.421+0000 7f8eee400640 1 -- start start 2024-03-18T16:29:59.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.422+0000 7f8eee400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee80c4fa0 0x7f8ee815b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:59.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.422+0000 7f8eee400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee8141650 0x7f8ee81f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:59.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.422+0000 7f8eee400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 0x7f8ee81394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:59.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.422+0000 7f8eee400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8159090 con 0x7f8ee8141650 2024-03-18T16:29:59.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.422+0000 7f8eee400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8158db0 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.422+0000 7f8eee400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee80c63c0 con 0x7f8ee8158420 2024-03-18T16:29:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.423+0000 7f8eed000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee80c4fa0 0x7f8ee815b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.423+0000 7f8eeda00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 0x7f8ee81394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.423+0000 7f8eed000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee80c4fa0 0x7f8ee815b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57184/0 (socket says 172.21.15.102:57184) 2024-03-18T16:29:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.423+0000 7f8eed000640 1 -- 172.21.15.102:0/4023652196 learned_addr learned my addr 172.21.15.102:0/4023652196 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:29:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.423+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee8141650 0x7f8ee81f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:59.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.426+0000 7f8eed000640 1 -- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 msgr2=0x7f8ee81394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:59.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.426+0000 7f8eed000640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 0x7f8ee81394d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:59.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.426+0000 7f8eed000640 1 -- 172.21.15.102:0/4023652196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee8141650 msgr2=0x7f8ee81f00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:29:59.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.426+0000 7f8eed000640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee8141650 0x7f8ee81f00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:29:59.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.426+0000 7f8eed000640 1 -- 172.21.15.102:0/4023652196 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee813cb60 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.428+0000 7f8eed000640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee80c4fa0 0x7f8ee815b180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00c490 tx=0x7f8edc00c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:59.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.432+0000 7f8ee4c00640 1 -- 172.21.15.102:0/4023652196 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8edc017070 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.432+0000 7f8ee4c00640 1 -- 172.21.15.102:0/4023652196 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8edc005ce0 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.432+0000 7f8ee4c00640 1 -- 172.21.15.102:0/4023652196 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8edc007290 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.432+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee813ce40 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.433+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee8139a10 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.434+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee8152360 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.691+0000 7f8ee4c00640 1 -- 172.21.15.102:0/4023652196 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8edc01a070 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.693+0000 7f8ee4c00640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ec806bc20 0x7f8ec806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:29:59.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.693+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ec806bc20 0x7f8ec806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:29:59.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.694+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ec806bc20 0x7f8ec806e0e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4036000 tx=0x7f8ed403c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:29:59.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.695+0000 7f8ee4c00640 1 -- 172.21.15.102:0/4023652196 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8edc056590 con 0x7f8ee80c4fa0 2024-03-18T16:29:59.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:29:59.707+0000 7f8ee4c00640 1 -- 172.21.15.102:0/4023652196 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8ee8152360 con 0x7f8ee80c4fa0 2024-03-18T16:30:00.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.112+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ee8074fd0 con 0x7f8ee80c4fa0 2024-03-18T16:30:00.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.125+0000 7f8ee4c00640 1 -- 172.21.15.102:0/4023652196 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8edc021080 con 0x7f8ee80c4fa0 2024-03-18T16:30:00.127 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:00.127 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:00.127 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.127+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ec806bc20 msgr2=0x7f8ec806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.127+0000 7f8eee400640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ec806bc20 0x7f8ec806e0e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4036000 tx=0x7f8ed403c040 comp rx=0 tx=0).stop 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.127+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee80c4fa0 msgr2=0x7f8ee815b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.127+0000 7f8eee400640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee80c4fa0 0x7f8ee815b180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00c490 tx=0x7f8edc00c950 comp rx=0 tx=0).stop 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.128+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 shutdown_connections 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.128+0000 7f8eee400640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ec806bc20 0x7f8ec806e0e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.128+0000 7f8eee400640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8ee8158420 0x7f8ee81394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.128+0000 7f8eee400640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee8141650 0x7f8ee81f00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.128+0000 7f8eee400640 1 --2- 172.21.15.102:0/4023652196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ee80c4fa0 0x7f8ee815b180 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.128+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 >> 172.21.15.102:0/4023652196 conn(0x7f8ee80cb890 msgr2=0x7f8ee8071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.128+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 shutdown_connections 2024-03-18T16:30:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:00.128+0000 7f8eee400640 1 -- 172.21.15.102:0/4023652196 wait complete. 2024-03-18T16:30:00.758 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:00.759 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:01.759 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.981+0000 7f1682e00640 1 Processor -- start 2024-03-18T16:30:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.981+0000 7f1682e00640 1 -- start start 2024-03-18T16:30:01.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.983+0000 7f1682e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c15ae70 0x7f167c0ca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:01.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.983+0000 7f1682e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f167c0cc4f0 0x7f167c0cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:01.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.983+0000 7f1682e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f167c154ab0 con 0x7f167c0c9be0 2024-03-18T16:30:01.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.983+0000 7f1682e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f167c0cce80 con 0x7f167c0cc4f0 2024-03-18T16:30:01.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.983+0000 7f1682e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f167c06d2f0 con 0x7f167c15ae70 2024-03-18T16:30:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.984+0000 7f1681a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c15ae70 0x7f167c0ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.984+0000 7f1681a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c15ae70 0x7f167c0ca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58926/0 (socket says 172.21.15.102:58926) 2024-03-18T16:30:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.984+0000 7f1681a00640 1 -- 172.21.15.102:0/1277145096 learned_addr learned my addr 172.21.15.102:0/1277145096 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.985+0000 7f1682400640 1 --2- 172.21.15.102:0/1277145096 >> v2:172.21.15.154:3300/0 conn(0x7f167c0cc4f0 0x7f167c0cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.987+0000 7f1681a00640 1 -- 172.21.15.102:0/1277145096 >> v2:172.21.15.154:3300/0 conn(0x7f167c0cc4f0 msgr2=0x7f167c0cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:01.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.987+0000 7f1681a00640 1 --2- 172.21.15.102:0/1277145096 >> v2:172.21.15.154:3300/0 conn(0x7f167c0cc4f0 0x7f167c0cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:01.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.987+0000 7f1681a00640 1 -- 172.21.15.102:0/1277145096 >> v1:172.21.15.102:6789/0 conn(0x7f167c0c9be0 legacy=0x7f167c0d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:01.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.987+0000 7f1681a00640 1 -- 172.21.15.102:0/1277145096 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c15b250 con 0x7f167c15ae70 2024-03-18T16:30:01.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.989+0000 7f1681a00640 1 --2- 172.21.15.102:0/1277145096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c15ae70 0x7f167c0ca490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1670009870 tx=0x7f167002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2191593b14e7131 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.994+0000 7f167be00640 1 -- 172.21.15.102:0/1277145096 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670034070 con 0x7f167c15ae70 2024-03-18T16:30:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.994+0000 7f167be00640 1 -- 172.21.15.102:0/1277145096 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1670030840 con 0x7f167c15ae70 2024-03-18T16:30:01.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.994+0000 7f167be00640 1 -- 172.21.15.102:0/1277145096 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670030c10 con 0x7f167c15ae70 2024-03-18T16:30:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.995+0000 7f1682e00640 1 -- 172.21.15.102:0/1277145096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c15ae70 msgr2=0x7f167c0ca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.996+0000 7f1682e00640 1 --2- 172.21.15.102:0/1277145096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c15ae70 0x7f167c0ca490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1670009870 tx=0x7f167002b980 comp rx=0 tx=0).stop 2024-03-18T16:30:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.996+0000 7f1682e00640 1 -- 172.21.15.102:0/1277145096 shutdown_connections 2024-03-18T16:30:01.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.996+0000 7f1682e00640 1 --2- 172.21.15.102:0/1277145096 >> v2:172.21.15.154:3300/0 conn(0x7f167c0cc4f0 0x7f167c0cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.996+0000 7f1682e00640 1 --2- 172.21.15.102:0/1277145096 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c15ae70 0x7f167c0ca490 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.996+0000 7f1682e00640 1 -- 172.21.15.102:0/1277145096 >> 172.21.15.102:0/1277145096 conn(0x7f167c0673a0 msgr2=0x7f167c0677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:01.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.996+0000 7f1682e00640 1 -- 172.21.15.102:0/1277145096 shutdown_connections 2024-03-18T16:30:01.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.996+0000 7f1682e00640 1 -- 172.21.15.102:0/1277145096 wait complete. 2024-03-18T16:30:02.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:01.999+0000 7f1682e00640 1 Processor -- start 2024-03-18T16:30:02.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.000+0000 7f1682e00640 1 -- start start 2024-03-18T16:30:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.001+0000 7f1682e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c0c9be0 0x7f167c0b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.002+0000 7f1682e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f167c0cc4f0 0x7f167c159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:02.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.002+0000 7f1682e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f167c15ae70 0x7f167c154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.002+0000 7f1681a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c0c9be0 0x7f167c0b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.002+0000 7f1682e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f167c06db90 con 0x7f167c0cc4f0 2024-03-18T16:30:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.002+0000 7f1682e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f167c06d8b0 con 0x7f167c15ae70 2024-03-18T16:30:02.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.002+0000 7f1682e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f167c06de70 con 0x7f167c0c9be0 2024-03-18T16:30:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.003+0000 7f1682400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f167c15ae70 0x7f167c154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.003+0000 7f1681a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c0c9be0 0x7f167c0b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58942/0 (socket says 172.21.15.102:58942) 2024-03-18T16:30:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.003+0000 7f1681a00640 1 -- 172.21.15.102:0/3095013778 learned_addr learned my addr 172.21.15.102:0/3095013778 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:02.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.003+0000 7f1681000640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f167c0cc4f0 0x7f167c159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.006+0000 7f1681a00640 1 -- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f167c15ae70 msgr2=0x7f167c154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.006+0000 7f1681a00640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f167c15ae70 0x7f167c154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.006+0000 7f1681a00640 1 -- 172.21.15.102:0/3095013778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f167c0cc4f0 msgr2=0x7f167c159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.006+0000 7f1681a00640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f167c0cc4f0 0x7f167c159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:02.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.006+0000 7f1681a00640 1 -- 172.21.15.102:0/3095013778 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c0afe60 con 0x7f167c0c9be0 2024-03-18T16:30:02.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.008+0000 7f1681a00640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c0c9be0 0x7f167c0b26d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16700090b0 tx=0x7f16700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.010+0000 7f1679600640 1 -- 172.21.15.102:0/3095013778 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670034040 con 0x7f167c0c9be0 2024-03-18T16:30:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.011+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167c0b5230 con 0x7f167c0c9be0 2024-03-18T16:30:02.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.011+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f167c0b57f0 con 0x7f167c0c9be0 2024-03-18T16:30:02.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.012+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167c074fd0 con 0x7f167c0c9be0 2024-03-18T16:30:02.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.047+0000 7f1679600640 1 -- 172.21.15.102:0/3095013778 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1670030840 con 0x7f167c0c9be0 2024-03-18T16:30:02.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.048+0000 7f1679600640 1 -- 172.21.15.102:0/3095013778 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167004d450 con 0x7f167c0c9be0 2024-03-18T16:30:02.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.050+0000 7f1679600640 1 -- 172.21.15.102:0/3095013778 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f167004b060 con 0x7f167c0c9be0 2024-03-18T16:30:02.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.051+0000 7f1679600640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f165c06bbd0 0x7f165c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:02.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.052+0000 7f1679600640 1 -- 172.21.15.102:0/3095013778 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f16700792b0 con 0x7f167c0c9be0 2024-03-18T16:30:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.052+0000 7f1681000640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f165c06bbd0 0x7f165c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.053+0000 7f1681000640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f165c06bbd0 0x7f165c06e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f167c155f20 tx=0x7f1668009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:02.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.362+0000 7f1679600640 1 -- 172.21.15.102:0/3095013778 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f167003f070 con 0x7f167c0c9be0 2024-03-18T16:30:02.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.631+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f167c0b5a90 con 0x7f167c0c9be0 2024-03-18T16:30:02.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.645+0000 7f1679600640 1 -- 172.21.15.102:0/3095013778 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f167007b910 con 0x7f167c0c9be0 2024-03-18T16:30:02.647 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:02.647 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:02.648 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f165c06bbd0 msgr2=0x7f165c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f165c06bbd0 0x7f165c06e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f167c155f20 tx=0x7f1668009040 comp rx=0 tx=0).stop 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c0c9be0 msgr2=0x7f167c0b26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c0c9be0 0x7f167c0b26d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16700090b0 tx=0x7f16700046c0 comp rx=0 tx=0).stop 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 shutdown_connections 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f165c06bbd0 0x7f165c06e090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f167c15ae70 0x7f167c154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f167c0cc4f0 0x7f167c159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 --2- 172.21.15.102:0/3095013778 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f167c0c9be0 0x7f167c0b26d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 >> 172.21.15.102:0/3095013778 conn(0x7f167c0673a0 msgr2=0x7f167c0cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:02.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 shutdown_connections 2024-03-18T16:30:02.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:02.649+0000 7f1682e00640 1 -- 172.21.15.102:0/3095013778 wait complete. 2024-03-18T16:30:03.273 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:03.273 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:04.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:04.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.447+0000 7f8285200640 1 Processor -- start 2024-03-18T16:30:04.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.447+0000 7f8285200640 1 -- start start 2024-03-18T16:30:04.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.448+0000 7f8285200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f82800d0250 0x7f828015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:04.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.448+0000 7f8285200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800cc2f0 0x7f82800cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:04.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.448+0000 7f8285200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82800cabd0 con 0x7f82800cfe70 2024-03-18T16:30:04.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.448+0000 7f8285200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f82800caea0 con 0x7f82800d0250 2024-03-18T16:30:04.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.448+0000 7f8285200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f82800cb180 con 0x7f82800cc2f0 2024-03-18T16:30:04.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.448+0000 7f827f400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f82800cfe70 0x7f82800c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:57192/0 (socket says 172.21.15.102:57192) 2024-03-18T16:30:04.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.449+0000 7f827f400640 1 -- 172.21.15.102:0/3626422850 learned_addr learned my addr 172.21.15.102:0/3626422850 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:04.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.449+0000 7f827ea00640 1 --2- 172.21.15.102:0/3626422850 >> v2:172.21.15.102:3300/0 conn(0x7f82800d0250 0x7f828015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:04.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.449+0000 7f827fe00640 1 --2- 172.21.15.102:0/3626422850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800cc2f0 0x7f82800cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:04.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.452+0000 7f827e000640 1 -- 172.21.15.102:0/3626422850 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4214297797 0 0) 0x7f82800cabd0 con 0x7f82800cfe70 2024-03-18T16:30:04.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.452+0000 7f827e000640 1 -- 172.21.15.102:0/3626422850 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8264003520 con 0x7f82800cfe70 2024-03-18T16:30:04.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.452+0000 7f827ea00640 1 -- 172.21.15.102:0/3626422850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800cc2f0 msgr2=0x7f82800cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:04.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.452+0000 7f827ea00640 1 --2- 172.21.15.102:0/3626422850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800cc2f0 0x7f82800cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:04.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.452+0000 7f827ea00640 1 -- 172.21.15.102:0/3626422850 >> v1:172.21.15.154:6789/0 conn(0x7f82800cfe70 legacy=0x7f82800c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:04.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.452+0000 7f827ea00640 1 -- 172.21.15.102:0/3626422850 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82800cb460 con 0x7f82800d0250 2024-03-18T16:30:04.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.455+0000 7f827ea00640 1 --2- 172.21.15.102:0/3626422850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82800d0250 0x7f828015ae40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f826c0097a0 tx=0x7f826c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46543f9d70e94ac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:04.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.459+0000 7f827e000640 1 -- 172.21.15.102:0/3626422850 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c034070 con 0x7f82800d0250 2024-03-18T16:30:04.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.459+0000 7f827e000640 1 -- 172.21.15.102:0/3626422850 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f826c030800 con 0x7f82800d0250 2024-03-18T16:30:04.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.459+0000 7f827e000640 1 -- 172.21.15.102:0/3626422850 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c030bd0 con 0x7f82800d0250 2024-03-18T16:30:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.460+0000 7f8285200640 1 -- 172.21.15.102:0/3626422850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82800d0250 msgr2=0x7f828015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.460+0000 7f8285200640 1 --2- 172.21.15.102:0/3626422850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82800d0250 0x7f828015ae40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f826c0097a0 tx=0x7f826c02b940 comp rx=0 tx=0).stop 2024-03-18T16:30:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.460+0000 7f8285200640 1 -- 172.21.15.102:0/3626422850 shutdown_connections 2024-03-18T16:30:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.460+0000 7f8285200640 1 --2- 172.21.15.102:0/3626422850 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800cc2f0 0x7f82800cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.460+0000 7f8285200640 1 --2- 172.21.15.102:0/3626422850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82800d0250 0x7f828015ae40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.460+0000 7f8285200640 1 -- 172.21.15.102:0/3626422850 >> 172.21.15.102:0/3626422850 conn(0x7f828008c3e0 msgr2=0x7f828006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.461+0000 7f8285200640 1 -- 172.21.15.102:0/3626422850 shutdown_connections 2024-03-18T16:30:04.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.461+0000 7f8285200640 1 -- 172.21.15.102:0/3626422850 wait complete. 2024-03-18T16:30:04.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.463+0000 7f8285200640 1 Processor -- start 2024-03-18T16:30:04.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.463+0000 7f8285200640 1 -- start start 2024-03-18T16:30:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.465+0000 7f8285200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82800cc2f0 0x7f82801e24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.465+0000 7f8285200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82800cfe70 0x7f82801f0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.465+0000 7f8285200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800d0250 0x7f82801f1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.465+0000 7f8285200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f82800cbd00 con 0x7f82800cfe70 2024-03-18T16:30:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.465+0000 7f8285200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82800cba20 con 0x7f82800cc2f0 2024-03-18T16:30:04.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.465+0000 7f8285200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f82800ccc50 con 0x7f82800d0250 2024-03-18T16:30:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.466+0000 7f827fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800d0250 0x7f82801f1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.466+0000 7f827f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82800cc2f0 0x7f82801e24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.466+0000 7f827ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82800cfe70 0x7f82801f0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.466+0000 7f827fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800d0250 0x7f82801f1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58968/0 (socket says 172.21.15.102:58968) 2024-03-18T16:30:04.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.466+0000 7f827fe00640 1 -- 172.21.15.102:0/1552340104 learned_addr learned my addr 172.21.15.102:0/1552340104 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.469+0000 7f827fe00640 1 -- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82800cc2f0 msgr2=0x7f82801e24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.469+0000 7f827fe00640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82800cc2f0 0x7f82801e24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.469+0000 7f827fe00640 1 -- 172.21.15.102:0/1552340104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82800cfe70 msgr2=0x7f82801f0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.469+0000 7f827fe00640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82800cfe70 0x7f82801f0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:04.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.469+0000 7f827fe00640 1 -- 172.21.15.102:0/1552340104 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8280156810 con 0x7f82800d0250 2024-03-18T16:30:04.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.471+0000 7f827fe00640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800d0250 0x7f82801f1200 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8278009dd0 tx=0x7f827800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:04.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.475+0000 7f8277400640 1 -- 172.21.15.102:0/1552340104 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8278018020 con 0x7f82800d0250 2024-03-18T16:30:04.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.475+0000 7f8277400640 1 -- 172.21.15.102:0/1552340104 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8278014480 con 0x7f82800d0250 2024-03-18T16:30:04.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.475+0000 7f8277400640 1 -- 172.21.15.102:0/1552340104 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8278014870 con 0x7f82800d0250 2024-03-18T16:30:04.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.475+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8280156af0 con 0x7f82800d0250 2024-03-18T16:30:04.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.475+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82801570b0 con 0x7f82800d0250 2024-03-18T16:30:04.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.477+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82801573e0 con 0x7f82800d0250 2024-03-18T16:30:04.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.496+0000 7f8277400640 1 -- 172.21.15.102:0/1552340104 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8278005d10 con 0x7f82800d0250 2024-03-18T16:30:04.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.497+0000 7f8277400640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f826006bbd0 0x7f826006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:04.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.498+0000 7f8277400640 1 -- 172.21.15.102:0/1552340104 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f82780584c0 con 0x7f82800d0250 2024-03-18T16:30:04.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.498+0000 7f827f400640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f826006bbd0 0x7f826006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:04.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.498+0000 7f827f400640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f826006bbd0 0x7f826006e090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8270005880 tx=0x7f8270009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:04.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:04.771+0000 7f8277400640 1 -- 172.21.15.102:0/1552340104 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8278022030 con 0x7f82800d0250 2024-03-18T16:30:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.041+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8280074fd0 con 0x7f82800d0250 2024-03-18T16:30:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.194+0000 7f8277400640 1 -- 172.21.15.102:0/1552340104 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8278010040 con 0x7f82800d0250 2024-03-18T16:30:05.196 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:05.196 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f826006bbd0 msgr2=0x7f826006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f826006bbd0 0x7f826006e090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8270005880 tx=0x7f8270009040 comp rx=0 tx=0).stop 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800d0250 msgr2=0x7f82801f1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800d0250 0x7f82801f1200 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8278009dd0 tx=0x7f827800c490 comp rx=0 tx=0).stop 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 shutdown_connections 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f826006bbd0 0x7f826006e090 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82800d0250 0x7f82801f1200 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82800cfe70 0x7f82801f0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 --2- 172.21.15.102:0/1552340104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82800cc2f0 0x7f82801e24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 >> 172.21.15.102:0/1552340104 conn(0x7f828008c3e0 msgr2=0x7f82800c6300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:05.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 shutdown_connections 2024-03-18T16:30:05.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:05.198+0000 7f8285200640 1 -- 172.21.15.102:0/1552340104 wait complete. 2024-03-18T16:30:05.387 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:31.647 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:30:05.404 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:31.663 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:30:05.405 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:31.750 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:30:05.408 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:09:31.753 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:30:05.408 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:31.754 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:30:05.412 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:09:31.757 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:30:05.474 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:09:31.819 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:30:05.475 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:09:31.820 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:30:05.478 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:09:31.737 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:30:05.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:05.823 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:06.824 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:07.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.012+0000 7f180f200640 1 Processor -- start 2024-03-18T16:30:07.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.012+0000 7f180f200640 1 -- start start 2024-03-18T16:30:07.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.013+0000 7f180f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1808141650 0x7f1808158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:07.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.013+0000 7f180f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18080c4fa0 0x7f18080c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:07.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.013+0000 7f180f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18080c72d0 con 0x7f180815c3a0 2024-03-18T16:30:07.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.013+0000 7f180f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1808159370 con 0x7f1808141650 2024-03-18T16:30:07.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.013+0000 7f180f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1808158ad0 con 0x7f18080c4fa0 2024-03-18T16:30:07.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.014+0000 7f180e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18080c4fa0 0x7f18080c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:07.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.014+0000 7f180de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1808141650 0x7f1808158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:07.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.015+0000 7f180d400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f180815c3a0 0x7f18080c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49182/0 (socket says 172.21.15.102:49182) 2024-03-18T16:30:07.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.015+0000 7f180d400640 1 -- 172.21.15.102:0/1674338755 learned_addr learned my addr 172.21.15.102:0/1674338755 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:07.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.017+0000 7f180e800640 1 -- 172.21.15.102:0/1674338755 >> v2:172.21.15.102:3300/0 conn(0x7f1808141650 msgr2=0x7f1808158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:07.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.017+0000 7f180e800640 1 --2- 172.21.15.102:0/1674338755 >> v2:172.21.15.102:3300/0 conn(0x7f1808141650 0x7f1808158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.017+0000 7f180e800640 1 -- 172.21.15.102:0/1674338755 >> v1:172.21.15.102:6789/0 conn(0x7f180815c3a0 legacy=0x7f18080c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:07.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.017+0000 7f180e800640 1 -- 172.21.15.102:0/1674338755 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1808158db0 con 0x7f18080c4fa0 2024-03-18T16:30:07.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.020+0000 7f180e800640 1 --2- 172.21.15.102:0/1674338755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18080c4fa0 0x7f18080c5430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1804009d90 tx=0x7f180402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d31fff3266f0d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.024+0000 7f180ca00640 1 -- 172.21.15.102:0/1674338755 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804039070 con 0x7f18080c4fa0 2024-03-18T16:30:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.024+0000 7f180ca00640 1 -- 172.21.15.102:0/1674338755 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f180402cc60 con 0x7f18080c4fa0 2024-03-18T16:30:07.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.024+0000 7f180ca00640 1 -- 172.21.15.102:0/1674338755 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804034660 con 0x7f18080c4fa0 2024-03-18T16:30:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.025+0000 7f180f200640 1 -- 172.21.15.102:0/1674338755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18080c4fa0 msgr2=0x7f18080c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.025+0000 7f180f200640 1 --2- 172.21.15.102:0/1674338755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18080c4fa0 0x7f18080c5430 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1804009d90 tx=0x7f180402beb0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.025+0000 7f180f200640 1 -- 172.21.15.102:0/1674338755 shutdown_connections 2024-03-18T16:30:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.025+0000 7f180f200640 1 --2- 172.21.15.102:0/1674338755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18080c4fa0 0x7f18080c5430 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.025+0000 7f180f200640 1 --2- 172.21.15.102:0/1674338755 >> v2:172.21.15.102:3300/0 conn(0x7f1808141650 0x7f1808158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.025+0000 7f180f200640 1 -- 172.21.15.102:0/1674338755 >> 172.21.15.102:0/1674338755 conn(0x7f18080cb890 msgr2=0x7f18080cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.025+0000 7f180f200640 1 -- 172.21.15.102:0/1674338755 shutdown_connections 2024-03-18T16:30:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.025+0000 7f180f200640 1 -- 172.21.15.102:0/1674338755 wait complete. 2024-03-18T16:30:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.027+0000 7f180f200640 1 Processor -- start 2024-03-18T16:30:07.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.027+0000 7f180f200640 1 -- start start 2024-03-18T16:30:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.029+0000 7f180f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18080c4fa0 0x7f180815b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.029+0000 7f180f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1808141650 0x7f180813ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.029+0000 7f180f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f180815c3a0 0x7f180813e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.029+0000 7f180f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f18080c66a0 con 0x7f180815c3a0 2024-03-18T16:30:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.029+0000 7f180f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18080c63c0 con 0x7f18080c4fa0 2024-03-18T16:30:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.029+0000 7f180f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f180815c7c0 con 0x7f1808141650 2024-03-18T16:30:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.029+0000 7f180de00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18080c4fa0 0x7f180815b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:07.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.029+0000 7f180d400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1808141650 0x7f180813ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:07.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.029+0000 7f180e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f180815c3a0 0x7f180813e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:07.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.030+0000 7f180de00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18080c4fa0 0x7f180815b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42974/0 (socket says 172.21.15.102:42974) 2024-03-18T16:30:07.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.030+0000 7f180de00640 1 -- 172.21.15.102:0/362043918 learned_addr learned my addr 172.21.15.102:0/362043918 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:07.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.035+0000 7f180d400640 1 -- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18080c4fa0 msgr2=0x7f180815b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.035+0000 7f180d400640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18080c4fa0 0x7f180815b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.035+0000 7f180d400640 1 -- 172.21.15.102:0/362043918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f180815c3a0 msgr2=0x7f180813e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.035+0000 7f180d400640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f180815c3a0 0x7f180813e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.035+0000 7f180d400640 1 -- 172.21.15.102:0/362043918 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18081eee00 con 0x7f1808141650 2024-03-18T16:30:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.035+0000 7f180de00640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18080c4fa0 0x7f180815b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:30:07.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.038+0000 7f180d400640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1808141650 0x7f180813ded0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f17f800c440 tx=0x7f17f800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.042+0000 7f1802000640 1 -- 172.21.15.102:0/362043918 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8017070 con 0x7f1808141650 2024-03-18T16:30:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.042+0000 7f1802000640 1 -- 172.21.15.102:0/362043918 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f8005ce0 con 0x7f1808141650 2024-03-18T16:30:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.042+0000 7f1802000640 1 -- 172.21.15.102:0/362043918 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f80072d0 con 0x7f1808141650 2024-03-18T16:30:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.042+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18081e51e0 con 0x7f1808141650 2024-03-18T16:30:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.042+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18081eeb10 con 0x7f1808141650 2024-03-18T16:30:07.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.043+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18080c74a0 con 0x7f1808141650 2024-03-18T16:30:07.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.068+0000 7f1802000640 1 -- 172.21.15.102:0/362043918 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f17f801a070 con 0x7f1808141650 2024-03-18T16:30:07.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.069+0000 7f1802000640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f17ec06bc20 0x7f17ec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:07.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.069+0000 7f180de00640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f17ec06bc20 0x7f17ec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:07.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.070+0000 7f180de00640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f17ec06bc20 0x7f17ec06e0e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0112c0 tx=0x7f17fc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:07.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.324+0000 7f1802000640 1 -- 172.21.15.102:0/362043918 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f17f8056730 con 0x7f1808141650 2024-03-18T16:30:07.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.343+0000 7f1802000640 1 -- 172.21.15.102:0/362043918 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f18080c74a0 con 0x7f1808141650 2024-03-18T16:30:07.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.634+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1808074fd0 con 0x7f1808141650 2024-03-18T16:30:07.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.671+0000 7f1802000640 1 -- 172.21.15.102:0/362043918 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f17f8058690 con 0x7f1808141650 2024-03-18T16:30:07.673 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:07.673 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:07.674 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.674+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f17ec06bc20 msgr2=0x7f17ec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.674+0000 7f180f200640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f17ec06bc20 0x7f17ec06e0e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0112c0 tx=0x7f17fc009190 comp rx=0 tx=0).stop 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1808141650 msgr2=0x7f180813ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1808141650 0x7f180813ded0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f17f800c440 tx=0x7f17f800c900 comp rx=0 tx=0).stop 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 shutdown_connections 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f17ec06bc20 0x7f17ec06e0e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f180815c3a0 0x7f180813e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1808141650 0x7f180813ded0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 --2- 172.21.15.102:0/362043918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18080c4fa0 0x7f180815b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 >> 172.21.15.102:0/362043918 conn(0x7f18080cb890 msgr2=0x7f180813fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:07.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 shutdown_connections 2024-03-18T16:30:07.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:07.675+0000 7f180f200640 1 -- 172.21.15.102:0/362043918 wait complete. 2024-03-18T16:30:08.297 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:08.297 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:09.298 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:09.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.474+0000 7f6b9be00640 1 Processor -- start 2024-03-18T16:30:09.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.475+0000 7f6b9be00640 1 -- start start 2024-03-18T16:30:09.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.476+0000 7f6b9be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0cfe70 0x7f6b9c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:09.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.476+0000 7f6b9be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b9c0cabd0 con 0x7f6b9c0cc2f0 2024-03-18T16:30:09.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.476+0000 7f6b9be00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b9c0caea0 con 0x7f6b9c0d0250 2024-03-18T16:30:09.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.476+0000 7f6b9be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9c0cb180 con 0x7f6b9c0cfe70 2024-03-18T16:30:09.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.476+0000 7f6b9aa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0cfe70 0x7f6b9c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:09.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.477+0000 7f6b9a000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f6b9c0d0250 0x7f6b9c0c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:57206/0 (socket says 172.21.15.102:57206) 2024-03-18T16:30:09.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.477+0000 7f6b9a000640 1 -- 172.21.15.102:0/3610960258 learned_addr learned my addr 172.21.15.102:0/3610960258 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:09.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.479+0000 7f6b99600640 1 -- 172.21.15.102:0/3610960258 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 606132699 0 0) 0x7f6b9c0caea0 con 0x7f6b9c0d0250 2024-03-18T16:30:09.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.479+0000 7f6b99600640 1 -- 172.21.15.102:0/3610960258 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b80003520 con 0x7f6b9c0d0250 2024-03-18T16:30:09.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.479+0000 7f6b9aa00640 1 -- 172.21.15.102:0/3610960258 >> v1:172.21.15.154:6789/0 conn(0x7f6b9c0d0250 legacy=0x7f6b9c0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:09.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.479+0000 7f6b9aa00640 1 -- 172.21.15.102:0/3610960258 >> v1:172.21.15.102:6789/0 conn(0x7f6b9c0cc2f0 legacy=0x7f6b9c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:09.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.479+0000 7f6b9aa00640 1 -- 172.21.15.102:0/3610960258 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c0cb460 con 0x7f6b9c0cfe70 2024-03-18T16:30:09.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.481+0000 7f6b9aa00640 1 --2- 172.21.15.102:0/3610960258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0cfe70 0x7f6b9c15ae40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0097d0 tx=0x7f6b8c02b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df110d4d34bff634 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:09.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.486+0000 7f6b99600640 1 -- 172.21.15.102:0/3610960258 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c034070 con 0x7f6b9c0cfe70 2024-03-18T16:30:09.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.486+0000 7f6b99600640 1 -- 172.21.15.102:0/3610960258 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b8c030890 con 0x7f6b9c0cfe70 2024-03-18T16:30:09.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.486+0000 7f6b99600640 1 -- 172.21.15.102:0/3610960258 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c030c60 con 0x7f6b9c0cfe70 2024-03-18T16:30:09.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.486+0000 7f6b9be00640 1 -- 172.21.15.102:0/3610960258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0cfe70 msgr2=0x7f6b9c15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:09.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.486+0000 7f6b9be00640 1 --2- 172.21.15.102:0/3610960258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0cfe70 0x7f6b9c15ae40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c0097d0 tx=0x7f6b8c02b9d0 comp rx=0 tx=0).stop 2024-03-18T16:30:09.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.487+0000 7f6b9be00640 1 -- 172.21.15.102:0/3610960258 shutdown_connections 2024-03-18T16:30:09.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.487+0000 7f6b9be00640 1 --2- 172.21.15.102:0/3610960258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0cfe70 0x7f6b9c15ae40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:09.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.487+0000 7f6b9be00640 1 -- 172.21.15.102:0/3610960258 >> 172.21.15.102:0/3610960258 conn(0x7f6b9c08c3e0 msgr2=0x7f6b9c06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:09.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.487+0000 7f6b9be00640 1 -- 172.21.15.102:0/3610960258 shutdown_connections 2024-03-18T16:30:09.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.487+0000 7f6b9be00640 1 -- 172.21.15.102:0/3610960258 wait complete. 2024-03-18T16:30:09.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.489+0000 7f6b9be00640 1 Processor -- start 2024-03-18T16:30:09.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.489+0000 7f6b9be00640 1 -- start start 2024-03-18T16:30:09.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.491+0000 7f6b9be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b9c0cc2f0 0x7f6b9c158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:09.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.491+0000 7f6b9be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b9c0cfe70 0x7f6b9c1e0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:09.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.491+0000 7f6b9be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0d0250 0x7f6b9c156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:09.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.491+0000 7f6b9be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c0cbd00 con 0x7f6b9c0cfe70 2024-03-18T16:30:09.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.491+0000 7f6b9be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c0cba20 con 0x7f6b9c0cc2f0 2024-03-18T16:30:09.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.491+0000 7f6b9be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9c0c8330 con 0x7f6b9c0d0250 2024-03-18T16:30:09.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.492+0000 7f6b9aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b9c0cc2f0 0x7f6b9c158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:09.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.492+0000 7f6b9b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0d0250 0x7f6b9c156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:09.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.492+0000 7f6b9a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b9c0cfe70 0x7f6b9c1e0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:09.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.492+0000 7f6b9b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0d0250 0x7f6b9c156910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:59018/0 (socket says 172.21.15.102:59018) 2024-03-18T16:30:09.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.492+0000 7f6b9b400640 1 -- 172.21.15.102:0/1290800449 learned_addr learned my addr 172.21.15.102:0/1290800449 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:09.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.495+0000 7f6b9b400640 1 -- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b9c0cc2f0 msgr2=0x7f6b9c158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:09.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.495+0000 7f6b9b400640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b9c0cc2f0 0x7f6b9c158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:09.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.495+0000 7f6b9b400640 1 -- 172.21.15.102:0/1290800449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b9c0cfe70 msgr2=0x7f6b9c1e0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:09.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.495+0000 7f6b9b400640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b9c0cfe70 0x7f6b9c1e0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:09.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.495+0000 7f6b9b400640 1 -- 172.21.15.102:0/1290800449 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c157f80 con 0x7f6b9c0d0250 2024-03-18T16:30:09.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.497+0000 7f6b9b400640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0d0250 0x7f6b9c156910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400c7d0 tx=0x7f6b9400cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:09.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.502+0000 7f6b92a00640 1 -- 172.21.15.102:0/1290800449 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94003b70 con 0x7f6b9c0d0250 2024-03-18T16:30:09.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.502+0000 7f6b92a00640 1 -- 172.21.15.102:0/1290800449 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b94003d30 con 0x7f6b9c0d0250 2024-03-18T16:30:09.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.502+0000 7f6b92a00640 1 -- 172.21.15.102:0/1290800449 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400aa90 con 0x7f6b9c0d0250 2024-03-18T16:30:09.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.502+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9c1e14e0 con 0x7f6b9c0d0250 2024-03-18T16:30:09.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.502+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9c1e1aa0 con 0x7f6b9c0d0250 2024-03-18T16:30:09.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.504+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9c074fd0 con 0x7f6b9c0d0250 2024-03-18T16:30:09.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.530+0000 7f6b92a00640 1 -- 172.21.15.102:0/1290800449 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6b9400ac30 con 0x7f6b9c0d0250 2024-03-18T16:30:09.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.531+0000 7f6b92a00640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6b7c06bc50 0x7f6b7c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:09.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.531+0000 7f6b9aa00640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6b7c06bc50 0x7f6b7c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:09.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.532+0000 7f6b9aa00640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6b7c06bc50 0x7f6b7c06e110 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c036000 tx=0x7f6b8c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:09.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.767+0000 7f6b92a00640 1 -- 172.21.15.102:0/1290800449 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6b94057850 con 0x7f6b9c0d0250 2024-03-18T16:30:09.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:09.779+0000 7f6b92a00640 1 -- 172.21.15.102:0/1290800449 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6b940597b0 con 0x7f6b9c0d0250 2024-03-18T16:30:10.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.153+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b9c0b5720 con 0x7f6b9c0d0250 2024-03-18T16:30:10.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.169+0000 7f6b92a00640 1 -- 172.21.15.102:0/1290800449 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f6b9405d850 con 0x7f6b9c0d0250 2024-03-18T16:30:10.170 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:10.171 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:10.171 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:10.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6b7c06bc50 msgr2=0x7f6b7c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:10.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6b7c06bc50 0x7f6b7c06e110 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c036000 tx=0x7f6b8c03b040 comp rx=0 tx=0).stop 2024-03-18T16:30:10.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0d0250 msgr2=0x7f6b9c156910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:10.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0d0250 0x7f6b9c156910 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400c7d0 tx=0x7f6b9400cc90 comp rx=0 tx=0).stop 2024-03-18T16:30:10.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 shutdown_connections 2024-03-18T16:30:10.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6b7c06bc50 0x7f6b7c06e110 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:10.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6b9c0d0250 0x7f6b9c156910 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:10.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b9c0cfe70 0x7f6b9c1e0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:10.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 --2- 172.21.15.102:0/1290800449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b9c0cc2f0 0x7f6b9c158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:10.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 >> 172.21.15.102:0/1290800449 conn(0x7f6b9c08c3e0 msgr2=0x7f6b9c0c38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:10.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 shutdown_connections 2024-03-18T16:30:10.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:10.172+0000 7f6b9be00640 1 -- 172.21.15.102:0/1290800449 wait complete. 2024-03-18T16:30:10.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:10.795 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:11.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:11.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:11.976+0000 7f0cc1e00640 1 Processor -- start 2024-03-18T16:30:11.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:11.976+0000 7f0cc1e00640 1 -- start start 2024-03-18T16:30:11.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:11.977+0000 7f0cc1e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0cbc0c2b40 0x7f0cbc06ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:11.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:11.977+0000 7f0cc1e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cbc0c6cd0 con 0x7f0cbc06b200 2024-03-18T16:30:11.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:11.977+0000 7f0cc1e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cbc0c73d0 con 0x7f0cbc158660 2024-03-18T16:30:11.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:11.977+0000 7f0cc1e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0cbc1581d0 con 0x7f0cbc0c2b40 2024-03-18T16:30:12.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.021+0000 7f0cc1400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0cbc158660 0x7f0cbc158a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:42130/0 (socket says 172.21.15.102:42130) 2024-03-18T16:30:12.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.021+0000 7f0cc0a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0cbc0c2b40 0x7f0cbc06ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:12.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.021+0000 7f0cc1400640 1 -- 172.21.15.102:0/4207965916 learned_addr learned my addr 172.21.15.102:0/4207965916 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:12.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.026+0000 7f0cc0a00640 1 -- 172.21.15.102:0/4207965916 >> v1:172.21.15.154:6789/0 conn(0x7f0cbc158660 legacy=0x7f0cbc158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:12.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.026+0000 7f0cc0a00640 1 -- 172.21.15.102:0/4207965916 >> v1:172.21.15.102:6789/0 conn(0x7f0cbc06b200 legacy=0x7f0cbc0c5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:12.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.026+0000 7f0cc0a00640 1 -- 172.21.15.102:0/4207965916 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc0c76b0 con 0x7f0cbc0c2b40 2024-03-18T16:30:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.031+0000 7f0cc0a00640 1 --2- 172.21.15.102:0/4207965916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 0x7f0cbc06ba00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00097a0 tx=0x7f0cb002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a37ad61c9d152e9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.036+0000 7f0cbb400640 1 -- 172.21.15.102:0/4207965916 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0038070 con 0x7f0cbc0c2b40 2024-03-18T16:30:12.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.036+0000 7f0cbb400640 1 -- 172.21.15.102:0/4207965916 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb0031700 con 0x7f0cbc0c2b40 2024-03-18T16:30:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.036+0000 7f0cbb400640 1 -- 172.21.15.102:0/4207965916 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0031ad0 con 0x7f0cbc0c2b40 2024-03-18T16:30:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.037+0000 7f0cc1e00640 1 -- 172.21.15.102:0/4207965916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 msgr2=0x7f0cbc06ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.037+0000 7f0cc1e00640 1 --2- 172.21.15.102:0/4207965916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 0x7f0cbc06ba00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00097a0 tx=0x7f0cb002b7a0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.037+0000 7f0cc1e00640 1 -- 172.21.15.102:0/4207965916 shutdown_connections 2024-03-18T16:30:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.037+0000 7f0cc1e00640 1 --2- 172.21.15.102:0/4207965916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 0x7f0cbc06ba00 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.037+0000 7f0cc1e00640 1 -- 172.21.15.102:0/4207965916 >> 172.21.15.102:0/4207965916 conn(0x7f0cbc0727e0 msgr2=0x7f0cbc072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.037+0000 7f0cc1e00640 1 -- 172.21.15.102:0/4207965916 shutdown_connections 2024-03-18T16:30:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.037+0000 7f0cc1e00640 1 -- 172.21.15.102:0/4207965916 wait complete. 2024-03-18T16:30:12.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.039+0000 7f0cc1e00640 1 Processor -- start 2024-03-18T16:30:12.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.039+0000 7f0cc1e00640 1 -- start start 2024-03-18T16:30:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.040+0000 7f0cc1e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cbc06b200 0x7f0cbc155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.040+0000 7f0cc1e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 0x7f0cbc0c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.040+0000 7f0cc1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cbc158660 0x7f0cbc13e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.040+0000 7f0cc1e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc0c7f50 con 0x7f0cbc158660 2024-03-18T16:30:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.040+0000 7f0cc1e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc0c7c70 con 0x7f0cbc0c2b40 2024-03-18T16:30:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.040+0000 7f0cc1e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0cbc0c2f20 con 0x7f0cbc06b200 2024-03-18T16:30:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.042+0000 7f0cbbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 0x7f0cbc0c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.042+0000 7f0cbbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 0x7f0cbc0c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:37678/0 (socket says 172.21.15.102:37678) 2024-03-18T16:30:12.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.042+0000 7f0cbbe00640 1 -- 172.21.15.102:0/978524279 learned_addr learned my addr 172.21.15.102:0/978524279 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:12.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.042+0000 7f0cc0a00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cbc06b200 0x7f0cbc155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.046+0000 7f0cc0a00640 1 -- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 msgr2=0x7f0cbc0c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.046+0000 7f0cc0a00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 0x7f0cbc0c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.046+0000 7f0cc0a00640 1 -- 172.21.15.102:0/978524279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cbc158660 msgr2=0x7f0cbc13e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.046+0000 7f0cc0a00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cbc158660 0x7f0cbc13e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.046+0000 7f0cc0a00640 1 -- 172.21.15.102:0/978524279 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc1f07c0 con 0x7f0cbc06b200 2024-03-18T16:30:12.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.048+0000 7f0cc0a00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cbc06b200 0x7f0cbc155320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0033000 tx=0x7f0cb00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.055+0000 7f0cb8c00640 1 -- 172.21.15.102:0/978524279 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0043070 con 0x7f0cbc06b200 2024-03-18T16:30:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.055+0000 7f0cb8c00640 1 -- 172.21.15.102:0/978524279 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb0033a30 con 0x7f0cbc06b200 2024-03-18T16:30:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.055+0000 7f0cb8c00640 1 -- 172.21.15.102:0/978524279 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0038040 con 0x7f0cbc06b200 2024-03-18T16:30:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.055+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cbc1f0a80 con 0x7f0cbc06b200 2024-03-18T16:30:12.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.055+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cbc1f1040 con 0x7f0cbc06b200 2024-03-18T16:30:12.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.057+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cbc152680 con 0x7f0cbc06b200 2024-03-18T16:30:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.089+0000 7f0cb8c00640 1 -- 172.21.15.102:0/978524279 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0cb0046070 con 0x7f0cbc06b200 2024-03-18T16:30:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.091+0000 7f0cb8c00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0c9c06bc00 0x7f0c9c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.091+0000 7f0cbbe00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0c9c06bc00 0x7f0c9c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.092+0000 7f0cbbe00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0c9c06bc00 0x7f0c9c06e0c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0c19d0 tx=0x7f0ca80090e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:12.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.331+0000 7f0cb8c00640 1 -- 172.21.15.102:0/978524279 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f0cb0078f90 con 0x7f0cbc06b200 2024-03-18T16:30:12.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.349+0000 7f0cb8c00640 1 -- 172.21.15.102:0/978524279 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0cbc152680 con 0x7f0cbc06b200 2024-03-18T16:30:12.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.611+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0cbc06b7e0 con 0x7f0cbc06b200 2024-03-18T16:30:12.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.626+0000 7f0cb8c00640 1 -- 172.21.15.102:0/978524279 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f0cb0030070 con 0x7f0cbc06b200 2024-03-18T16:30:12.629 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:12.629 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:12.629 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0c9c06bc00 msgr2=0x7f0c9c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0c9c06bc00 0x7f0c9c06e0c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0c19d0 tx=0x7f0ca80090e0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cbc06b200 msgr2=0x7f0cbc155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cbc06b200 0x7f0cbc155320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0033000 tx=0x7f0cb00046c0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 shutdown_connections 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0c9c06bc00 0x7f0c9c06e0c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cbc158660 0x7f0cbc13e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cbc0c2b40 0x7f0cbc0c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 --2- 172.21.15.102:0/978524279 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0cbc06b200 0x7f0cbc155320 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.628+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 >> 172.21.15.102:0/978524279 conn(0x7f0cbc0727e0 msgr2=0x7f0cbc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.629+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 shutdown_connections 2024-03-18T16:30:12.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:12.629+0000 7f0cc1e00640 1 -- 172.21.15.102:0/978524279 wait complete. 2024-03-18T16:30:13.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:13.253 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:14.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.424+0000 7fe70cc00640 1 Processor -- start 2024-03-18T16:30:14.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.424+0000 7fe70cc00640 1 -- start start 2024-03-18T16:30:14.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.425+0000 7fe70cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080d0250 0x7fe70815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:14.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.426+0000 7fe70cc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe7080cc2f0 0x7fe7080cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.426+0000 7fe70cc00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7080ccca0 con 0x7fe7080cfe70 2024-03-18T16:30:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.426+0000 7fe70cc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe7080cabd0 con 0x7fe7080cc2f0 2024-03-18T16:30:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.426+0000 7fe70cc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe7080caea0 con 0x7fe7080d0250 2024-03-18T16:30:14.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.426+0000 7fe707400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe7080cfe70 0x7fe7080c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:42146/0 (socket says 172.21.15.102:42146) 2024-03-18T16:30:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.426+0000 7fe707400640 1 -- 172.21.15.102:0/164647765 learned_addr learned my addr 172.21.15.102:0/164647765 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.427+0000 7fe706a00640 1 --2- 172.21.15.102:0/164647765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080d0250 0x7fe70815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:14.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.427+0000 7fe707e00640 1 --2- 172.21.15.102:0/164647765 >> v2:172.21.15.102:3300/0 conn(0x7fe7080cc2f0 0x7fe7080cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:14.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.429+0000 7fe706000640 1 -- 172.21.15.102:0/164647765 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516114567 0 0) 0x7fe7080ccca0 con 0x7fe7080cfe70 2024-03-18T16:30:14.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.429+0000 7fe706a00640 1 -- 172.21.15.102:0/164647765 >> v2:172.21.15.102:3300/0 conn(0x7fe7080cc2f0 msgr2=0x7fe7080cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:14.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.429+0000 7fe706a00640 1 --2- 172.21.15.102:0/164647765 >> v2:172.21.15.102:3300/0 conn(0x7fe7080cc2f0 0x7fe7080cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:14.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.430+0000 7fe706a00640 1 -- 172.21.15.102:0/164647765 >> v1:172.21.15.154:6789/0 conn(0x7fe7080cfe70 legacy=0x7fe7080c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:14.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.430+0000 7fe706a00640 1 -- 172.21.15.102:0/164647765 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7080cb180 con 0x7fe7080d0250 2024-03-18T16:30:14.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.432+0000 7fe706a00640 1 --2- 172.21.15.102:0/164647765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080d0250 0x7fe70815ae40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40097a0 tx=0x7fe6f402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97ea5857d4f4f8cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:14.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.436+0000 7fe706000640 1 -- 172.21.15.102:0/164647765 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4034070 con 0x7fe7080d0250 2024-03-18T16:30:14.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.436+0000 7fe706000640 1 -- 172.21.15.102:0/164647765 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f4030800 con 0x7fe7080d0250 2024-03-18T16:30:14.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.436+0000 7fe706000640 1 -- 172.21.15.102:0/164647765 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4030bd0 con 0x7fe7080d0250 2024-03-18T16:30:14.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.437+0000 7fe70cc00640 1 -- 172.21.15.102:0/164647765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080d0250 msgr2=0x7fe70815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:14.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.437+0000 7fe70cc00640 1 --2- 172.21.15.102:0/164647765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080d0250 0x7fe70815ae40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40097a0 tx=0x7fe6f402b940 comp rx=0 tx=0).stop 2024-03-18T16:30:14.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.437+0000 7fe70cc00640 1 -- 172.21.15.102:0/164647765 shutdown_connections 2024-03-18T16:30:14.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.437+0000 7fe70cc00640 1 --2- 172.21.15.102:0/164647765 >> v2:172.21.15.102:3300/0 conn(0x7fe7080cc2f0 0x7fe7080cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:14.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.437+0000 7fe70cc00640 1 --2- 172.21.15.102:0/164647765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080d0250 0x7fe70815ae40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:14.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.437+0000 7fe70cc00640 1 -- 172.21.15.102:0/164647765 >> 172.21.15.102:0/164647765 conn(0x7fe70808c3e0 msgr2=0x7fe70806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:14.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.437+0000 7fe70cc00640 1 -- 172.21.15.102:0/164647765 shutdown_connections 2024-03-18T16:30:14.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.437+0000 7fe70cc00640 1 -- 172.21.15.102:0/164647765 wait complete. 2024-03-18T16:30:14.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.439+0000 7fe70cc00640 1 Processor -- start 2024-03-18T16:30:14.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.439+0000 7fe70cc00640 1 -- start start 2024-03-18T16:30:14.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.440+0000 7fe70cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7080cc2f0 0x7fe7081e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:14.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.440+0000 7fe70cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080cfe70 0x7fe708159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:14.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.440+0000 7fe70cc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7080d0250 0x7fe7081e0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:14.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.440+0000 7fe70cc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe7080cba20 con 0x7fe7080cc2f0 2024-03-18T16:30:14.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.440+0000 7fe70cc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe7080cb740 con 0x7fe7080d0250 2024-03-18T16:30:14.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.440+0000 7fe70cc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe7080cbd00 con 0x7fe7080cfe70 2024-03-18T16:30:14.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.441+0000 7fe707400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7080cc2f0 0x7fe7081e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:14.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.441+0000 7fe707e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7080d0250 0x7fe7081e0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:14.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.441+0000 7fe706a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080cfe70 0x7fe708159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:14.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.441+0000 7fe707400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7080cc2f0 0x7fe7081e03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48190/0 (socket says 172.21.15.102:48190) 2024-03-18T16:30:14.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.441+0000 7fe707400640 1 -- 172.21.15.102:0/3106113322 learned_addr learned my addr 172.21.15.102:0/3106113322 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:14.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.445+0000 7fe706a00640 1 -- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7080d0250 msgr2=0x7fe7081e0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:14.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.445+0000 7fe706a00640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7080d0250 0x7fe7081e0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:14.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.445+0000 7fe706a00640 1 -- 172.21.15.102:0/3106113322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7080cc2f0 msgr2=0x7fe7081e03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:14.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.445+0000 7fe706a00640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7080cc2f0 0x7fe7081e03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:14.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.445+0000 7fe706a00640 1 -- 172.21.15.102:0/3106113322 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7081f3790 con 0x7fe7080cfe70 2024-03-18T16:30:14.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.445+0000 7fe707400640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7080cc2f0 0x7fe7081e03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:30:14.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.447+0000 7fe706a00640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080cfe70 0x7fe708159420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe7080cb020 tx=0x7fe6f40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:14.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.449+0000 7fe6ff400640 1 -- 172.21.15.102:0/3106113322 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4034040 con 0x7fe7080cfe70 2024-03-18T16:30:14.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.450+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7081e6fd0 con 0x7fe7080cfe70 2024-03-18T16:30:14.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.450+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe708159960 con 0x7fe7080cfe70 2024-03-18T16:30:14.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.451+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe708074fd0 con 0x7fe7080cfe70 2024-03-18T16:30:14.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.452+0000 7fe6ff400640 1 -- 172.21.15.102:0/3106113322 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f4030800 con 0x7fe7080cfe70 2024-03-18T16:30:14.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.452+0000 7fe6ff400640 1 -- 172.21.15.102:0/3106113322 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f4006e10 con 0x7fe7080cfe70 2024-03-18T16:30:14.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.785+0000 7fe6ff400640 1 -- 172.21.15.102:0/3106113322 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe6f404b060 con 0x7fe7080cfe70 2024-03-18T16:30:14.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.787+0000 7fe6ff400640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe6e806bbd0 0x7fe6e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:14.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.788+0000 7fe707400640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe6e806bbd0 0x7fe6e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:14.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.788+0000 7fe707400640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe6e806bbd0 0x7fe6e806e090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8000f60 tx=0x7fe6f8007e10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:14.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.789+0000 7fe6ff400640 1 -- 172.21.15.102:0/3106113322 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe6f40791e0 con 0x7fe7080cfe70 2024-03-18T16:30:14.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:14.800+0000 7fe6ff400640 1 -- 172.21.15.102:0/3106113322 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe6f403f070 con 0x7fe7080cfe70 2024-03-18T16:30:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.043+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7080c5510 con 0x7fe7080cfe70 2024-03-18T16:30:15.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.131+0000 7fe6ff400640 1 -- 172.21.15.102:0/3106113322 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe6f407b140 con 0x7fe7080cfe70 2024-03-18T16:30:15.135 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:15.135 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:15.135 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe6e806bbd0 msgr2=0x7fe6e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe6e806bbd0 0x7fe6e806e090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8000f60 tx=0x7fe6f8007e10 comp rx=0 tx=0).stop 2024-03-18T16:30:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080cfe70 msgr2=0x7fe708159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080cfe70 0x7fe708159420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe7080cb020 tx=0x7fe6f40046c0 comp rx=0 tx=0).stop 2024-03-18T16:30:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 shutdown_connections 2024-03-18T16:30:15.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe6e806bbd0 0x7fe6e806e090 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7080d0250 0x7fe7081e0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe7080cfe70 0x7fe708159420 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 --2- 172.21.15.102:0/3106113322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7080cc2f0 0x7fe7081e03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 >> 172.21.15.102:0/3106113322 conn(0x7fe70808c3e0 msgr2=0x7fe7080cc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 shutdown_connections 2024-03-18T16:30:15.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:15.135+0000 7fe70cc00640 1 -- 172.21.15.102:0/3106113322 wait complete. 2024-03-18T16:30:15.767 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:15.767 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:16.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.940+0000 7eff50a00640 1 Processor -- start 2024-03-18T16:30:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.940+0000 7eff50a00640 1 -- start start 2024-03-18T16:30:16.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.941+0000 7eff50a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7eff4c15a310 0x7eff4c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.941+0000 7eff50a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff4c0ca500 con 0x7eff4c15a6f0 2024-03-18T16:30:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.941+0000 7eff50a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff4c0ca7a0 con 0x7eff4c15baa0 2024-03-18T16:30:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.941+0000 7eff50a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7eff4c0cd410 con 0x7eff4c15a310 2024-03-18T16:30:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.942+0000 7eff4a800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7eff4c15a310 0x7eff4c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.942+0000 7eff4b200640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7eff4c15baa0 0x7eff4c15bf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:42152/0 (socket says 172.21.15.102:42152) 2024-03-18T16:30:16.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.942+0000 7eff4b200640 1 -- 172.21.15.102:0/935899072 learned_addr learned my addr 172.21.15.102:0/935899072 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:16.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.945+0000 7eff4a800640 1 -- 172.21.15.102:0/935899072 >> v1:172.21.15.154:6789/0 conn(0x7eff4c15baa0 legacy=0x7eff4c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:16.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.945+0000 7eff4a800640 1 -- 172.21.15.102:0/935899072 >> v1:172.21.15.102:6789/0 conn(0x7eff4c15a6f0 legacy=0x7eff4c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:16.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.945+0000 7eff4a800640 1 -- 172.21.15.102:0/935899072 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4c0caa80 con 0x7eff4c15a310 2024-03-18T16:30:16.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.951+0000 7eff4a800640 1 --2- 172.21.15.102:0/935899072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a310 0x7eff4c06d3e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0097a0 tx=0x7eff3c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6797850e7b2095f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.957+0000 7eff49400640 1 -- 172.21.15.102:0/935899072 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c035020 con 0x7eff4c15a310 2024-03-18T16:30:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.957+0000 7eff49400640 1 -- 172.21.15.102:0/935899072 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3c031830 con 0x7eff4c15a310 2024-03-18T16:30:16.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.957+0000 7eff49400640 1 -- 172.21.15.102:0/935899072 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c031c00 con 0x7eff4c15a310 2024-03-18T16:30:16.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.958+0000 7eff50a00640 1 -- 172.21.15.102:0/935899072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a310 msgr2=0x7eff4c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:16.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.958+0000 7eff50a00640 1 --2- 172.21.15.102:0/935899072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a310 0x7eff4c06d3e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0097a0 tx=0x7eff3c02b8d0 comp rx=0 tx=0).stop 2024-03-18T16:30:16.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.958+0000 7eff50a00640 1 -- 172.21.15.102:0/935899072 shutdown_connections 2024-03-18T16:30:16.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.958+0000 7eff50a00640 1 --2- 172.21.15.102:0/935899072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a310 0x7eff4c06d3e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:16.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.958+0000 7eff50a00640 1 -- 172.21.15.102:0/935899072 >> 172.21.15.102:0/935899072 conn(0x7eff4c08c1a0 msgr2=0x7eff4c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:16.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.958+0000 7eff50a00640 1 -- 172.21.15.102:0/935899072 shutdown_connections 2024-03-18T16:30:16.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.959+0000 7eff50a00640 1 -- 172.21.15.102:0/935899072 wait complete. 2024-03-18T16:30:16.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.960+0000 7eff50a00640 1 Processor -- start 2024-03-18T16:30:16.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.960+0000 7eff50a00640 1 -- start start 2024-03-18T16:30:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.961+0000 7eff50a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff4c15a310 0x7eff4c14c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.961+0000 7eff50a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a6f0 0x7eff4c0c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.961+0000 7eff50a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4c15baa0 0x7eff4c148cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.961+0000 7eff50a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c06d920 con 0x7eff4c15baa0 2024-03-18T16:30:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.961+0000 7eff50a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c0cd130 con 0x7eff4c15a6f0 2024-03-18T16:30:16.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.961+0000 7eff50a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff4c1548c0 con 0x7eff4c15a310 2024-03-18T16:30:16.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.962+0000 7eff4a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff4c15a310 0x7eff4c14c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:16.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.962+0000 7eff49e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a6f0 0x7eff4c0c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:16.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.962+0000 7eff49e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a6f0 0x7eff4c0c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:37712/0 (socket says 172.21.15.102:37712) 2024-03-18T16:30:16.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.962+0000 7eff49e00640 1 -- 172.21.15.102:0/3331510246 learned_addr learned my addr 172.21.15.102:0/3331510246 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:16.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.966+0000 7eff49e00640 1 -- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff4c15a310 msgr2=0x7eff4c14c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:16.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.966+0000 7eff49e00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff4c15a310 0x7eff4c14c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:16.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.966+0000 7eff49e00640 1 -- 172.21.15.102:0/3331510246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4c15baa0 msgr2=0x7eff4c148cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:16.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.966+0000 7eff49e00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4c15baa0 0x7eff4c148cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:16.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.966+0000 7eff49e00640 1 -- 172.21.15.102:0/3331510246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4c14bbf0 con 0x7eff4c15a6f0 2024-03-18T16:30:16.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.968+0000 7eff49e00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a6f0 0x7eff4c0c9620 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c430 tx=0x7eff3800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:16.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.972+0000 7eff42a00640 1 -- 172.21.15.102:0/3331510246 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38017070 con 0x7eff4c15a6f0 2024-03-18T16:30:16.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.972+0000 7eff42a00640 1 -- 172.21.15.102:0/3331510246 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff380035f0 con 0x7eff4c15a6f0 2024-03-18T16:30:16.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.972+0000 7eff42a00640 1 -- 172.21.15.102:0/3331510246 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff380039e0 con 0x7eff4c15a6f0 2024-03-18T16:30:16.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.973+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff4c159ed0 con 0x7eff4c15a6f0 2024-03-18T16:30:16.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.973+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff4c1f4520 con 0x7eff4c15a6f0 2024-03-18T16:30:16.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.974+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff4c1f4280 con 0x7eff4c15a6f0 2024-03-18T16:30:17.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:16.999+0000 7eff42a00640 1 -- 172.21.15.102:0/3331510246 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7eff3801a070 con 0x7eff4c15a6f0 2024-03-18T16:30:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.000+0000 7eff42a00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff2c06bc20 0x7eff2c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.001+0000 7eff4a800640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff2c06bc20 0x7eff2c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.001+0000 7eff4a800640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff2c06bc20 0x7eff2c06e0e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7eff3c036000 tx=0x7eff3c03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.271+0000 7eff42a00640 1 -- 172.21.15.102:0/3331510246 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7eff38056b50 con 0x7eff4c15a6f0 2024-03-18T16:30:17.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.282+0000 7eff42a00640 1 -- 172.21.15.102:0/3331510246 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7eff38058ab0 con 0x7eff4c15a6f0 2024-03-18T16:30:17.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.585+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff4c074fd0 con 0x7eff4c15a6f0 2024-03-18T16:30:17.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.597+0000 7eff42a00640 1 -- 172.21.15.102:0/3331510246 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7eff3805cb50 con 0x7eff4c15a6f0 2024-03-18T16:30:17.600 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:17.601 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:17.601 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff2c06bc20 msgr2=0x7eff2c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff2c06bc20 0x7eff2c06e0e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7eff3c036000 tx=0x7eff3c03c040 comp rx=0 tx=0).stop 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a6f0 msgr2=0x7eff4c0c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a6f0 0x7eff4c0c9620 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c430 tx=0x7eff3800c8f0 comp rx=0 tx=0).stop 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 shutdown_connections 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff2c06bc20 0x7eff2c06e0e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4c15baa0 0x7eff4c148cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff4c15a6f0 0x7eff4c0c9620 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 --2- 172.21.15.102:0/3331510246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff4c15a310 0x7eff4c14c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 >> 172.21.15.102:0/3331510246 conn(0x7eff4c08c1a0 msgr2=0x7eff4c14ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 shutdown_connections 2024-03-18T16:30:17.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:17.600+0000 7eff50a00640 1 -- 172.21.15.102:0/3331510246 wait complete. 2024-03-18T16:30:18.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:18.224 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:19.225 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:19.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.412+0000 7f3d37e00640 1 Processor -- start 2024-03-18T16:30:19.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.412+0000 7f3d37e00640 1 -- start start 2024-03-18T16:30:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.413+0000 7f3d37e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d3815c3a0 0x7f3d38163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.413+0000 7f3d37e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d380c72d0 con 0x7f3d38141650 2024-03-18T16:30:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.413+0000 7f3d37e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d38158840 con 0x7f3d38158420 2024-03-18T16:30:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.413+0000 7f3d37e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3d38159370 con 0x7f3d3815c3a0 2024-03-18T16:30:19.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.414+0000 7f3d36000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3d38158420 0x7f3d380c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:42164/0 (socket says 172.21.15.102:42164) 2024-03-18T16:30:19.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.414+0000 7f3d36000640 1 -- 172.21.15.102:0/2076069192 learned_addr learned my addr 172.21.15.102:0/2076069192 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:19.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.417+0000 7f3d37400640 1 --2- 172.21.15.102:0/2076069192 >> v2:172.21.15.154:3300/0 conn(0x7f3d3815c3a0 0x7f3d38163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:19.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.417+0000 7f3d35600640 1 -- 172.21.15.102:0/2076069192 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3908834945 0 0) 0x7f3d38158840 con 0x7f3d38158420 2024-03-18T16:30:19.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.418+0000 7f3d35600640 1 -- 172.21.15.102:0/2076069192 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d1c003520 con 0x7f3d38158420 2024-03-18T16:30:19.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.418+0000 7f3d35600640 1 -- 172.21.15.102:0/2076069192 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2076327711 0 0) 0x7f3d380c72d0 con 0x7f3d38141650 2024-03-18T16:30:19.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.418+0000 7f3d35600640 1 -- 172.21.15.102:0/2076069192 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d38158840 con 0x7f3d38141650 2024-03-18T16:30:19.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.421+0000 7f3d37400640 1 -- 172.21.15.102:0/2076069192 >> v1:172.21.15.154:6789/0 conn(0x7f3d38158420 legacy=0x7f3d380c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:19.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.421+0000 7f3d37400640 1 -- 172.21.15.102:0/2076069192 >> v1:172.21.15.102:6789/0 conn(0x7f3d38141650 legacy=0x7f3d380c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:19.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.421+0000 7f3d37400640 1 -- 172.21.15.102:0/2076069192 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d38158ad0 con 0x7f3d3815c3a0 2024-03-18T16:30:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.428+0000 7f3d37400640 1 --2- 172.21.15.102:0/2076069192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d3815c3a0 0x7f3d38163cd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009d90 tx=0x7f3d3002bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc1093a4985b8eeb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:19.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.432+0000 7f3d35600640 1 -- 172.21.15.102:0/2076069192 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30039070 con 0x7f3d3815c3a0 2024-03-18T16:30:19.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.432+0000 7f3d35600640 1 -- 172.21.15.102:0/2076069192 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3002ccf0 con 0x7f3d3815c3a0 2024-03-18T16:30:19.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.432+0000 7f3d35600640 1 -- 172.21.15.102:0/2076069192 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30034660 con 0x7f3d3815c3a0 2024-03-18T16:30:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.433+0000 7f3d37e00640 1 -- 172.21.15.102:0/2076069192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d3815c3a0 msgr2=0x7f3d38163cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.433+0000 7f3d37e00640 1 --2- 172.21.15.102:0/2076069192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d3815c3a0 0x7f3d38163cd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009d90 tx=0x7f3d3002bf40 comp rx=0 tx=0).stop 2024-03-18T16:30:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.433+0000 7f3d37e00640 1 -- 172.21.15.102:0/2076069192 shutdown_connections 2024-03-18T16:30:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.433+0000 7f3d37e00640 1 --2- 172.21.15.102:0/2076069192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d3815c3a0 0x7f3d38163cd0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.433+0000 7f3d37e00640 1 -- 172.21.15.102:0/2076069192 >> 172.21.15.102:0/2076069192 conn(0x7f3d380cb890 msgr2=0x7f3d380cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.433+0000 7f3d37e00640 1 -- 172.21.15.102:0/2076069192 shutdown_connections 2024-03-18T16:30:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.433+0000 7f3d37e00640 1 -- 172.21.15.102:0/2076069192 wait complete. 2024-03-18T16:30:19.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.435+0000 7f3d37e00640 1 Processor -- start 2024-03-18T16:30:19.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.435+0000 7f3d37e00640 1 -- start start 2024-03-18T16:30:19.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.436+0000 7f3d37e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d38141650 0x7f3d38153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:19.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.436+0000 7f3d37e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d38158420 0x7f3d3815aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:19.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.436+0000 7f3d37e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d3815c3a0 0x7f3d381e2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:19.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.436+0000 7f3d37e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3d380c63c0 con 0x7f3d38141650 2024-03-18T16:30:19.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.436+0000 7f3d37e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d38159090 con 0x7f3d38158420 2024-03-18T16:30:19.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.436+0000 7f3d37e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3d380c66a0 con 0x7f3d3815c3a0 2024-03-18T16:30:19.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.437+0000 7f3d36000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d38158420 0x7f3d3815aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:19.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.437+0000 7f3d37400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d3815c3a0 0x7f3d381e2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:19.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.437+0000 7f3d36a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d38141650 0x7f3d38153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:19.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.437+0000 7f3d36000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d38158420 0x7f3d3815aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:37726/0 (socket says 172.21.15.102:37726) 2024-03-18T16:30:19.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.437+0000 7f3d36000640 1 -- 172.21.15.102:0/3025125333 learned_addr learned my addr 172.21.15.102:0/3025125333 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:19.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.440+0000 7f3d37400640 1 -- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d38158420 msgr2=0x7f3d3815aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:19.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.440+0000 7f3d37400640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d38158420 0x7f3d3815aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:19.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.440+0000 7f3d37400640 1 -- 172.21.15.102:0/3025125333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d38141650 msgr2=0x7f3d38153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:19.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.440+0000 7f3d37400640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d38141650 0x7f3d38153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:19.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.440+0000 7f3d37400640 1 -- 172.21.15.102:0/3025125333 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d380c21c0 con 0x7f3d3815c3a0 2024-03-18T16:30:19.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.443+0000 7f3d37400640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d3815c3a0 0x7f3d381e2740 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3d30033000 tx=0x7f3d30004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.446+0000 7f3d2ea00640 1 -- 172.21.15.102:0/3025125333 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30039050 con 0x7f3d3815c3a0 2024-03-18T16:30:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.446+0000 7f3d2ea00640 1 -- 172.21.15.102:0/3025125333 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d30045070 con 0x7f3d3815c3a0 2024-03-18T16:30:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.446+0000 7f3d2ea00640 1 -- 172.21.15.102:0/3025125333 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30009450 con 0x7f3d3815c3a0 2024-03-18T16:30:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.446+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d381ee2b0 con 0x7f3d3815c3a0 2024-03-18T16:30:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.447+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d380c2480 con 0x7f3d3815c3a0 2024-03-18T16:30:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.448+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3806c010 con 0x7f3d3815c3a0 2024-03-18T16:30:19.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.705+0000 7f3d2ea00640 1 -- 172.21.15.102:0/3025125333 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3d30003990 con 0x7f3d3815c3a0 2024-03-18T16:30:19.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.707+0000 7f3d2ea00640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d1806bc50 0x7f3d1806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:19.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.707+0000 7f3d2ea00640 1 -- 172.21.15.102:0/3025125333 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3d3007a480 con 0x7f3d3815c3a0 2024-03-18T16:30:19.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.707+0000 7f3d36a00640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d1806bc50 0x7f3d1806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:19.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.709+0000 7f3d36a00640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d1806bc50 0x7f3d1806e110 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3d28007fd0 tx=0x7f3d28009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:19.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:19.721+0000 7f3d2ea00640 1 -- 172.21.15.102:0/3025125333 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3d3806c010 con 0x7f3d3815c3a0 2024-03-18T16:30:20.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.007+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d3815c810 con 0x7f3d3815c3a0 2024-03-18T16:30:20.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.032+0000 7f3d2ea00640 1 -- 172.21.15.102:0/3025125333 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3d3815c810 con 0x7f3d3815c3a0 2024-03-18T16:30:20.035 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:20.035 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:20.035 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:20.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d1806bc50 msgr2=0x7f3d1806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:20.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d1806bc50 0x7f3d1806e110 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3d28007fd0 tx=0x7f3d28009040 comp rx=0 tx=0).stop 2024-03-18T16:30:20.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d3815c3a0 msgr2=0x7f3d381e2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:20.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d3815c3a0 0x7f3d381e2740 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3d30033000 tx=0x7f3d30004a40 comp rx=0 tx=0).stop 2024-03-18T16:30:20.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 shutdown_connections 2024-03-18T16:30:20.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d1806bc50 0x7f3d1806e110 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:20.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d3815c3a0 0x7f3d381e2740 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:20.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d38158420 0x7f3d3815aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:20.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 --2- 172.21.15.102:0/3025125333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d38141650 0x7f3d38153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:20.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 >> 172.21.15.102:0/3025125333 conn(0x7f3d380cb890 msgr2=0x7f3d38157410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:20.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 shutdown_connections 2024-03-18T16:30:20.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:20.036+0000 7f3d37e00640 1 -- 172.21.15.102:0/3025125333 wait complete. 2024-03-18T16:30:20.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:20.669 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:21.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:21.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.846+0000 7f14f4600640 1 Processor -- start 2024-03-18T16:30:21.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.846+0000 7f14f4600640 1 -- start start 2024-03-18T16:30:21.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.847+0000 7f14f4600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f14ec1524c0 0x7f14ec072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:21.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.847+0000 7f14f4600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14ec15b8e0 0x7f14ec0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:21.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.847+0000 7f14f4600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14ec15c200 con 0x7f14ec158ee0 2024-03-18T16:30:21.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.847+0000 7f14f4600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f14ec159300 con 0x7f14ec15b8e0 2024-03-18T16:30:21.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.847+0000 7f14f4600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f14ec15c4e0 con 0x7f14ec1524c0 2024-03-18T16:30:21.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.848+0000 7f14f1c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f14ec1524c0 0x7f14ec072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:21.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.848+0000 7f14f2600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14ec15b8e0 0x7f14ec0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:21.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.849+0000 7f14f1c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f14ec1524c0 0x7f14ec072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:45514/0 (socket says 172.21.15.102:45514) 2024-03-18T16:30:21.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.849+0000 7f14f1c00640 1 -- 172.21.15.102:0/1048434978 learned_addr learned my addr 172.21.15.102:0/1048434978 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:21.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.852+0000 7f14f1c00640 1 -- 172.21.15.102:0/1048434978 >> v2:172.21.15.102:3300/0 conn(0x7f14ec15b8e0 msgr2=0x7f14ec0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:21.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.852+0000 7f14f1c00640 1 --2- 172.21.15.102:0/1048434978 >> v2:172.21.15.102:3300/0 conn(0x7f14ec15b8e0 0x7f14ec0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:21.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.852+0000 7f14f1c00640 1 -- 172.21.15.102:0/1048434978 >> v1:172.21.15.102:6789/0 conn(0x7f14ec158ee0 legacy=0x7f14ec06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:21.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.852+0000 7f14f1c00640 1 -- 172.21.15.102:0/1048434978 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ec15c7c0 con 0x7f14ec1524c0 2024-03-18T16:30:21.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.855+0000 7f14f1c00640 1 --2- 172.21.15.102:0/1048434978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 0x7f14ec072700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f14ec06bf90 tx=0x7f14e002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fa33bea2cd9b9fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:21.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.860+0000 7f14ebe00640 1 -- 172.21.15.102:0/1048434978 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0034070 con 0x7f14ec1524c0 2024-03-18T16:30:21.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.860+0000 7f14ebe00640 1 -- 172.21.15.102:0/1048434978 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e0030860 con 0x7f14ec1524c0 2024-03-18T16:30:21.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.860+0000 7f14ebe00640 1 -- 172.21.15.102:0/1048434978 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0030c30 con 0x7f14ec1524c0 2024-03-18T16:30:21.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.861+0000 7f14f4600640 1 -- 172.21.15.102:0/1048434978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 msgr2=0x7f14ec072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:21.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.861+0000 7f14f4600640 1 --2- 172.21.15.102:0/1048434978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 0x7f14ec072700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f14ec06bf90 tx=0x7f14e002b9a0 comp rx=0 tx=0).stop 2024-03-18T16:30:21.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.861+0000 7f14f4600640 1 -- 172.21.15.102:0/1048434978 shutdown_connections 2024-03-18T16:30:21.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.861+0000 7f14f4600640 1 --2- 172.21.15.102:0/1048434978 >> v2:172.21.15.102:3300/0 conn(0x7f14ec15b8e0 0x7f14ec0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:21.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.861+0000 7f14f4600640 1 --2- 172.21.15.102:0/1048434978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 0x7f14ec072700 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:21.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.861+0000 7f14f4600640 1 -- 172.21.15.102:0/1048434978 >> 172.21.15.102:0/1048434978 conn(0x7f14ec0700d0 msgr2=0x7f14ec06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:21.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.861+0000 7f14f4600640 1 -- 172.21.15.102:0/1048434978 shutdown_connections 2024-03-18T16:30:21.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.861+0000 7f14f4600640 1 -- 172.21.15.102:0/1048434978 wait complete. 2024-03-18T16:30:21.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.863+0000 7f14f4600640 1 Processor -- start 2024-03-18T16:30:21.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.863+0000 7f14f4600640 1 -- start start 2024-03-18T16:30:21.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.864+0000 7f14f4600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 0x7f14ec0ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:21.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.864+0000 7f14f4600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14ec158ee0 0x7f14ec0af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:21.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.864+0000 7f14f4600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14ec15b8e0 0x7f14ec153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:21.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.864+0000 7f14f4600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec1589c0 con 0x7f14ec158ee0 2024-03-18T16:30:21.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.864+0000 7f14f4600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec1586e0 con 0x7f14ec1524c0 2024-03-18T16:30:21.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.864+0000 7f14f4600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f14ec1528a0 con 0x7f14ec15b8e0 2024-03-18T16:30:21.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.864+0000 7f14f1c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 0x7f14ec0ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:21.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.865+0000 7f14f2600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14ec15b8e0 0x7f14ec153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:21.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.865+0000 7f14f1c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 0x7f14ec0ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:45518/0 (socket says 172.21.15.102:45518) 2024-03-18T16:30:21.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.865+0000 7f14f1c00640 1 -- 172.21.15.102:0/3428730915 learned_addr learned my addr 172.21.15.102:0/3428730915 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:21.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.865+0000 7f14f1200640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14ec158ee0 0x7f14ec0af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:21.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.868+0000 7f14f1c00640 1 -- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14ec15b8e0 msgr2=0x7f14ec153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:21.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.868+0000 7f14f1c00640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14ec15b8e0 0x7f14ec153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:21.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.868+0000 7f14f1c00640 1 -- 172.21.15.102:0/3428730915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14ec158ee0 msgr2=0x7f14ec0af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:21.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.868+0000 7f14f1c00640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14ec158ee0 0x7f14ec0af570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:21.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.868+0000 7f14f1c00640 1 -- 172.21.15.102:0/3428730915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ec0ae880 con 0x7f14ec1524c0 2024-03-18T16:30:21.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.870+0000 7f14f1c00640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 0x7f14ec0ae2a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009960 tx=0x7f14e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.874+0000 7f14e9600640 1 -- 172.21.15.102:0/3428730915 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0034040 con 0x7f14ec1524c0 2024-03-18T16:30:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.874+0000 7f14e9600640 1 -- 172.21.15.102:0/3428730915 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e0039d90 con 0x7f14ec1524c0 2024-03-18T16:30:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.874+0000 7f14e9600640 1 -- 172.21.15.102:0/3428730915 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e0043610 con 0x7f14ec1524c0 2024-03-18T16:30:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.874+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14ec0aeb60 con 0x7f14ec1524c0 2024-03-18T16:30:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.874+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14ec1f3310 con 0x7f14ec1524c0 2024-03-18T16:30:21.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.876+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14ec074fd0 con 0x7f14ec1524c0 2024-03-18T16:30:21.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.895+0000 7f14e9600640 1 -- 172.21.15.102:0/3428730915 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f14e0037070 con 0x7f14ec1524c0 2024-03-18T16:30:21.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.897+0000 7f14e9600640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f14cc06bbd0 0x7f14cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:21.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.897+0000 7f14e9600640 1 -- 172.21.15.102:0/3428730915 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f14e00ad690 con 0x7f14ec1524c0 2024-03-18T16:30:21.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.897+0000 7f14f1200640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f14cc06bbd0 0x7f14cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:21.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:21.898+0000 7f14f1200640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f14cc06bbd0 0x7f14cc06e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f14ec068690 tx=0x7f14d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:22.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.155+0000 7f14e9600640 1 -- 172.21.15.102:0/3428730915 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f14e003e070 con 0x7f14ec1524c0 2024-03-18T16:30:22.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.395+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14ec06c590 con 0x7f14ec1524c0 2024-03-18T16:30:22.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.408+0000 7f14e9600640 1 -- 172.21.15.102:0/3428730915 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f14e007b7b0 con 0x7f14ec1524c0 2024-03-18T16:30:22.410 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:22.410 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:22.410 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f14cc06bbd0 msgr2=0x7f14cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f14cc06bbd0 0x7f14cc06e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f14ec068690 tx=0x7f14d8009040 comp rx=0 tx=0).stop 2024-03-18T16:30:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 msgr2=0x7f14ec0ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 0x7f14ec0ae2a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009960 tx=0x7f14e00046c0 comp rx=0 tx=0).stop 2024-03-18T16:30:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 shutdown_connections 2024-03-18T16:30:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f14cc06bbd0 0x7f14cc06e090 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14ec15b8e0 0x7f14ec153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14ec158ee0 0x7f14ec0af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:22.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 --2- 172.21.15.102:0/3428730915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ec1524c0 0x7f14ec0ae2a0 secure :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f14e0009960 tx=0x7f14e00046c0 comp rx=0 tx=0).stop 2024-03-18T16:30:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 >> 172.21.15.102:0/3428730915 conn(0x7f14ec0700d0 msgr2=0x7f14ec06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 shutdown_connections 2024-03-18T16:30:22.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:22.411+0000 7f14f4600640 1 -- 172.21.15.102:0/3428730915 wait complete. 2024-03-18T16:30:22.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:30:22.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:30:23.032 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:23.032 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:24.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:24.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.207+0000 7f3d45a00640 1 Processor -- start 2024-03-18T16:30:24.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.207+0000 7f3d45a00640 1 -- start start 2024-03-18T16:30:24.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.208+0000 7f3d45a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d4015c3a0 0x7f3d40163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:24.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.208+0000 7f3d45a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d400c72d0 con 0x7f3d40158420 2024-03-18T16:30:24.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.208+0000 7f3d45a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d40158840 con 0x7f3d40141650 2024-03-18T16:30:24.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.208+0000 7f3d45a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3d40159370 con 0x7f3d4015c3a0 2024-03-18T16:30:24.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.209+0000 7f3d3fe00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3d40141650 0x7f3d400c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:56682/0 (socket says 172.21.15.102:56682) 2024-03-18T16:30:24.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.209+0000 7f3d3fe00640 1 -- 172.21.15.102:0/3529279968 learned_addr learned my addr 172.21.15.102:0/3529279968 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:24.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.212+0000 7f3d45000640 1 --2- 172.21.15.102:0/3529279968 >> v2:172.21.15.154:3300/0 conn(0x7f3d4015c3a0 0x7f3d40163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:24.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.212+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229604611 0 0) 0x7f3d40158840 con 0x7f3d40141650 2024-03-18T16:30:24.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.212+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d24003520 con 0x7f3d40141650 2024-03-18T16:30:24.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.212+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1931706383 0 0) 0x7f3d400c72d0 con 0x7f3d40158420 2024-03-18T16:30:24.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.212+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d40158840 con 0x7f3d40158420 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1081745954 0 0) 0x7f3d24003520 con 0x7f3d40141650 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 >> v2:172.21.15.154:3300/0 conn(0x7f3d4015c3a0 msgr2=0x7f3d40163cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 --2- 172.21.15.102:0/3529279968 >> v2:172.21.15.154:3300/0 conn(0x7f3d4015c3a0 0x7f3d40163cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 >> v1:172.21.15.102:6789/0 conn(0x7f3d40158420 legacy=0x7f3d400c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d40158ad0 con 0x7f3d40141650 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f3d30001670 con 0x7f3d40141650 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 >> v1:172.21.15.154:6789/0 conn(0x7f3d40141650 legacy=0x7f3d400c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d24003b40 0x7f3d24003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d24004800 0x7f3d24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d240051c0 0x7f3d2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:24.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3d40159370 con 0x7f3d240051c0 2024-03-18T16:30:24.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2400c0f0 con 0x7f3d24003b40 2024-03-18T16:30:24.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.217+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2400c2f0 con 0x7f3d24004800 2024-03-18T16:30:24.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.218+0000 7f3d3f400640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d24004800 0x7f3d24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:24.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.219+0000 7f3d45000640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d240051c0 0x7f3d2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:24.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.220+0000 7f3d3fe00640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d24003b40 0x7f3d24003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:24.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.222+0000 7f3d3fe00640 1 -- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d24004800 msgr2=0x7f3d24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:24.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.222+0000 7f3d3fe00640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d24004800 0x7f3d24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.222+0000 7f3d3fe00640 1 -- 172.21.15.102:0/3529279968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d240051c0 msgr2=0x7f3d2400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:24.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.222+0000 7f3d3fe00640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d240051c0 0x7f3d2400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.222+0000 7f3d3fe00640 1 -- 172.21.15.102:0/3529279968 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d40158ad0 con 0x7f3d24003b40 2024-03-18T16:30:24.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.222+0000 7f3d45000640 1 -- 172.21.15.102:0/3529279968 reap_dead start 2024-03-18T16:30:24.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.225+0000 7f3d3fe00640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d24003b40 0x7f3d24003f40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000c260 tx=0x7f3d24005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8e972da4944820f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:24.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.229+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30007d10 con 0x7f3d24003b40 2024-03-18T16:30:24.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.229+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d30030070 con 0x7f3d24003b40 2024-03-18T16:30:24.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.229+0000 7f3d3ea00640 1 -- 172.21.15.102:0/3529279968 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30005d10 con 0x7f3d24003b40 2024-03-18T16:30:24.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.230+0000 7f3d45a00640 1 -- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d24003b40 msgr2=0x7f3d24003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:24.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.230+0000 7f3d45a00640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d24003b40 0x7f3d24003f40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000c260 tx=0x7f3d24005830 comp rx=0 tx=0).stop 2024-03-18T16:30:24.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.230+0000 7f3d45a00640 1 -- 172.21.15.102:0/3529279968 shutdown_connections 2024-03-18T16:30:24.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.230+0000 7f3d45a00640 1 --2- 172.21.15.102:0/3529279968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d24003b40 0x7f3d24003f40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.230+0000 7f3d45a00640 1 -- 172.21.15.102:0/3529279968 >> 172.21.15.102:0/3529279968 conn(0x7f3d400cb890 msgr2=0x7f3d400cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:24.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.230+0000 7f3d45a00640 1 -- 172.21.15.102:0/3529279968 shutdown_connections 2024-03-18T16:30:24.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.231+0000 7f3d45a00640 1 -- 172.21.15.102:0/3529279968 wait complete. 2024-03-18T16:30:24.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.232+0000 7f3d45a00640 1 Processor -- start 2024-03-18T16:30:24.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.232+0000 7f3d45a00640 1 -- start start 2024-03-18T16:30:24.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.233+0000 7f3d45a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d24003b40 0x7f3d401386c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:24.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.233+0000 7f3d45a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d400c9c50 0x7f3d401567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:24.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.233+0000 7f3d45a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d4015b3d0 0x7f3d4013e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:24.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.233+0000 7f3d45a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3d400c63c0 con 0x7f3d24003b40 2024-03-18T16:30:24.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.233+0000 7f3d45a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d40159090 con 0x7f3d400c9c50 2024-03-18T16:30:24.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.233+0000 7f3d45a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3d400c66a0 con 0x7f3d4015b3d0 2024-03-18T16:30:24.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.234+0000 7f3d3fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d24003b40 0x7f3d401386c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:24.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.234+0000 7f3d3f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d400c9c50 0x7f3d401567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:24.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.234+0000 7f3d45000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d4015b3d0 0x7f3d4013e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:24.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.235+0000 7f3d45000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d4015b3d0 0x7f3d4013e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41606/0 (socket says 172.21.15.102:41606) 2024-03-18T16:30:24.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.235+0000 7f3d45000640 1 -- 172.21.15.102:0/2154004966 learned_addr learned my addr 172.21.15.102:0/2154004966 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.238+0000 7f3d45000640 1 -- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d400c9c50 msgr2=0x7f3d401567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.238+0000 7f3d45000640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d400c9c50 0x7f3d401567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.238+0000 7f3d45000640 1 -- 172.21.15.102:0/2154004966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d24003b40 msgr2=0x7f3d401386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.238+0000 7f3d45000640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d24003b40 0x7f3d401386c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.238+0000 7f3d45000640 1 -- 172.21.15.102:0/2154004966 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d40138c00 con 0x7f3d4015b3d0 2024-03-18T16:30:24.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.240+0000 7f3d45000640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d4015b3d0 0x7f3d4013e360 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3d40067fa0 tx=0x7f3d38003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.242+0000 7f3d37e00640 1 -- 172.21.15.102:0/2154004966 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38007b70 con 0x7f3d4015b3d0 2024-03-18T16:30:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.242+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d40156cf0 con 0x7f3d4015b3d0 2024-03-18T16:30:24.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.243+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d4015b950 con 0x7f3d4015b3d0 2024-03-18T16:30:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.244+0000 7f3d37e00640 1 -- 172.21.15.102:0/2154004966 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3801a070 con 0x7f3d4015b3d0 2024-03-18T16:30:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.244+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d40067c20 con 0x7f3d4015b3d0 2024-03-18T16:30:24.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.244+0000 7f3d37e00640 1 -- 172.21.15.102:0/2154004966 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3801f460 con 0x7f3d4015b3d0 2024-03-18T16:30:24.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.515+0000 7f3d37e00640 1 -- 172.21.15.102:0/2154004966 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3d38015420 con 0x7f3d4015b3d0 2024-03-18T16:30:24.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.517+0000 7f3d37e00640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d2006bbd0 0x7f3d2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:24.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.518+0000 7f3d3fe00640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d2006bbd0 0x7f3d2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:24.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.518+0000 7f3d3fe00640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d2006bbd0 0x7f3d2006e090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3d30034000 tx=0x7f3d30009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:24.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.521+0000 7f3d37e00640 1 -- 172.21.15.102:0/2154004966 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3d38058430 con 0x7f3d4015b3d0 2024-03-18T16:30:24.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.539+0000 7f3d37e00640 1 -- 172.21.15.102:0/2154004966 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3d40067c20 con 0x7f3d4015b3d0 2024-03-18T16:30:24.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.776+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d4015c810 con 0x7f3d4015b3d0 2024-03-18T16:30:24.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.790+0000 7f3d37e00640 1 -- 172.21.15.102:0/2154004966 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3d4015c810 con 0x7f3d4015b3d0 2024-03-18T16:30:24.791 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:24.792 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:24.792 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d2006bbd0 msgr2=0x7f3d2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d2006bbd0 0x7f3d2006e090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3d30034000 tx=0x7f3d30009030 comp rx=0 tx=0).stop 2024-03-18T16:30:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d4015b3d0 msgr2=0x7f3d4013e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d4015b3d0 0x7f3d4013e360 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3d40067fa0 tx=0x7f3d38003eb0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 shutdown_connections 2024-03-18T16:30:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d2006bbd0 0x7f3d2006e090 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d4015b3d0 0x7f3d4013e360 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d400c9c50 0x7f3d401567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 --2- 172.21.15.102:0/2154004966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d24003b40 0x7f3d401386c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 >> 172.21.15.102:0/2154004966 conn(0x7f3d400cb890 msgr2=0x7f3d40157c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 shutdown_connections 2024-03-18T16:30:24.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:24.792+0000 7f3d45a00640 1 -- 172.21.15.102:0/2154004966 wait complete. 2024-03-18T16:30:25.415 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:25.415 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:26.416 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:26.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.588+0000 7f9d4a200640 1 Processor -- start 2024-03-18T16:30:26.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.588+0000 7f9d4a200640 1 -- start start 2024-03-18T16:30:26.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.589+0000 7f9d4a200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9d44158420 0x7f9d4415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:26.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.589+0000 7f9d4a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d440c72d0 con 0x7f9d440c4fa0 2024-03-18T16:30:26.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.589+0000 7f9d4a200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d44158840 con 0x7f9d44141650 2024-03-18T16:30:26.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.589+0000 7f9d4a200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9d4415c8e0 con 0x7f9d44158420 2024-03-18T16:30:26.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.590+0000 7f9d49800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f9d440c4fa0 0x7f9d44162cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53852/0 (socket says 172.21.15.102:53852) 2024-03-18T16:30:26.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.590+0000 7f9d49800640 1 -- 172.21.15.102:0/1898741079 learned_addr learned my addr 172.21.15.102:0/1898741079 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:26.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.590+0000 7f9d43e00640 1 --2- 172.21.15.102:0/1898741079 >> v2:172.21.15.154:3300/0 conn(0x7f9d44158420 0x7f9d4415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:26.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.594+0000 7f9d43400640 1 -- 172.21.15.102:0/1898741079 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2774727671 0 0) 0x7f9d440c72d0 con 0x7f9d440c4fa0 2024-03-18T16:30:26.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.594+0000 7f9d43400640 1 -- 172.21.15.102:0/1898741079 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d2c003520 con 0x7f9d440c4fa0 2024-03-18T16:30:26.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.594+0000 7f9d43e00640 1 -- 172.21.15.102:0/1898741079 >> v1:172.21.15.154:6789/0 conn(0x7f9d44141650 legacy=0x7f9d440c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:26.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.594+0000 7f9d43e00640 1 -- 172.21.15.102:0/1898741079 >> v1:172.21.15.102:6789/0 conn(0x7f9d440c4fa0 legacy=0x7f9d44162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:26.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.594+0000 7f9d43e00640 1 -- 172.21.15.102:0/1898741079 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d44158ad0 con 0x7f9d44158420 2024-03-18T16:30:26.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.598+0000 7f9d43e00640 1 --2- 172.21.15.102:0/1898741079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d44158420 0x7f9d4415c3a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9d300097a0 tx=0x7f9d3002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edd06623e285b87b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:26.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.603+0000 7f9d43400640 1 -- 172.21.15.102:0/1898741079 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d30034070 con 0x7f9d44158420 2024-03-18T16:30:26.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.603+0000 7f9d43400640 1 -- 172.21.15.102:0/1898741079 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d30030610 con 0x7f9d44158420 2024-03-18T16:30:26.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.603+0000 7f9d43400640 1 -- 172.21.15.102:0/1898741079 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d300309e0 con 0x7f9d44158420 2024-03-18T16:30:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.604+0000 7f9d4a200640 1 -- 172.21.15.102:0/1898741079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d44158420 msgr2=0x7f9d4415c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.604+0000 7f9d4a200640 1 --2- 172.21.15.102:0/1898741079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d44158420 0x7f9d4415c3a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9d300097a0 tx=0x7f9d3002b960 comp rx=0 tx=0).stop 2024-03-18T16:30:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.604+0000 7f9d4a200640 1 -- 172.21.15.102:0/1898741079 shutdown_connections 2024-03-18T16:30:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.604+0000 7f9d4a200640 1 --2- 172.21.15.102:0/1898741079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d44158420 0x7f9d4415c3a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.604+0000 7f9d4a200640 1 -- 172.21.15.102:0/1898741079 >> 172.21.15.102:0/1898741079 conn(0x7f9d440cb890 msgr2=0x7f9d440cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.604+0000 7f9d4a200640 1 -- 172.21.15.102:0/1898741079 shutdown_connections 2024-03-18T16:30:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.604+0000 7f9d4a200640 1 -- 172.21.15.102:0/1898741079 wait complete. 2024-03-18T16:30:26.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.607+0000 7f9d4a200640 1 Processor -- start 2024-03-18T16:30:26.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.607+0000 7f9d4a200640 1 -- start start 2024-03-18T16:30:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.608+0000 7f9d4a200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d440c4fa0 0x7f9d4413e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.608+0000 7f9d4a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d44141650 0x7f9d44152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.608+0000 7f9d4a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d44158420 0x7f9d4413feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.608+0000 7f9d4a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9d44159090 con 0x7f9d44158420 2024-03-18T16:30:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.608+0000 7f9d4a200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d44159370 con 0x7f9d440c4fa0 2024-03-18T16:30:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.608+0000 7f9d4a200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9d440c63c0 con 0x7f9d44141650 2024-03-18T16:30:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.609+0000 7f9d43e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d44141650 0x7f9d44152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.609+0000 7f9d48e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d440c4fa0 0x7f9d4413e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.609+0000 7f9d49800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d44158420 0x7f9d4413feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.609+0000 7f9d48e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d440c4fa0 0x7f9d4413e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:45560/0 (socket says 172.21.15.102:45560) 2024-03-18T16:30:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.609+0000 7f9d43e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d44141650 0x7f9d44152d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41620/0 (socket says 172.21.15.102:41620) 2024-03-18T16:30:26.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.609+0000 7f9d48e00640 1 -- 172.21.15.102:0/395707208 learned_addr learned my addr 172.21.15.102:0/395707208 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:26.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.613+0000 7f9d43e00640 1 -- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d440c4fa0 msgr2=0x7f9d4413e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.613+0000 7f9d43e00640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d440c4fa0 0x7f9d4413e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.613+0000 7f9d43e00640 1 -- 172.21.15.102:0/395707208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d44158420 msgr2=0x7f9d4413feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.613+0000 7f9d43e00640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d44158420 0x7f9d4413feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.613+0000 7f9d43e00640 1 -- 172.21.15.102:0/395707208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d441e1f30 con 0x7f9d44141650 2024-03-18T16:30:26.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.616+0000 7f9d43e00640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d44141650 0x7f9d44152d90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9d30033000 tx=0x7f9d300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.619+0000 7f9d40c00640 1 -- 172.21.15.102:0/395707208 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d30034040 con 0x7f9d44141650 2024-03-18T16:30:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.619+0000 7f9d40c00640 1 -- 172.21.15.102:0/395707208 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d3003aaa0 con 0x7f9d44141650 2024-03-18T16:30:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.619+0000 7f9d40c00640 1 -- 172.21.15.102:0/395707208 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d300443f0 con 0x7f9d44141650 2024-03-18T16:30:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.619+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d441e1c70 con 0x7f9d44141650 2024-03-18T16:30:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.620+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d44154470 con 0x7f9d44141650 2024-03-18T16:30:26.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.621+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d44074fd0 con 0x7f9d44141650 2024-03-18T16:30:26.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.889+0000 7f9d40c00640 1 -- 172.21.15.102:0/395707208 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9d3004b060 con 0x7f9d44141650 2024-03-18T16:30:26.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.890+0000 7f9d40c00640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d2406bc20 0x7f9d2406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:26.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.891+0000 7f9d48e00640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d2406bc20 0x7f9d2406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:26.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.892+0000 7f9d48e00640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d2406bc20 0x7f9d2406e0e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9d38001840 tx=0x7f9d38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.893+0000 7f9d40c00640 1 -- 172.21.15.102:0/395707208 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9d30079190 con 0x7f9d44141650 2024-03-18T16:30:26.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:26.905+0000 7f9d40c00640 1 -- 172.21.15.102:0/395707208 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9d3003f070 con 0x7f9d44141650 2024-03-18T16:30:27.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.209+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d440b7f00 con 0x7f9d44141650 2024-03-18T16:30:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.269+0000 7f9d40c00640 1 -- 172.21.15.102:0/395707208 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f9d3007b0f0 con 0x7f9d44141650 2024-03-18T16:30:27.271 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:27.271 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:27.271 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d2406bc20 msgr2=0x7f9d2406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d2406bc20 0x7f9d2406e0e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9d38001840 tx=0x7f9d38009040 comp rx=0 tx=0).stop 2024-03-18T16:30:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d44141650 msgr2=0x7f9d44152d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d44141650 0x7f9d44152d90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9d30033000 tx=0x7f9d300046c0 comp rx=0 tx=0).stop 2024-03-18T16:30:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 shutdown_connections 2024-03-18T16:30:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d2406bc20 0x7f9d2406e0e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d44158420 0x7f9d4413feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:27.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d44141650 0x7f9d44152d90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:27.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 --2- 172.21.15.102:0/395707208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d440c4fa0 0x7f9d4413e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:27.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 >> 172.21.15.102:0/395707208 conn(0x7f9d440cb890 msgr2=0x7f9d440cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:27.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 shutdown_connections 2024-03-18T16:30:27.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:27.273+0000 7f9d4a200640 1 -- 172.21.15.102:0/395707208 wait complete. 2024-03-18T16:30:27.903 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:27.904 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:28.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:29.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.096+0000 7fc9b6400640 1 Processor -- start 2024-03-18T16:30:29.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.096+0000 7fc9b6400640 1 -- start start 2024-03-18T16:30:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.098+0000 7fc9b6400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9b015a1a0 0x7fc9b006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.098+0000 7fc9b6400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b006c9c0 con 0x7fc9b00ca960 2024-03-18T16:30:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.098+0000 7fc9b6400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9b006cca0 con 0x7fc9b00cad40 2024-03-18T16:30:29.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.098+0000 7fc9b6400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc9b00c8680 con 0x7fc9b015a1a0 2024-03-18T16:30:29.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.098+0000 7fc9b5000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc9b00ca960 0x7fc9b015e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53868/0 (socket says 172.21.15.102:53868) 2024-03-18T16:30:29.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.098+0000 7fc9b5000640 1 -- 172.21.15.102:0/4041917257 learned_addr learned my addr 172.21.15.102:0/4041917257 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:29.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.099+0000 7fc9b5a00640 1 --2- 172.21.15.102:0/4041917257 >> v2:172.21.15.154:3300/0 conn(0x7fc9b015a1a0 0x7fc9b006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:29.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.102+0000 7fc9b5a00640 1 -- 172.21.15.102:0/4041917257 >> v1:172.21.15.154:6789/0 conn(0x7fc9b00cad40 legacy=0x7fc9b00705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:29.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.102+0000 7fc9b5a00640 1 -- 172.21.15.102:0/4041917257 >> v1:172.21.15.102:6789/0 conn(0x7fc9b00ca960 legacy=0x7fc9b015e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:29.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.102+0000 7fc9b5a00640 1 -- 172.21.15.102:0/4041917257 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b006cf80 con 0x7fc9b015a1a0 2024-03-18T16:30:29.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.107+0000 7fc9b5a00640 1 --2- 172.21.15.102:0/4041917257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 0x7fc9b006c310 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009d90 tx=0x7fc9a802bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76d85d18923034bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:29.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.112+0000 7fc9af400640 1 -- 172.21.15.102:0/4041917257 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8039070 con 0x7fc9b015a1a0 2024-03-18T16:30:29.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.112+0000 7fc9af400640 1 -- 172.21.15.102:0/4041917257 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a802cd80 con 0x7fc9b015a1a0 2024-03-18T16:30:29.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.112+0000 7fc9af400640 1 -- 172.21.15.102:0/4041917257 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8034660 con 0x7fc9b015a1a0 2024-03-18T16:30:29.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.113+0000 7fc9b6400640 1 -- 172.21.15.102:0/4041917257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 msgr2=0x7fc9b006c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.113+0000 7fc9b6400640 1 --2- 172.21.15.102:0/4041917257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 0x7fc9b006c310 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009d90 tx=0x7fc9a802bfd0 comp rx=0 tx=0).stop 2024-03-18T16:30:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.113+0000 7fc9b6400640 1 -- 172.21.15.102:0/4041917257 shutdown_connections 2024-03-18T16:30:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.113+0000 7fc9b6400640 1 --2- 172.21.15.102:0/4041917257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 0x7fc9b006c310 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.113+0000 7fc9b6400640 1 -- 172.21.15.102:0/4041917257 >> 172.21.15.102:0/4041917257 conn(0x7fc9b0067190 msgr2=0x7fc9b00675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.113+0000 7fc9b6400640 1 -- 172.21.15.102:0/4041917257 shutdown_connections 2024-03-18T16:30:29.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.113+0000 7fc9b6400640 1 -- 172.21.15.102:0/4041917257 wait complete. 2024-03-18T16:30:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.115+0000 7fc9b6400640 1 Processor -- start 2024-03-18T16:30:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.115+0000 7fc9b6400640 1 -- start start 2024-03-18T16:30:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.116+0000 7fc9b6400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b00ca960 0x7fc9b01f1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.116+0000 7fc9b6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9b00cad40 0x7fc9b0158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.116+0000 7fc9b6400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 0x7fc9b00c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.116+0000 7fc9b6400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b015dc10 con 0x7fc9b00cad40 2024-03-18T16:30:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.116+0000 7fc9b6400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b0070cb0 con 0x7fc9b015a1a0 2024-03-18T16:30:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.116+0000 7fc9b6400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b0154030 con 0x7fc9b00ca960 2024-03-18T16:30:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.117+0000 7fc9b5a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 0x7fc9b00c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.117+0000 7fc9b5000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b00ca960 0x7fc9b01f1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.117+0000 7fc9b5a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 0x7fc9b00c6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:45586/0 (socket says 172.21.15.102:45586) 2024-03-18T16:30:29.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.117+0000 7fc9afe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9b00cad40 0x7fc9b0158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:29.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.117+0000 7fc9b5a00640 1 -- 172.21.15.102:0/416367145 learned_addr learned my addr 172.21.15.102:0/416367145 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.122+0000 7fc9b5a00640 1 -- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b00ca960 msgr2=0x7fc9b01f1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.122+0000 7fc9b5a00640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b00ca960 0x7fc9b01f1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.122+0000 7fc9b5a00640 1 -- 172.21.15.102:0/416367145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9b00cad40 msgr2=0x7fc9b0158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.122+0000 7fc9b5a00640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9b00cad40 0x7fc9b0158400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.122+0000 7fc9b5a00640 1 -- 172.21.15.102:0/416367145 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b01f0900 con 0x7fc9b015a1a0 2024-03-18T16:30:29.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.125+0000 7fc9b5a00640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 0x7fc9b00c6c70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009e80 tx=0x7fc9a8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.130+0000 7fc9acc00640 1 -- 172.21.15.102:0/416367145 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8044020 con 0x7fc9b015a1a0 2024-03-18T16:30:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.130+0000 7fc9acc00640 1 -- 172.21.15.102:0/416367145 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a803f670 con 0x7fc9b015a1a0 2024-03-18T16:30:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.130+0000 7fc9acc00640 1 -- 172.21.15.102:0/416367145 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8039040 con 0x7fc9b015a1a0 2024-03-18T16:30:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.130+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b01f0be0 con 0x7fc9b015a1a0 2024-03-18T16:30:29.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.130+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b01f3880 con 0x7fc9b015a1a0 2024-03-18T16:30:29.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.131+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b0074fd0 con 0x7fc9b015a1a0 2024-03-18T16:30:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.151+0000 7fc9acc00640 1 -- 172.21.15.102:0/416367145 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc9a8047020 con 0x7fc9b015a1a0 2024-03-18T16:30:29.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.152+0000 7fc9acc00640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc99006bc20 0x7fc99006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:29.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.153+0000 7fc9b5000640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc99006bc20 0x7fc99006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:29.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.153+0000 7fc9b5000640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc99006bc20 0x7fc99006e0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4007fd0 tx=0x7fc9a4007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:29.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.154+0000 7fc9acc00640 1 -- 172.21.15.102:0/416367145 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fc9a807a0d0 con 0x7fc9b015a1a0 2024-03-18T16:30:29.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.412+0000 7fc9acc00640 1 -- 172.21.15.102:0/416367145 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc9a804e030 con 0x7fc9b015a1a0 2024-03-18T16:30:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.762+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9b0070660 con 0x7fc9b015a1a0 2024-03-18T16:30:29.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.774+0000 7fc9acc00640 1 -- 172.21.15.102:0/416367145 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fc9a807c030 con 0x7fc9b015a1a0 2024-03-18T16:30:29.777 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:29.777 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:29.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc99006bc20 msgr2=0x7fc99006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:29.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc99006bc20 0x7fc99006e0e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4007fd0 tx=0x7fc9a4007e80 comp rx=0 tx=0).stop 2024-03-18T16:30:29.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 msgr2=0x7fc9b00c6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:29.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 0x7fc9b00c6c70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009e80 tx=0x7fc9a8004a40 comp rx=0 tx=0).stop 2024-03-18T16:30:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 shutdown_connections 2024-03-18T16:30:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc99006bc20 0x7fc99006e0e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9b015a1a0 0x7fc9b00c6c70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9b00cad40 0x7fc9b0158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 --2- 172.21.15.102:0/416367145 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9b00ca960 0x7fc9b01f1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 >> 172.21.15.102:0/416367145 conn(0x7fc9b0067190 msgr2=0x7fc9b0142290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 shutdown_connections 2024-03-18T16:30:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:29.777+0000 7fc9b6400640 1 -- 172.21.15.102:0/416367145 wait complete. 2024-03-18T16:30:30.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:30.400 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:31.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:31.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.577+0000 7f924e600640 1 Processor -- start 2024-03-18T16:30:31.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.577+0000 7f924e600640 1 -- start start 2024-03-18T16:30:31.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.578+0000 7f924e600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9248072700 0x7f9248158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:31.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.578+0000 7f924e600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f924815b8e0 0x7f92480c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:31.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.579+0000 7f924e600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f924815c200 con 0x7f92481524c0 2024-03-18T16:30:31.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.579+0000 7f924e600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9248072b20 con 0x7f924815b8e0 2024-03-18T16:30:31.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.579+0000 7f924e600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f924815c4e0 con 0x7f9248072700 2024-03-18T16:30:31.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.579+0000 7f9246a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9248072700 0x7f9248158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:31.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.580+0000 7f9246a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9248072700 0x7f9248158ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:39066/0 (socket says 172.21.15.102:39066) 2024-03-18T16:30:31.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.580+0000 7f9246a00640 1 -- 172.21.15.102:0/1589447138 learned_addr learned my addr 172.21.15.102:0/1589447138 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:31.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.580+0000 7f9247e00640 1 --2- 172.21.15.102:0/1589447138 >> v2:172.21.15.154:3300/0 conn(0x7f924815b8e0 0x7f92480c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.587+0000 7f9246a00640 1 -- 172.21.15.102:0/1589447138 >> v2:172.21.15.154:3300/0 conn(0x7f924815b8e0 msgr2=0x7f92480c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.587+0000 7f9246a00640 1 --2- 172.21.15.102:0/1589447138 >> v2:172.21.15.154:3300/0 conn(0x7f924815b8e0 0x7f92480c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.587+0000 7f9246a00640 1 -- 172.21.15.102:0/1589447138 >> v1:172.21.15.154:6789/0 conn(0x7f92481524c0 legacy=0x7f924806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:31.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.587+0000 7f9246a00640 1 -- 172.21.15.102:0/1589447138 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924815c7c0 con 0x7f9248072700 2024-03-18T16:30:31.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.639+0000 7f9246a00640 1 --2- 172.21.15.102:0/1589447138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9248072700 0x7f9248158ee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9234009870 tx=0x7f923402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=632bebdfa4673b13 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:31.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.646+0000 7f9246000640 1 -- 172.21.15.102:0/1589447138 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234034070 con 0x7f9248072700 2024-03-18T16:30:31.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.646+0000 7f9246000640 1 -- 172.21.15.102:0/1589447138 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9234030840 con 0x7f9248072700 2024-03-18T16:30:31.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.646+0000 7f9246000640 1 -- 172.21.15.102:0/1589447138 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234030c10 con 0x7f9248072700 2024-03-18T16:30:31.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.646+0000 7f924e600640 1 -- 172.21.15.102:0/1589447138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9248072700 msgr2=0x7f9248158ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:31.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.646+0000 7f924e600640 1 --2- 172.21.15.102:0/1589447138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9248072700 0x7f9248158ee0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9234009870 tx=0x7f923402b980 comp rx=0 tx=0).stop 2024-03-18T16:30:31.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.646+0000 7f924e600640 1 -- 172.21.15.102:0/1589447138 shutdown_connections 2024-03-18T16:30:31.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.646+0000 7f924e600640 1 --2- 172.21.15.102:0/1589447138 >> v2:172.21.15.154:3300/0 conn(0x7f924815b8e0 0x7f92480c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:31.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.647+0000 7f924e600640 1 --2- 172.21.15.102:0/1589447138 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9248072700 0x7f9248158ee0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:31.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.647+0000 7f924e600640 1 -- 172.21.15.102:0/1589447138 >> 172.21.15.102:0/1589447138 conn(0x7f92480700d0 msgr2=0x7f924806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:31.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.647+0000 7f924e600640 1 -- 172.21.15.102:0/1589447138 shutdown_connections 2024-03-18T16:30:31.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.647+0000 7f924e600640 1 -- 172.21.15.102:0/1589447138 wait complete. 2024-03-18T16:30:31.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.648+0000 7f924e600640 1 Processor -- start 2024-03-18T16:30:31.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.649+0000 7f924e600640 1 -- start start 2024-03-18T16:30:31.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.649+0000 7f924e600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9248072700 0x7f92480c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:31.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.649+0000 7f924e600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92481524c0 0x7f9248150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:31.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.649+0000 7f924e600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f924815b8e0 0x7f92480c3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:31.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.650+0000 7f924e600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f92481589c0 con 0x7f9248072700 2024-03-18T16:30:31.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.650+0000 7f924e600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f92481586e0 con 0x7f924815b8e0 2024-03-18T16:30:31.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.650+0000 7f924e600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f92481528a0 con 0x7f92481524c0 2024-03-18T16:30:31.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.650+0000 7f9246a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92481524c0 0x7f9248150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:31.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.650+0000 7f9247400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9248072700 0x7f92480c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:31.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.651+0000 7f9247400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9248072700 0x7f92480c2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45126/0 (socket says 172.21.15.102:45126) 2024-03-18T16:30:31.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.651+0000 7f9247400640 1 -- 172.21.15.102:0/2667150924 learned_addr learned my addr 172.21.15.102:0/2667150924 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:31.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.655+0000 7f9246a00640 1 -- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f924815b8e0 msgr2=0x7f92480c3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:31.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.655+0000 7f9246a00640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f924815b8e0 0x7f92480c3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:31.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.655+0000 7f9246a00640 1 -- 172.21.15.102:0/2667150924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9248072700 msgr2=0x7f92480c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:31.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.655+0000 7f9246a00640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9248072700 0x7f92480c2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:31.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.655+0000 7f9246a00640 1 -- 172.21.15.102:0/2667150924 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92480c48d0 con 0x7f92481524c0 2024-03-18T16:30:31.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.658+0000 7f9246a00640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92481524c0 0x7f9248150f00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9234033000 tx=0x7f92340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:31.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.663+0000 7f923f400640 1 -- 172.21.15.102:0/2667150924 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234034040 con 0x7f92481524c0 2024-03-18T16:30:31.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.663+0000 7f923f400640 1 -- 172.21.15.102:0/2667150924 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9234039d10 con 0x7f92481524c0 2024-03-18T16:30:31.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.663+0000 7f923f400640 1 -- 172.21.15.102:0/2667150924 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234043610 con 0x7f92481524c0 2024-03-18T16:30:31.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.663+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92480c4bb0 con 0x7f92481524c0 2024-03-18T16:30:31.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.663+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92480c1a30 con 0x7f92481524c0 2024-03-18T16:30:31.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.664+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9248074fd0 con 0x7f92481524c0 2024-03-18T16:30:31.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.938+0000 7f923f400640 1 -- 172.21.15.102:0/2667150924 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f923404a060 con 0x7f92481524c0 2024-03-18T16:30:31.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.940+0000 7f923f400640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f922806bc20 0x7f922806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:31.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.940+0000 7f9247400640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f922806bc20 0x7f922806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:31.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.941+0000 7f9247400640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f922806bc20 0x7f922806e0e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9238001840 tx=0x7f9238009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:31.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.943+0000 7f923f400640 1 -- 172.21.15.102:0/2667150924 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9234079160 con 0x7f92481524c0 2024-03-18T16:30:31.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:31.961+0000 7f923f400640 1 -- 172.21.15.102:0/2667150924 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f923403e070 con 0x7f92481524c0 2024-03-18T16:30:32.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.206+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92480caa60 con 0x7f92481524c0 2024-03-18T16:30:32.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.221+0000 7f923f400640 1 -- 172.21.15.102:0/2667150924 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f923407b7c0 con 0x7f92481524c0 2024-03-18T16:30:32.223 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:32.223 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:32.223 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:32.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f922806bc20 msgr2=0x7f922806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:32.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f922806bc20 0x7f922806e0e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9238001840 tx=0x7f9238009040 comp rx=0 tx=0).stop 2024-03-18T16:30:32.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92481524c0 msgr2=0x7f9248150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:32.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92481524c0 0x7f9248150f00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9234033000 tx=0x7f92340046c0 comp rx=0 tx=0).stop 2024-03-18T16:30:32.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 shutdown_connections 2024-03-18T16:30:32.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f922806bc20 0x7f922806e0e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:32.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f924815b8e0 0x7f92480c3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:32.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f92481524c0 0x7f9248150f00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:32.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 --2- 172.21.15.102:0/2667150924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9248072700 0x7f92480c2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:32.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 >> 172.21.15.102:0/2667150924 conn(0x7f92480700d0 msgr2=0x7f9248159dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:32.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 shutdown_connections 2024-03-18T16:30:32.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:32.224+0000 7f924e600640 1 -- 172.21.15.102:0/2667150924 wait complete. 2024-03-18T16:30:32.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:32.847 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:33.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:34.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.026+0000 7f8060a00640 1 Processor -- start 2024-03-18T16:30:34.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.026+0000 7f8060a00640 1 -- start start 2024-03-18T16:30:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.028+0000 7f8060a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f805c141650 0x7f805c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.028+0000 7f8060a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c0c4fa0 0x7f805c0c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.028+0000 7f8060a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f805c0c72d0 con 0x7f805c15c3a0 2024-03-18T16:30:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.028+0000 7f8060a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f805c159370 con 0x7f805c141650 2024-03-18T16:30:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.028+0000 7f8060a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f805c158ad0 con 0x7f805c0c4fa0 2024-03-18T16:30:34.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.029+0000 7f805be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c0c4fa0 0x7f805c0c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:34.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.029+0000 7f805b400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f805c141650 0x7f805c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:34.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.029+0000 7f805aa00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f805c15c3a0 0x7f805c0c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:56084/0 (socket says 172.21.15.102:56084) 2024-03-18T16:30:34.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.029+0000 7f805aa00640 1 -- 172.21.15.102:0/1143224191 learned_addr learned my addr 172.21.15.102:0/1143224191 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:34.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.033+0000 7f805b400640 1 -- 172.21.15.102:0/1143224191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c0c4fa0 msgr2=0x7f805c0c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:34.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.033+0000 7f805b400640 1 --2- 172.21.15.102:0/1143224191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c0c4fa0 0x7f805c0c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.033+0000 7f805b400640 1 -- 172.21.15.102:0/1143224191 >> v1:172.21.15.154:6789/0 conn(0x7f805c15c3a0 legacy=0x7f805c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:34.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.033+0000 7f805b400640 1 -- 172.21.15.102:0/1143224191 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805c158db0 con 0x7f805c141650 2024-03-18T16:30:34.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.037+0000 7f805b400640 1 --2- 172.21.15.102:0/1143224191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 0x7f805c158420 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f804c0097a0 tx=0x7f804c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eecca749c0cc632 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:34.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.041+0000 7f805a000640 1 -- 172.21.15.102:0/1143224191 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c035020 con 0x7f805c141650 2024-03-18T16:30:34.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.043+0000 7f805a000640 1 -- 172.21.15.102:0/1143224191 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f804c031810 con 0x7f805c141650 2024-03-18T16:30:34.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.043+0000 7f805a000640 1 -- 172.21.15.102:0/1143224191 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c031be0 con 0x7f805c141650 2024-03-18T16:30:34.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.043+0000 7f8060a00640 1 -- 172.21.15.102:0/1143224191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 msgr2=0x7f805c158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:34.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.043+0000 7f8060a00640 1 --2- 172.21.15.102:0/1143224191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 0x7f805c158420 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f804c0097a0 tx=0x7f804c02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.043+0000 7f8060a00640 1 -- 172.21.15.102:0/1143224191 shutdown_connections 2024-03-18T16:30:34.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.043+0000 7f8060a00640 1 --2- 172.21.15.102:0/1143224191 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c0c4fa0 0x7f805c0c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.043+0000 7f8060a00640 1 --2- 172.21.15.102:0/1143224191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 0x7f805c158420 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.043+0000 7f8060a00640 1 -- 172.21.15.102:0/1143224191 >> 172.21.15.102:0/1143224191 conn(0x7f805c0cb890 msgr2=0x7f805c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:34.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.044+0000 7f8060a00640 1 -- 172.21.15.102:0/1143224191 shutdown_connections 2024-03-18T16:30:34.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.044+0000 7f8060a00640 1 -- 172.21.15.102:0/1143224191 wait complete. 2024-03-18T16:30:34.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.045+0000 7f8060a00640 1 Processor -- start 2024-03-18T16:30:34.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.045+0000 7f8060a00640 1 -- start start 2024-03-18T16:30:34.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.046+0000 7f8060a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805c0c4fa0 0x7f805c13e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:34.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.046+0000 7f8060a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 0x7f805c1f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:34.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.046+0000 7f8060a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c15c3a0 0x7f805c0c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:34.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.046+0000 7f8060a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f805c0c66a0 con 0x7f805c0c4fa0 2024-03-18T16:30:34.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.046+0000 7f8060a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f805c0c63c0 con 0x7f805c141650 2024-03-18T16:30:34.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.046+0000 7f8060a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f805c15c7c0 con 0x7f805c15c3a0 2024-03-18T16:30:34.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.047+0000 7f805aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 0x7f805c1f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:34.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.047+0000 7f805be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c15c3a0 0x7f805c0c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:34.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.047+0000 7f805aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 0x7f805c1f3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56720/0 (socket says 172.21.15.102:56720) 2024-03-18T16:30:34.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.047+0000 7f805aa00640 1 -- 172.21.15.102:0/3111843079 learned_addr learned my addr 172.21.15.102:0/3111843079 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:34.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.047+0000 7f805b400640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805c0c4fa0 0x7f805c13e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:34.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.051+0000 7f805aa00640 1 -- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c15c3a0 msgr2=0x7f805c0c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:34.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.051+0000 7f805aa00640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c15c3a0 0x7f805c0c85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.051+0000 7f805aa00640 1 -- 172.21.15.102:0/3111843079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805c0c4fa0 msgr2=0x7f805c13e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:34.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.051+0000 7f805aa00640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805c0c4fa0 0x7f805c13e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.051+0000 7f805aa00640 1 -- 172.21.15.102:0/3111843079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805c157c70 con 0x7f805c141650 2024-03-18T16:30:34.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.053+0000 7f805aa00640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 0x7f805c1f3b00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f804800c400 tx=0x7f804800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.057+0000 7f8053400640 1 -- 172.21.15.102:0/3111843079 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8048017070 con 0x7f805c141650 2024-03-18T16:30:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.057+0000 7f8053400640 1 -- 172.21.15.102:0/3111843079 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8048005ce0 con 0x7f805c141650 2024-03-18T16:30:34.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.057+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805c1f0760 con 0x7f805c141650 2024-03-18T16:30:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.057+0000 7f8053400640 1 -- 172.21.15.102:0/3111843079 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80480072f0 con 0x7f805c141650 2024-03-18T16:30:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.057+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f805c1f0d20 con 0x7f805c141650 2024-03-18T16:30:34.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.059+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f805c152590 con 0x7f805c141650 2024-03-18T16:30:34.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.077+0000 7f8053400640 1 -- 172.21.15.102:0/3111843079 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f804801a070 con 0x7f805c141650 2024-03-18T16:30:34.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.079+0000 7f8053400640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f803c06bc50 0x7f803c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:34.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.079+0000 7f805b400640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f803c06bc50 0x7f803c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:34.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.080+0000 7f805b400640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f803c06bc50 0x7f803c06e110 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f804c036000 tx=0x7f804c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:34.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.080+0000 7f8053400640 1 -- 172.21.15.102:0/3111843079 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8048056780 con 0x7f805c141650 2024-03-18T16:30:34.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.332+0000 7f8053400640 1 -- 172.21.15.102:0/3111843079 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f805c152590 con 0x7f805c141650 2024-03-18T16:30:34.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.594+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f805c074fd0 con 0x7f805c141650 2024-03-18T16:30:34.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.606+0000 7f8053400640 1 -- 172.21.15.102:0/3111843079 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8048021020 con 0x7f805c141650 2024-03-18T16:30:34.608 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:34.608 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:34.609 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:34.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f803c06bc50 msgr2=0x7f803c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:34.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f803c06bc50 0x7f803c06e110 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f804c036000 tx=0x7f804c008030 comp rx=0 tx=0).stop 2024-03-18T16:30:34.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 msgr2=0x7f805c1f3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:34.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 0x7f805c1f3b00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f804800c400 tx=0x7f804800c8c0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 shutdown_connections 2024-03-18T16:30:34.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f803c06bc50 0x7f803c06e110 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f805c15c3a0 0x7f805c0c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c141650 0x7f805c1f3b00 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 --2- 172.21.15.102:0/3111843079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f805c0c4fa0 0x7f805c13e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 >> 172.21.15.102:0/3111843079 conn(0x7f805c0cb890 msgr2=0x7f805c13ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 shutdown_connections 2024-03-18T16:30:34.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:34.609+0000 7f8060a00640 1 -- 172.21.15.102:0/3111843079 wait complete. 2024-03-18T16:30:35.232 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:35.232 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:36.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:36.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.403+0000 7f3e8be00640 1 Processor -- start 2024-03-18T16:30:36.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.404+0000 7f3e8be00640 1 -- start start 2024-03-18T16:30:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.405+0000 7f3e8be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3e8c0c9be0 0x7f3e8c0d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.405+0000 7f3e8be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e8c154ab0 con 0x7f3e8c0d05a0 2024-03-18T16:30:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.406+0000 7f3e8be00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e8c15b250 con 0x7f3e8c15ae70 2024-03-18T16:30:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.406+0000 7f3e8be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3e8c0caba0 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.406+0000 7f3e8aa00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3e8c15ae70 0x7f3e8c0ca490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:56092/0 (socket says 172.21.15.102:56092) 2024-03-18T16:30:36.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.406+0000 7f3e8aa00640 1 -- 172.21.15.102:0/3110483858 learned_addr learned my addr 172.21.15.102:0/3110483858 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:36.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.407+0000 7f3e8a000640 1 --2- 172.21.15.102:0/3110483858 >> v2:172.21.15.102:3300/0 conn(0x7f3e8c0c9be0 0x7f3e8c0d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.409+0000 7f3e89600640 1 -- 172.21.15.102:0/3110483858 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 719534824 0 0) 0x7f3e8c15b250 con 0x7f3e8c15ae70 2024-03-18T16:30:36.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.409+0000 7f3e89600640 1 -- 172.21.15.102:0/3110483858 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e70003520 con 0x7f3e8c15ae70 2024-03-18T16:30:36.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.412+0000 7f3e8a000640 1 -- 172.21.15.102:0/3110483858 >> v1:172.21.15.154:6789/0 conn(0x7f3e8c15ae70 legacy=0x7f3e8c0ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:36.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.412+0000 7f3e8a000640 1 -- 172.21.15.102:0/3110483858 >> v1:172.21.15.102:6789/0 conn(0x7f3e8c0d05a0 legacy=0x7f3e8c0cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:36.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.412+0000 7f3e8a000640 1 -- 172.21.15.102:0/3110483858 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c06d2f0 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.417+0000 7f3e8a000640 1 --2- 172.21.15.102:0/3110483858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e8c0c9be0 0x7f3e8c0d0060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3e780097a0 tx=0x7f3e7802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b6afa2c0b0a020 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.420+0000 7f3e89600640 1 -- 172.21.15.102:0/3110483858 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e78034070 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.423+0000 7f3e89600640 1 -- 172.21.15.102:0/3110483858 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e78030610 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.423+0000 7f3e89600640 1 -- 172.21.15.102:0/3110483858 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e780309c0 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.424+0000 7f3e8be00640 1 -- 172.21.15.102:0/3110483858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e8c0c9be0 msgr2=0x7f3e8c0d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.424+0000 7f3e8be00640 1 --2- 172.21.15.102:0/3110483858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e8c0c9be0 0x7f3e8c0d0060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3e780097a0 tx=0x7f3e7802b960 comp rx=0 tx=0).stop 2024-03-18T16:30:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.424+0000 7f3e8be00640 1 -- 172.21.15.102:0/3110483858 shutdown_connections 2024-03-18T16:30:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.424+0000 7f3e8be00640 1 --2- 172.21.15.102:0/3110483858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e8c0c9be0 0x7f3e8c0d0060 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:36.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.424+0000 7f3e8be00640 1 -- 172.21.15.102:0/3110483858 >> 172.21.15.102:0/3110483858 conn(0x7f3e8c0673a0 msgr2=0x7f3e8c0677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.424+0000 7f3e8be00640 1 -- 172.21.15.102:0/3110483858 shutdown_connections 2024-03-18T16:30:36.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.424+0000 7f3e8be00640 1 -- 172.21.15.102:0/3110483858 wait complete. 2024-03-18T16:30:36.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.427+0000 7f3e8be00640 1 Processor -- start 2024-03-18T16:30:36.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.427+0000 7f3e8be00640 1 -- start start 2024-03-18T16:30:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.428+0000 7f3e8be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e8c0c9be0 0x7f3e8c1f10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.429+0000 7f3e8be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e8c0d05a0 0x7f3e8c154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.429+0000 7f3e8be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8c15ae70 0x7f3e8c0c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:36.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.429+0000 7f3e8be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c06db90 con 0x7f3e8c0d05a0 2024-03-18T16:30:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.429+0000 7f3e8be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e8c06d8b0 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.429+0000 7f3e8be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3e8c06de70 con 0x7f3e8c15ae70 2024-03-18T16:30:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.429+0000 7f3e8aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e8c0c9be0 0x7f3e8c1f10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:36.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.429+0000 7f3e8a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e8c0d05a0 0x7f3e8c154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:36.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.429+0000 7f3e8b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8c15ae70 0x7f3e8c0c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:36.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.430+0000 7f3e8aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e8c0c9be0 0x7f3e8c1f10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56724/0 (socket says 172.21.15.102:56724) 2024-03-18T16:30:36.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.430+0000 7f3e8aa00640 1 -- 172.21.15.102:0/419466558 learned_addr learned my addr 172.21.15.102:0/419466558 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.433+0000 7f3e8aa00640 1 -- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8c15ae70 msgr2=0x7f3e8c0c48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.433+0000 7f3e8aa00640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8c15ae70 0x7f3e8c0c48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.433+0000 7f3e8aa00640 1 -- 172.21.15.102:0/419466558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e8c0d05a0 msgr2=0x7f3e8c154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.433+0000 7f3e8aa00640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e8c0d05a0 0x7f3e8c154fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:36.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.433+0000 7f3e8aa00640 1 -- 172.21.15.102:0/419466558 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e8c0c5cc0 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.436+0000 7f3e8aa00640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e8c0c9be0 0x7f3e8c1f10f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00c500 tx=0x7f3e7c00c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.441+0000 7f3e83400640 1 -- 172.21.15.102:0/419466558 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c017070 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.441+0000 7f3e83400640 1 -- 172.21.15.102:0/419466558 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e7c005ce0 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.441+0000 7f3e83400640 1 -- 172.21.15.102:0/419466558 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c007330 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.441+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e8c0c5fa0 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.441+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e8c0c3e00 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.443+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e8c074fd0 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.462+0000 7f3e83400640 1 -- 172.21.15.102:0/419466558 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3e7c01a070 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.463+0000 7f3e83400640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e6c06bc50 0x7f3e6c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:36.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.463+0000 7f3e8a000640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e6c06bc50 0x7f3e6c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:36.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.464+0000 7f3e8a000640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e6c06bc50 0x7f3e6c06e110 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3e78036000 tx=0x7f3e78008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:36.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.464+0000 7f3e83400640 1 -- 172.21.15.102:0/419466558 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3e7c0568d0 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.714+0000 7f3e83400640 1 -- 172.21.15.102:0/419466558 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3e7c021020 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.972+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e8c0b5a90 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.992+0000 7f3e83400640 1 -- 172.21.15.102:0/419466558 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3e7c058830 con 0x7f3e8c0c9be0 2024-03-18T16:30:36.994 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:36.994 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:36.995 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e6c06bc50 msgr2=0x7f3e6c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e6c06bc50 0x7f3e6c06e110 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3e78036000 tx=0x7f3e78008030 comp rx=0 tx=0).stop 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e8c0c9be0 msgr2=0x7f3e8c1f10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e8c0c9be0 0x7f3e8c1f10f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00c500 tx=0x7f3e7c00c9c0 comp rx=0 tx=0).stop 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 shutdown_connections 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3e6c06bc50 0x7f3e6c06e110 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3e8c15ae70 0x7f3e8c0c48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e8c0d05a0 0x7f3e8c154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 --2- 172.21.15.102:0/419466558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e8c0c9be0 0x7f3e8c1f10f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:36.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 >> 172.21.15.102:0/419466558 conn(0x7f3e8c0673a0 msgr2=0x7f3e8c0cdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:36.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 shutdown_connections 2024-03-18T16:30:36.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:36.995+0000 7f3e8be00640 1 -- 172.21.15.102:0/419466558 wait complete. 2024-03-18T16:30:37.618 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:37.618 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:38.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:38.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.790+0000 7f242ee00640 1 Processor -- start 2024-03-18T16:30:38.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.790+0000 7f242ee00640 1 -- start start 2024-03-18T16:30:38.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.791+0000 7f242ee00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f242815a310 0x7f242806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:38.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.791+0000 7f242ee00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f242815ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:38.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.791+0000 7f242ee00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f242815baa0 0x7f242815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:38.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.791+0000 7f242ee00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f24280cd410 con 0x7f242815baa0 2024-03-18T16:30:38.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.791+0000 7f242ee00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f242815c470 con 0x7f242815a310 2024-03-18T16:30:38.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.791+0000 7f242ee00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24280ca500 con 0x7f242815a6f0 2024-03-18T16:30:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.792+0000 7f242d000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f242815ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.792+0000 7f242da00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f242815a310 0x7f242806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.792+0000 7f242e400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f242815baa0 0x7f242815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.792+0000 7f242da00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f242815a310 0x7f242806d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56734/0 (socket says 172.21.15.102:56734) 2024-03-18T16:30:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.792+0000 7f242da00640 1 -- 172.21.15.102:0/2438992197 learned_addr learned my addr 172.21.15.102:0/2438992197 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:38.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.795+0000 7f242e400640 1 -- 172.21.15.102:0/2438992197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 msgr2=0x7f242815ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:38.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.795+0000 7f242e400640 1 --2- 172.21.15.102:0/2438992197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f242815ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:38.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.795+0000 7f242e400640 1 -- 172.21.15.102:0/2438992197 >> v2:172.21.15.154:3300/0 conn(0x7f242815a310 msgr2=0x7f242806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:38.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.795+0000 7f242e400640 1 --2- 172.21.15.102:0/2438992197 >> v2:172.21.15.154:3300/0 conn(0x7f242815a310 0x7f242806d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:38.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.795+0000 7f242e400640 1 -- 172.21.15.102:0/2438992197 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242815b050 con 0x7f242815baa0 2024-03-18T16:30:38.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.798+0000 7f242e400640 1 --2- 172.21.15.102:0/2438992197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f242815baa0 0x7f242815bf30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2428067db0 tx=0x7f242002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dada1d5ce01998b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:38.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.802+0000 7f2427e00640 1 -- 172.21.15.102:0/2438992197 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420039070 con 0x7f242815baa0 2024-03-18T16:30:38.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.802+0000 7f2427e00640 1 -- 172.21.15.102:0/2438992197 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f242002cbd0 con 0x7f242815baa0 2024-03-18T16:30:38.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.802+0000 7f2427e00640 1 -- 172.21.15.102:0/2438992197 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420034440 con 0x7f242815baa0 2024-03-18T16:30:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.803+0000 7f242ee00640 1 -- 172.21.15.102:0/2438992197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f242815baa0 msgr2=0x7f242815bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.803+0000 7f242ee00640 1 --2- 172.21.15.102:0/2438992197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f242815baa0 0x7f242815bf30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2428067db0 tx=0x7f242002be20 comp rx=0 tx=0).stop 2024-03-18T16:30:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.803+0000 7f242ee00640 1 -- 172.21.15.102:0/2438992197 shutdown_connections 2024-03-18T16:30:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.803+0000 7f242ee00640 1 --2- 172.21.15.102:0/2438992197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f242815baa0 0x7f242815bf30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.803+0000 7f242ee00640 1 --2- 172.21.15.102:0/2438992197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f242815ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.803+0000 7f242ee00640 1 --2- 172.21.15.102:0/2438992197 >> v2:172.21.15.154:3300/0 conn(0x7f242815a310 0x7f242806d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.803+0000 7f242ee00640 1 -- 172.21.15.102:0/2438992197 >> 172.21.15.102:0/2438992197 conn(0x7f242808c1a0 msgr2=0x7f2428066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.804+0000 7f242ee00640 1 -- 172.21.15.102:0/2438992197 shutdown_connections 2024-03-18T16:30:38.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.804+0000 7f242ee00640 1 -- 172.21.15.102:0/2438992197 wait complete. 2024-03-18T16:30:38.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.805+0000 7f242ee00640 1 Processor -- start 2024-03-18T16:30:38.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.806+0000 7f242ee00640 1 -- start start 2024-03-18T16:30:38.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.807+0000 7f242ee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f242815a310 0x7f24281f14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.807+0000 7f242ee00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f24281f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.807+0000 7f242ee00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f242815baa0 0x7f24281538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.807+0000 7f242ee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24280cce50 con 0x7f242815a310 2024-03-18T16:30:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.807+0000 7f242ee00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24280caa80 con 0x7f242815baa0 2024-03-18T16:30:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.807+0000 7f242ee00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24280cd130 con 0x7f242815a6f0 2024-03-18T16:30:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.807+0000 7f242d000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f24281f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.808+0000 7f242d000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f24281f4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:39138/0 (socket says 172.21.15.102:39138) 2024-03-18T16:30:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.808+0000 7f242d000640 1 -- 172.21.15.102:0/1146108620 learned_addr learned my addr 172.21.15.102:0/1146108620 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.808+0000 7f242e400640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f242815baa0 0x7f24281538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:38.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.809+0000 7f242da00640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f242815a310 0x7f24281f14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:38.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.811+0000 7f242d000640 1 -- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f242815baa0 msgr2=0x7f24281538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:38.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.811+0000 7f242d000640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f242815baa0 0x7f24281538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:38.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.811+0000 7f242d000640 1 -- 172.21.15.102:0/1146108620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f242815a310 msgr2=0x7f24281f14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:38.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.811+0000 7f242d000640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f242815a310 0x7f24281f14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:38.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.811+0000 7f242d000640 1 -- 172.21.15.102:0/1146108620 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24280c8fa0 con 0x7f242815a6f0 2024-03-18T16:30:38.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.814+0000 7f242d000640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f24281f4160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f241400e8a0 tx=0x7f241400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:38.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.818+0000 7f2425600640 1 -- 172.21.15.102:0/1146108620 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414017070 con 0x7f242815a6f0 2024-03-18T16:30:38.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.818+0000 7f2425600640 1 -- 172.21.15.102:0/1146108620 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f241400c940 con 0x7f242815a6f0 2024-03-18T16:30:38.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.818+0000 7f2425600640 1 -- 172.21.15.102:0/1146108620 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241400cd30 con 0x7f242815a6f0 2024-03-18T16:30:38.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.818+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24280c9280 con 0x7f242815a6f0 2024-03-18T16:30:38.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.819+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24281f3960 con 0x7f242815a6f0 2024-03-18T16:30:38.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.820+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2428074fd0 con 0x7f242815a6f0 2024-03-18T16:30:38.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.839+0000 7f2425600640 1 -- 172.21.15.102:0/1146108620 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f241401a070 con 0x7f242815a6f0 2024-03-18T16:30:38.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.840+0000 7f2425600640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f240806bbd0 0x7f240806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:38.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.841+0000 7f242da00640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f240806bbd0 0x7f240806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:38.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.842+0000 7f242da00640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f240806bbd0 0x7f240806e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f241c006cc0 tx=0x7f241c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:38.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:38.842+0000 7f2425600640 1 -- 172.21.15.102:0/1146108620 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f24140575d0 con 0x7f242815a6f0 2024-03-18T16:30:39.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.092+0000 7f2425600640 1 -- 172.21.15.102:0/1146108620 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2414059530 con 0x7f242815a6f0 2024-03-18T16:30:39.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.333+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24280caed0 con 0x7f242815a6f0 2024-03-18T16:30:39.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.402+0000 7f2425600640 1 -- 172.21.15.102:0/1146108620 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f241405d5d0 con 0x7f242815a6f0 2024-03-18T16:30:39.404 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:39.405 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:39.405 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:39.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f240806bbd0 msgr2=0x7f240806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:39.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f240806bbd0 0x7f240806e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f241c006cc0 tx=0x7f241c009240 comp rx=0 tx=0).stop 2024-03-18T16:30:39.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 msgr2=0x7f24281f4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:39.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f24281f4160 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f241400e8a0 tx=0x7f241400ed60 comp rx=0 tx=0).stop 2024-03-18T16:30:39.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 shutdown_connections 2024-03-18T16:30:39.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f240806bbd0 0x7f240806e090 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:39.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f242815baa0 0x7f24281538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:39.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f242815a6f0 0x7f24281f4160 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:39.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 --2- 172.21.15.102:0/1146108620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f242815a310 0x7f24281f14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:39.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 >> 172.21.15.102:0/1146108620 conn(0x7f242808c1a0 msgr2=0x7f242815d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:39.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.405+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 shutdown_connections 2024-03-18T16:30:39.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:39.406+0000 7f242ee00640 1 -- 172.21.15.102:0/1146108620 wait complete. 2024-03-18T16:30:40.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:40.028 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:41.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:41.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.201+0000 7f08a0e00640 1 Processor -- start 2024-03-18T16:30:41.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.202+0000 7f08a0e00640 1 -- start start 2024-03-18T16:30:41.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.203+0000 7f08a0e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0cfe70 0x7f089c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:41.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.203+0000 7f08a0e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089c0cabd0 con 0x7f089c0d0250 2024-03-18T16:30:41.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.203+0000 7f08a0e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089c0caea0 con 0x7f089c0cc2f0 2024-03-18T16:30:41.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.203+0000 7f08a0e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f089c0cb180 con 0x7f089c0cfe70 2024-03-18T16:30:41.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.203+0000 7f089aa00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f089c0d0250 0x7f089c0c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53150/0 (socket says 172.21.15.102:53150) 2024-03-18T16:30:41.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.204+0000 7f089aa00640 1 -- 172.21.15.102:0/4139804251 learned_addr learned my addr 172.21.15.102:0/4139804251 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:41.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.204+0000 7f089b400640 1 --2- 172.21.15.102:0/4139804251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0cfe70 0x7f089c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:41.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.207+0000 7f089a000640 1 -- 172.21.15.102:0/4139804251 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677234796 0 0) 0x7f089c0caea0 con 0x7f089c0cc2f0 2024-03-18T16:30:41.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.207+0000 7f089a000640 1 -- 172.21.15.102:0/4139804251 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0880003520 con 0x7f089c0cc2f0 2024-03-18T16:30:41.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.207+0000 7f089a000640 1 -- 172.21.15.102:0/4139804251 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2825194447 0 0) 0x7f089c0cabd0 con 0x7f089c0d0250 2024-03-18T16:30:41.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.207+0000 7f089a000640 1 -- 172.21.15.102:0/4139804251 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f089c0caea0 con 0x7f089c0d0250 2024-03-18T16:30:41.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.207+0000 7f089b400640 1 -- 172.21.15.102:0/4139804251 >> v1:172.21.15.154:6789/0 conn(0x7f089c0cc2f0 legacy=0x7f089c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:41.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.207+0000 7f089b400640 1 -- 172.21.15.102:0/4139804251 >> v1:172.21.15.102:6789/0 conn(0x7f089c0d0250 legacy=0x7f089c0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:41.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.207+0000 7f089b400640 1 -- 172.21.15.102:0/4139804251 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089c0cb460 con 0x7f089c0cfe70 2024-03-18T16:30:41.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.210+0000 7f089b400640 1 --2- 172.21.15.102:0/4139804251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0cfe70 0x7f089c15ae40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f088c0097d0 tx=0x7f088c02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771fe230bff8379a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:41.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.215+0000 7f089a000640 1 -- 172.21.15.102:0/4139804251 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c034070 con 0x7f089c0cfe70 2024-03-18T16:30:41.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.215+0000 7f089a000640 1 -- 172.21.15.102:0/4139804251 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f088c030880 con 0x7f089c0cfe70 2024-03-18T16:30:41.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.215+0000 7f089a000640 1 -- 172.21.15.102:0/4139804251 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c030c50 con 0x7f089c0cfe70 2024-03-18T16:30:41.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.216+0000 7f08a0e00640 1 -- 172.21.15.102:0/4139804251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0cfe70 msgr2=0x7f089c15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:41.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.216+0000 7f08a0e00640 1 --2- 172.21.15.102:0/4139804251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0cfe70 0x7f089c15ae40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f088c0097d0 tx=0x7f088c02b9c0 comp rx=0 tx=0).stop 2024-03-18T16:30:41.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.216+0000 7f08a0e00640 1 -- 172.21.15.102:0/4139804251 shutdown_connections 2024-03-18T16:30:41.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.216+0000 7f08a0e00640 1 --2- 172.21.15.102:0/4139804251 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0cfe70 0x7f089c15ae40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:41.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.216+0000 7f08a0e00640 1 -- 172.21.15.102:0/4139804251 >> 172.21.15.102:0/4139804251 conn(0x7f089c08c3e0 msgr2=0x7f089c06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:41.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.216+0000 7f08a0e00640 1 -- 172.21.15.102:0/4139804251 shutdown_connections 2024-03-18T16:30:41.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.216+0000 7f08a0e00640 1 -- 172.21.15.102:0/4139804251 wait complete. 2024-03-18T16:30:41.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.219+0000 7f08a0e00640 1 Processor -- start 2024-03-18T16:30:41.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.219+0000 7f08a0e00640 1 -- start start 2024-03-18T16:30:41.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.220+0000 7f08a0e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f089c0cc2f0 0x7f089c157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:41.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.220+0000 7f08a0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f089c0cfe70 0x7f089c158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:41.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.220+0000 7f08a0e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0d0250 0x7f089c1594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:41.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.220+0000 7f08a0e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f089c0cbd00 con 0x7f089c0cfe70 2024-03-18T16:30:41.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.220+0000 7f08a0e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f089c0cba20 con 0x7f089c0cc2f0 2024-03-18T16:30:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.220+0000 7f08a0e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f089c0c8330 con 0x7f089c0d0250 2024-03-18T16:30:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.221+0000 7f089b400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f089c0cc2f0 0x7f089c157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.221+0000 7f089aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f089c0cfe70 0x7f089c158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.221+0000 7f089be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0d0250 0x7f089c1594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.221+0000 7f089b400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f089c0cc2f0 0x7f089c157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47780/0 (socket says 172.21.15.102:47780) 2024-03-18T16:30:41.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.221+0000 7f089b400640 1 -- 172.21.15.102:0/1362379179 learned_addr learned my addr 172.21.15.102:0/1362379179 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:41.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.225+0000 7f089b400640 1 -- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0d0250 msgr2=0x7f089c1594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:41.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.225+0000 7f089b400640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0d0250 0x7f089c1594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:41.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.225+0000 7f089b400640 1 -- 172.21.15.102:0/1362379179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f089c0cfe70 msgr2=0x7f089c158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:41.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.225+0000 7f089b400640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f089c0cfe70 0x7f089c158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:41.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.225+0000 7f089b400640 1 -- 172.21.15.102:0/1362379179 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089c1efe80 con 0x7f089c0cc2f0 2024-03-18T16:30:41.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.227+0000 7f089b400640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f089c0cc2f0 0x7f089c157e40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f088c02bfd0 tx=0x7f088c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:41.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.232+0000 7f0893400640 1 -- 172.21.15.102:0/1362379179 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c034040 con 0x7f089c0cc2f0 2024-03-18T16:30:41.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.232+0000 7f0893400640 1 -- 172.21.15.102:0/1362379179 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f088c030680 con 0x7f089c0cc2f0 2024-03-18T16:30:41.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.232+0000 7f0893400640 1 -- 172.21.15.102:0/1362379179 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c006e10 con 0x7f089c0cc2f0 2024-03-18T16:30:41.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.232+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f089c157080 con 0x7f089c0cc2f0 2024-03-18T16:30:41.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.232+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f089c157640 con 0x7f089c0cc2f0 2024-03-18T16:30:41.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.234+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f089c074fd0 con 0x7f089c0cc2f0 2024-03-18T16:30:41.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.258+0000 7f0893400640 1 -- 172.21.15.102:0/1362379179 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f088c037070 con 0x7f089c0cc2f0 2024-03-18T16:30:41.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.260+0000 7f0893400640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f087c06bc20 0x7f087c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:41.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.260+0000 7f089aa00640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f087c06bc20 0x7f087c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:41.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.261+0000 7f089aa00640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f087c06bc20 0x7f087c06e0e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f089c1591c0 tx=0x7f0888007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:41.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.505+0000 7f0893400640 1 -- 172.21.15.102:0/1362379179 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f088c030830 con 0x7f089c0cc2f0 2024-03-18T16:30:41.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.516+0000 7f0893400640 1 -- 172.21.15.102:0/1362379179 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f088c07bb50 con 0x7f089c0cc2f0 2024-03-18T16:30:41.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.846+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f089c067d70 con 0x7f089c0cc2f0 2024-03-18T16:30:41.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.861+0000 7f0893400640 1 -- 172.21.15.102:0/1362379179 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f088c07fbf0 con 0x7f089c0cc2f0 2024-03-18T16:30:41.863 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:41.863 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:41.864 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:41.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f087c06bc20 msgr2=0x7f087c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:41.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f087c06bc20 0x7f087c06e0e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f089c1591c0 tx=0x7f0888007d30 comp rx=0 tx=0).stop 2024-03-18T16:30:41.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f089c0cc2f0 msgr2=0x7f089c157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:41.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f089c0cc2f0 0x7f089c157e40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f088c02bfd0 tx=0x7f088c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:30:41.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 shutdown_connections 2024-03-18T16:30:41.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f087c06bc20 0x7f087c06e0e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:41.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f089c0d0250 0x7f089c1594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:41.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f089c0cfe70 0x7f089c158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:41.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 --2- 172.21.15.102:0/1362379179 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f089c0cc2f0 0x7f089c157e40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:41.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 >> 172.21.15.102:0/1362379179 conn(0x7f089c08c3e0 msgr2=0x7f089c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:41.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 shutdown_connections 2024-03-18T16:30:41.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:41.865+0000 7f08a0e00640 1 -- 172.21.15.102:0/1362379179 wait complete. 2024-03-18T16:30:42.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:42.487 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:43.489 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:43.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.671+0000 7f19b4600640 1 Processor -- start 2024-03-18T16:30:43.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.671+0000 7f19b4600640 1 -- start start 2024-03-18T16:30:43.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.673+0000 7f19b4600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f19ac154690 0x7f19ac15e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:43.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.673+0000 7f19b4600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19ac154a70 0x7f19ac0c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:43.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.673+0000 7f19b4600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19ac15a710 con 0x7f19ac0c9e10 2024-03-18T16:30:43.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.673+0000 7f19b4600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f19ac15a9f0 con 0x7f19ac154a70 2024-03-18T16:30:43.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.673+0000 7f19b4600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f19ac15acd0 con 0x7f19ac154690 2024-03-18T16:30:43.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.740+0000 7f19b1200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19ac154a70 0x7f19ac0c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:43.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.742+0000 7f19b1200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19ac154a70 0x7f19ac0c98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46858/0 (socket says 172.21.15.102:46858) 2024-03-18T16:30:43.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.742+0000 7f19b1200640 1 -- 172.21.15.102:0/2893182395 learned_addr learned my addr 172.21.15.102:0/2893182395 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:43.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.745+0000 7f19b1200640 1 -- 172.21.15.102:0/2893182395 >> v2:172.21.15.154:3300/0 conn(0x7f19ac154690 msgr2=0x7f19ac15e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:43.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.745+0000 7f19b1200640 1 --2- 172.21.15.102:0/2893182395 >> v2:172.21.15.154:3300/0 conn(0x7f19ac154690 0x7f19ac15e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:43.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.745+0000 7f19b1200640 1 -- 172.21.15.102:0/2893182395 >> v1:172.21.15.154:6789/0 conn(0x7f19ac0c9e10 legacy=0x7f19ac071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:43.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.745+0000 7f19b1200640 1 -- 172.21.15.102:0/2893182395 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac15afb0 con 0x7f19ac154a70 2024-03-18T16:30:43.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.750+0000 7f19b1200640 1 --2- 172.21.15.102:0/2893182395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 0x7f19ac0c98d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f19ac071db0 tx=0x7f199802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad0aecda9c65f1e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:43.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.754+0000 7f19abe00640 1 -- 172.21.15.102:0/2893182395 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1998035020 con 0x7f19ac154a70 2024-03-18T16:30:43.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.757+0000 7f19abe00640 1 -- 172.21.15.102:0/2893182395 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1998031870 con 0x7f19ac154a70 2024-03-18T16:30:43.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.757+0000 7f19abe00640 1 -- 172.21.15.102:0/2893182395 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1998031c40 con 0x7f19ac154a70 2024-03-18T16:30:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.758+0000 7f19b4600640 1 -- 172.21.15.102:0/2893182395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 msgr2=0x7f19ac0c98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.758+0000 7f19b4600640 1 --2- 172.21.15.102:0/2893182395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 0x7f19ac0c98d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f19ac071db0 tx=0x7f199802b910 comp rx=0 tx=0).stop 2024-03-18T16:30:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.758+0000 7f19b4600640 1 -- 172.21.15.102:0/2893182395 shutdown_connections 2024-03-18T16:30:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.758+0000 7f19b4600640 1 --2- 172.21.15.102:0/2893182395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 0x7f19ac0c98d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:43.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.758+0000 7f19b4600640 1 --2- 172.21.15.102:0/2893182395 >> v2:172.21.15.154:3300/0 conn(0x7f19ac154690 0x7f19ac15e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:43.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.758+0000 7f19b4600640 1 -- 172.21.15.102:0/2893182395 >> 172.21.15.102:0/2893182395 conn(0x7f19ac0700d0 msgr2=0x7f19ac06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:43.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.758+0000 7f19b4600640 1 -- 172.21.15.102:0/2893182395 shutdown_connections 2024-03-18T16:30:43.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.758+0000 7f19b4600640 1 -- 172.21.15.102:0/2893182395 wait complete. 2024-03-18T16:30:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.760+0000 7f19b4600640 1 Processor -- start 2024-03-18T16:30:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.761+0000 7f19b4600640 1 -- start start 2024-03-18T16:30:43.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.761+0000 7f19b4600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19ac0c9e10 0x7f19ac157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:43.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.761+0000 7f19b4600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19ac154690 0x7f19ac152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:43.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.761+0000 7f19b4600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 0x7f19ac1e3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:43.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.761+0000 7f19b4600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac0726b0 con 0x7f19ac154a70 2024-03-18T16:30:43.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.761+0000 7f19b4600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac072530 con 0x7f19ac0c9e10 2024-03-18T16:30:43.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.761+0000 7f19b4600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f19ac0ca8a0 con 0x7f19ac154690 2024-03-18T16:30:43.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.762+0000 7f19b1200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19ac154690 0x7f19ac152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:43.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.762+0000 7f19b2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 0x7f19ac1e3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:43.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.762+0000 7f19b2600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 0x7f19ac1e3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46860/0 (socket says 172.21.15.102:46860) 2024-03-18T16:30:43.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.762+0000 7f19b2600640 1 -- 172.21.15.102:0/3598557732 learned_addr learned my addr 172.21.15.102:0/3598557732 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:43.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.767+0000 7f19b1c00640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19ac0c9e10 0x7f19ac157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:43.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.768+0000 7f19b2600640 1 -- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19ac154690 msgr2=0x7f19ac152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:43.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.768+0000 7f19b2600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19ac154690 0x7f19ac152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:43.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.768+0000 7f19b2600640 1 -- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19ac0c9e10 msgr2=0x7f19ac157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:43.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.768+0000 7f19b2600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19ac0c9e10 0x7f19ac157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:43.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.768+0000 7f19b2600640 1 -- 172.21.15.102:0/3598557732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac15d7c0 con 0x7f19ac154a70 2024-03-18T16:30:43.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.798+0000 7f19b2600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 0x7f19ac1e3ea0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f19a400c7b0 tx=0x7f19a400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.878+0000 7f19a9600640 1 -- 172.21.15.102:0/3598557732 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a4017070 con 0x7f19ac154a70 2024-03-18T16:30:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.878+0000 7f19a9600640 1 -- 172.21.15.102:0/3598557732 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a4003da0 con 0x7f19ac154a70 2024-03-18T16:30:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.878+0000 7f19a9600640 1 -- 172.21.15.102:0/3598557732 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a40052b0 con 0x7f19ac154a70 2024-03-18T16:30:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.878+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19ac15c760 con 0x7f19ac154a70 2024-03-18T16:30:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.878+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19ac15ca00 con 0x7f19ac154a70 2024-03-18T16:30:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:43.880+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19ac074fd0 con 0x7f19ac154a70 2024-03-18T16:30:44.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.204+0000 7f19a9600640 1 -- 172.21.15.102:0/3598557732 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f19a401a070 con 0x7f19ac154a70 2024-03-18T16:30:44.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.205+0000 7f19a9600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f198c06bc20 0x7f198c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:44.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.206+0000 7f19b1c00640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f198c06bc20 0x7f198c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:44.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.207+0000 7f19b1c00640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f198c06bc20 0x7f198c06e0e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f19a000a810 tx=0x7f19a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:44.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.227+0000 7f19a9600640 1 -- 172.21.15.102:0/3598557732 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f19a4056a20 con 0x7f19ac154a70 2024-03-18T16:30:44.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.228+0000 7f19a9600640 1 -- 172.21.15.102:0/3598557732 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f19a4021020 con 0x7f19ac154a70 2024-03-18T16:30:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.469+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19ac0730f0 con 0x7f19ac154a70 2024-03-18T16:30:44.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.492+0000 7f19a9600640 1 -- 172.21.15.102:0/3598557732 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f19a4058980 con 0x7f19ac154a70 2024-03-18T16:30:44.495 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:44.495 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:44.495 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:44.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f198c06bc20 msgr2=0x7f198c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f198c06bc20 0x7f198c06e0e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f19a000a810 tx=0x7f19a0008040 comp rx=0 tx=0).stop 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 msgr2=0x7f19ac1e3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 0x7f19ac1e3ea0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f19a400c7b0 tx=0x7f19a400cc70 comp rx=0 tx=0).stop 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 shutdown_connections 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f198c06bc20 0x7f198c06e0e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19ac154a70 0x7f19ac1e3ea0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19ac154690 0x7f19ac152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 --2- 172.21.15.102:0/3598557732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19ac0c9e10 0x7f19ac157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 >> 172.21.15.102:0/3598557732 conn(0x7f19ac0700d0 msgr2=0x7f19ac06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.495+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 shutdown_connections 2024-03-18T16:30:44.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:44.496+0000 7f19b4600640 1 -- 172.21.15.102:0/3598557732 wait complete. 2024-03-18T16:30:45.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:45.117 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:46.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:46.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.290+0000 7fe967e00640 1 Processor -- start 2024-03-18T16:30:46.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.291+0000 7fe967e00640 1 -- start start 2024-03-18T16:30:46.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.293+0000 7fe967e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe968154490 0x7fe96815e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:46.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.293+0000 7fe967e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe9680c3180 0x7fe9680c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:46.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.293+0000 7fe967e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe96815e6f0 con 0x7fe96806fcb0 2024-03-18T16:30:46.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.293+0000 7fe967e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe96815a3c0 con 0x7fe9680c3180 2024-03-18T16:30:46.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.293+0000 7fe967e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe96815a6a0 con 0x7fe968154490 2024-03-18T16:30:46.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.294+0000 7fe966000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe96806fcb0 0x7fe9680cdac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:47848/0 (socket says 172.21.15.102:47848) 2024-03-18T16:30:46.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.294+0000 7fe966000640 1 -- 172.21.15.102:0/1069839921 learned_addr learned my addr 172.21.15.102:0/1069839921 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:46.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.294+0000 7fe966a00640 1 --2- 172.21.15.102:0/1069839921 >> v2:172.21.15.154:3300/0 conn(0x7fe968154490 0x7fe96815e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:46.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.296+0000 7fe967400640 1 --2- 172.21.15.102:0/1069839921 >> v2:172.21.15.102:3300/0 conn(0x7fe9680c3180 0x7fe9680c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:46.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.299+0000 7fe967400640 1 -- 172.21.15.102:0/1069839921 >> v2:172.21.15.154:3300/0 conn(0x7fe968154490 msgr2=0x7fe96815e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:46.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.299+0000 7fe967400640 1 --2- 172.21.15.102:0/1069839921 >> v2:172.21.15.154:3300/0 conn(0x7fe968154490 0x7fe96815e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.299+0000 7fe967400640 1 -- 172.21.15.102:0/1069839921 >> v1:172.21.15.154:6789/0 conn(0x7fe96806fcb0 legacy=0x7fe9680cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:46.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.299+0000 7fe967400640 1 -- 172.21.15.102:0/1069839921 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96815a980 con 0x7fe9680c3180 2024-03-18T16:30:46.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.302+0000 7fe967400640 1 --2- 172.21.15.102:0/1069839921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9680c3180 0x7fe9680c35a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe9680ce5c0 tx=0x7fe96002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966eb59322ed1b57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:46.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.307+0000 7fe965600640 1 -- 172.21.15.102:0/1069839921 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960039070 con 0x7fe9680c3180 2024-03-18T16:30:46.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.307+0000 7fe965600640 1 -- 172.21.15.102:0/1069839921 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96002cc70 con 0x7fe9680c3180 2024-03-18T16:30:46.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.307+0000 7fe965600640 1 -- 172.21.15.102:0/1069839921 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960034660 con 0x7fe9680c3180 2024-03-18T16:30:46.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.308+0000 7fe967e00640 1 -- 172.21.15.102:0/1069839921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9680c3180 msgr2=0x7fe9680c35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:46.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.308+0000 7fe967e00640 1 --2- 172.21.15.102:0/1069839921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9680c3180 0x7fe9680c35a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe9680ce5c0 tx=0x7fe96002bec0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.309+0000 7fe967e00640 1 -- 172.21.15.102:0/1069839921 shutdown_connections 2024-03-18T16:30:46.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.309+0000 7fe967e00640 1 --2- 172.21.15.102:0/1069839921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9680c3180 0x7fe9680c35a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.309+0000 7fe967e00640 1 --2- 172.21.15.102:0/1069839921 >> v2:172.21.15.154:3300/0 conn(0x7fe968154490 0x7fe96815e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.309+0000 7fe967e00640 1 -- 172.21.15.102:0/1069839921 >> 172.21.15.102:0/1069839921 conn(0x7fe968066f80 msgr2=0x7fe968067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:46.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.309+0000 7fe967e00640 1 -- 172.21.15.102:0/1069839921 shutdown_connections 2024-03-18T16:30:46.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.309+0000 7fe967e00640 1 -- 172.21.15.102:0/1069839921 wait complete. 2024-03-18T16:30:46.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.311+0000 7fe967e00640 1 Processor -- start 2024-03-18T16:30:46.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.312+0000 7fe967e00640 1 -- start start 2024-03-18T16:30:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.313+0000 7fe967e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe96806fcb0 0x7fe96814a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.313+0000 7fe967e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9680c3180 0x7fe96814a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.313+0000 7fe967e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe968154490 0x7fe9681efc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.313+0000 7fe967e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe9680c3ae0 con 0x7fe968154490 2024-03-18T16:30:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.313+0000 7fe967e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe96815af40 con 0x7fe96806fcb0 2024-03-18T16:30:46.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.313+0000 7fe967e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe968154870 con 0x7fe9680c3180 2024-03-18T16:30:46.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.314+0000 7fe966a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe96806fcb0 0x7fe96814a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:46.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.314+0000 7fe966000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9680c3180 0x7fe96814a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:46.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.314+0000 7fe967400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe968154490 0x7fe9681efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:46.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.314+0000 7fe966a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe96806fcb0 0x7fe96814a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:47838/0 (socket says 172.21.15.102:47838) 2024-03-18T16:30:46.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.314+0000 7fe966a00640 1 -- 172.21.15.102:0/3539612848 learned_addr learned my addr 172.21.15.102:0/3539612848 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:46.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.317+0000 7fe966a00640 1 -- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9680c3180 msgr2=0x7fe96814a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:46.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.317+0000 7fe966a00640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9680c3180 0x7fe96814a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.317+0000 7fe966a00640 1 -- 172.21.15.102:0/3539612848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe968154490 msgr2=0x7fe9681efc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:46.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.317+0000 7fe966a00640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe968154490 0x7fe9681efc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.317+0000 7fe966a00640 1 -- 172.21.15.102:0/3539612848 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9680ca470 con 0x7fe96806fcb0 2024-03-18T16:30:46.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.320+0000 7fe966a00640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe96806fcb0 0x7fe96814a310 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe95800e870 tx=0x7fe95800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.324+0000 7fe95f400640 1 -- 172.21.15.102:0/3539612848 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe958017070 con 0x7fe96806fcb0 2024-03-18T16:30:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.324+0000 7fe95f400640 1 -- 172.21.15.102:0/3539612848 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe95800c940 con 0x7fe96806fcb0 2024-03-18T16:30:46.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.324+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe968155110 con 0x7fe96806fcb0 2024-03-18T16:30:46.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.324+0000 7fe95f400640 1 -- 172.21.15.102:0/3539612848 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe95800cd30 con 0x7fe96806fcb0 2024-03-18T16:30:46.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.325+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe96815a050 con 0x7fe96806fcb0 2024-03-18T16:30:46.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.326+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe968155310 con 0x7fe96806fcb0 2024-03-18T16:30:46.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.595+0000 7fe95f400640 1 -- 172.21.15.102:0/3539612848 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe95801a070 con 0x7fe96806fcb0 2024-03-18T16:30:46.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.597+0000 7fe95f400640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe94806bc20 0x7fe94806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:46.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.598+0000 7fe966000640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe94806bc20 0x7fe94806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:46.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.598+0000 7fe966000640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe94806bc20 0x7fe94806e0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe968151930 tx=0x7fe954007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:46.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.600+0000 7fe95f400640 1 -- 172.21.15.102:0/3539612848 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe958057630 con 0x7fe96806fcb0 2024-03-18T16:30:46.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.615+0000 7fe95f400640 1 -- 172.21.15.102:0/3539612848 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe968155310 con 0x7fe96806fcb0 2024-03-18T16:30:46.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.855+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe968074fd0 con 0x7fe96806fcb0 2024-03-18T16:30:46.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.868+0000 7fe95f400640 1 -- 172.21.15.102:0/3539612848 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe958059590 con 0x7fe96806fcb0 2024-03-18T16:30:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:46.870 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:46.870 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.871+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe94806bc20 msgr2=0x7fe94806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.871+0000 7fe967e00640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe94806bc20 0x7fe94806e0e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe968151930 tx=0x7fe954007e80 comp rx=0 tx=0).stop 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.871+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe96806fcb0 msgr2=0x7fe96814a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.871+0000 7fe967e00640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe96806fcb0 0x7fe96814a310 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fe95800e870 tx=0x7fe95800ed30 comp rx=0 tx=0).stop 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.872+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 shutdown_connections 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.872+0000 7fe967e00640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe94806bc20 0x7fe94806e0e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.872+0000 7fe967e00640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe968154490 0x7fe9681efc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.872+0000 7fe967e00640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe9680c3180 0x7fe96814a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.872+0000 7fe967e00640 1 --2- 172.21.15.102:0/3539612848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe96806fcb0 0x7fe96814a310 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.872+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 >> 172.21.15.102:0/3539612848 conn(0x7fe968066f80 msgr2=0x7fe968151020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.872+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 shutdown_connections 2024-03-18T16:30:46.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:46.872+0000 7fe967e00640 1 -- 172.21.15.102:0/3539612848 wait complete. 2024-03-18T16:30:47.502 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:47.503 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:48.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.683+0000 7f7a95000640 1 Processor -- start 2024-03-18T16:30:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.683+0000 7f7a95000640 1 -- start start 2024-03-18T16:30:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.684+0000 7f7a95000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7a9015c3a0 0x7f7a90163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.684+0000 7f7a95000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a900c72d0 con 0x7f7a90141650 2024-03-18T16:30:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.684+0000 7f7a95000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a90158840 con 0x7f7a90158420 2024-03-18T16:30:48.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.685+0000 7f7a95000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7a90159370 con 0x7f7a9015c3a0 2024-03-18T16:30:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.685+0000 7f7a8ea00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f7a90158420 0x7f7a900c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:47860/0 (socket says 172.21.15.102:47860) 2024-03-18T16:30:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.685+0000 7f7a8ea00640 1 -- 172.21.15.102:0/3655085307 learned_addr learned my addr 172.21.15.102:0/3655085307 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:48.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.685+0000 7f7a8fe00640 1 --2- 172.21.15.102:0/3655085307 >> v2:172.21.15.102:3300/0 conn(0x7f7a9015c3a0 0x7f7a90163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:48.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.688+0000 7f7a8e000640 1 -- 172.21.15.102:0/3655085307 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2595904796 0 0) 0x7f7a90158840 con 0x7f7a90158420 2024-03-18T16:30:48.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.688+0000 7f7a8e000640 1 -- 172.21.15.102:0/3655085307 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a74003520 con 0x7f7a90158420 2024-03-18T16:30:48.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.689+0000 7f7a8fe00640 1 -- 172.21.15.102:0/3655085307 >> v1:172.21.15.154:6789/0 conn(0x7f7a90158420 legacy=0x7f7a900c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:48.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.689+0000 7f7a8fe00640 1 -- 172.21.15.102:0/3655085307 >> v1:172.21.15.102:6789/0 conn(0x7f7a90141650 legacy=0x7f7a900c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:48.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.690+0000 7f7a8fe00640 1 -- 172.21.15.102:0/3655085307 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a90158ad0 con 0x7f7a9015c3a0 2024-03-18T16:30:48.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.695+0000 7f7a8fe00640 1 --2- 172.21.15.102:0/3655085307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a9015c3a0 0x7f7a90163cd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009d90 tx=0x7f7a8802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d18dccc022e933e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:48.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.700+0000 7f7a8e000640 1 -- 172.21.15.102:0/3655085307 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88039070 con 0x7f7a9015c3a0 2024-03-18T16:30:48.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.700+0000 7f7a8e000640 1 -- 172.21.15.102:0/3655085307 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a8802ccf0 con 0x7f7a9015c3a0 2024-03-18T16:30:48.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.700+0000 7f7a8e000640 1 -- 172.21.15.102:0/3655085307 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88034660 con 0x7f7a9015c3a0 2024-03-18T16:30:48.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.701+0000 7f7a95000640 1 -- 172.21.15.102:0/3655085307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a9015c3a0 msgr2=0x7f7a90163cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:48.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.701+0000 7f7a95000640 1 --2- 172.21.15.102:0/3655085307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a9015c3a0 0x7f7a90163cd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009d90 tx=0x7f7a8802bf40 comp rx=0 tx=0).stop 2024-03-18T16:30:48.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.701+0000 7f7a95000640 1 -- 172.21.15.102:0/3655085307 shutdown_connections 2024-03-18T16:30:48.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.701+0000 7f7a95000640 1 --2- 172.21.15.102:0/3655085307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a9015c3a0 0x7f7a90163cd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:48.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.701+0000 7f7a95000640 1 -- 172.21.15.102:0/3655085307 >> 172.21.15.102:0/3655085307 conn(0x7f7a900cb890 msgr2=0x7f7a900cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:48.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.701+0000 7f7a95000640 1 -- 172.21.15.102:0/3655085307 shutdown_connections 2024-03-18T16:30:48.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.702+0000 7f7a95000640 1 -- 172.21.15.102:0/3655085307 wait complete. 2024-03-18T16:30:48.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.704+0000 7f7a95000640 1 Processor -- start 2024-03-18T16:30:48.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.704+0000 7f7a95000640 1 -- start start 2024-03-18T16:30:48.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.706+0000 7f7a95000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a90141650 0x7f7a9013e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:48.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.706+0000 7f7a95000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90158420 0x7f7a9013e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:48.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.706+0000 7f7a95000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a9015c3a0 0x7f7a901e3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:48.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.706+0000 7f7a95000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7a900c63c0 con 0x7f7a90141650 2024-03-18T16:30:48.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.706+0000 7f7a95000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7a90159090 con 0x7f7a90158420 2024-03-18T16:30:48.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.706+0000 7f7a95000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7a900c66a0 con 0x7f7a9015c3a0 2024-03-18T16:30:48.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.707+0000 7f7a8f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a90141650 0x7f7a9013e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:48.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.707+0000 7f7a8ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90158420 0x7f7a9013e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:48.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.707+0000 7f7a8fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a9015c3a0 0x7f7a901e3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:48.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.707+0000 7f7a8f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a90141650 0x7f7a9013e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46896/0 (socket says 172.21.15.102:46896) 2024-03-18T16:30:48.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.707+0000 7f7a8f400640 1 -- 172.21.15.102:0/3676464314 learned_addr learned my addr 172.21.15.102:0/3676464314 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:48.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.710+0000 7f7a8ea00640 1 -- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a9015c3a0 msgr2=0x7f7a901e3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:48.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.710+0000 7f7a8ea00640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a9015c3a0 0x7f7a901e3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:48.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.710+0000 7f7a8ea00640 1 -- 172.21.15.102:0/3676464314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a90141650 msgr2=0x7f7a9013e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:48.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.710+0000 7f7a8ea00640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a90141650 0x7f7a9013e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:48.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.710+0000 7f7a8ea00640 1 -- 172.21.15.102:0/3676464314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a90155b20 con 0x7f7a90158420 2024-03-18T16:30:48.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.713+0000 7f7a8ea00640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90158420 0x7f7a9013e630 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c430 tx=0x7f7a7c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:48.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.716+0000 7f7a87400640 1 -- 172.21.15.102:0/3676464314 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c017070 con 0x7f7a90158420 2024-03-18T16:30:48.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.717+0000 7f7a87400640 1 -- 172.21.15.102:0/3676464314 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a7c005ce0 con 0x7f7a90158420 2024-03-18T16:30:48.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.717+0000 7f7a87400640 1 -- 172.21.15.102:0/3676464314 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c0072d0 con 0x7f7a90158420 2024-03-18T16:30:48.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.717+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a901e26c0 con 0x7f7a90158420 2024-03-18T16:30:48.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.717+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a901e2c80 con 0x7f7a90158420 2024-03-18T16:30:48.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.718+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a901573b0 con 0x7f7a90158420 2024-03-18T16:30:48.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.738+0000 7f7a87400640 1 -- 172.21.15.102:0/3676464314 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7a7c01a070 con 0x7f7a90158420 2024-03-18T16:30:48.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.739+0000 7f7a87400640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7a7006bc20 0x7f7a7006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:48.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.740+0000 7f7a87400640 1 -- 172.21.15.102:0/3676464314 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7a7c08a690 con 0x7f7a90158420 2024-03-18T16:30:48.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.740+0000 7f7a8f400640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7a7006bc20 0x7f7a7006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:48.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.740+0000 7f7a8f400640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7a7006bc20 0x7f7a7006e0e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7a80001840 tx=0x7f7a80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:48.990+0000 7f7a87400640 1 -- 172.21.15.102:0/3676464314 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7a901573b0 con 0x7f7a90158420 2024-03-18T16:30:49.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.235+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a90074fd0 con 0x7f7a90158420 2024-03-18T16:30:49.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.318+0000 7f7a87400640 1 -- 172.21.15.102:0/3676464314 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7a7c058720 con 0x7f7a90158420 2024-03-18T16:30:49.320 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:49.320 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:49.320 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.320+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7a7006bc20 msgr2=0x7f7a7006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.320+0000 7f7a95000640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7a7006bc20 0x7f7a7006e0e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7a80001840 tx=0x7f7a80009040 comp rx=0 tx=0).stop 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.320+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90158420 msgr2=0x7f7a9013e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.320+0000 7f7a95000640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90158420 0x7f7a9013e630 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c430 tx=0x7f7a7c00c8f0 comp rx=0 tx=0).stop 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.320+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 shutdown_connections 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.321+0000 7f7a95000640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7a7006bc20 0x7f7a7006e0e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.321+0000 7f7a95000640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7a9015c3a0 0x7f7a901e3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.321+0000 7f7a95000640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a90158420 0x7f7a9013e630 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.321+0000 7f7a95000640 1 --2- 172.21.15.102:0/3676464314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a90141650 0x7f7a9013e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.321+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 >> 172.21.15.102:0/3676464314 conn(0x7f7a900cb890 msgr2=0x7f7a90157a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.321+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 shutdown_connections 2024-03-18T16:30:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:49.321+0000 7f7a95000640 1 -- 172.21.15.102:0/3676464314 wait complete. 2024-03-18T16:30:49.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:49.945 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:50.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:51.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.117+0000 7fed15800640 1 Processor -- start 2024-03-18T16:30:51.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.117+0000 7fed15800640 1 -- start start 2024-03-18T16:30:51.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.118+0000 7fed15800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fed1015a310 0x7fed1006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:51.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.118+0000 7fed15800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a6f0 0x7fed1015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:51.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.118+0000 7fed15800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed100ca500 con 0x7fed1015baa0 2024-03-18T16:30:51.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.118+0000 7fed15800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fed100cd410 con 0x7fed1015a310 2024-03-18T16:30:51.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.118+0000 7fed15800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fed1015b050 con 0x7fed1015a6f0 2024-03-18T16:30:51.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.119+0000 7fed14e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fed1015baa0 0x7fed1015bf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:37830/0 (socket says 172.21.15.102:37830) 2024-03-18T16:30:51.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.119+0000 7fed14e00640 1 -- 172.21.15.102:0/3274184435 learned_addr learned my addr 172.21.15.102:0/3274184435 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:51.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.119+0000 7fed0fe00640 1 --2- 172.21.15.102:0/3274184435 >> v2:172.21.15.102:3300/0 conn(0x7fed1015a310 0x7fed1006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:51.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.119+0000 7fed0f400640 1 --2- 172.21.15.102:0/3274184435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a6f0 0x7fed1015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:51.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.122+0000 7fed0ea00640 1 -- 172.21.15.102:0/3274184435 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747860799 0 0) 0x7fed100ca500 con 0x7fed1015baa0 2024-03-18T16:30:51.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.122+0000 7fed0ea00640 1 -- 172.21.15.102:0/3274184435 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fecf4003520 con 0x7fed1015baa0 2024-03-18T16:30:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.122+0000 7fed0fe00640 1 -- 172.21.15.102:0/3274184435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a6f0 msgr2=0x7fed1015ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.122+0000 7fed0fe00640 1 --2- 172.21.15.102:0/3274184435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a6f0 0x7fed1015ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.122+0000 7fed0fe00640 1 -- 172.21.15.102:0/3274184435 >> v1:172.21.15.154:6789/0 conn(0x7fed1015baa0 legacy=0x7fed1015bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:51.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.122+0000 7fed0fe00640 1 -- 172.21.15.102:0/3274184435 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed100ca7a0 con 0x7fed1015a310 2024-03-18T16:30:51.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.125+0000 7fed0fe00640 1 --2- 172.21.15.102:0/3274184435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed1015a310 0x7fed1006d3e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fed000097a0 tx=0x7fed0002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9223a569fbf57016 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:51.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.127+0000 7fed0ea00640 1 -- 172.21.15.102:0/3274184435 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00035020 con 0x7fed1015a310 2024-03-18T16:30:51.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.130+0000 7fed0ea00640 1 -- 172.21.15.102:0/3274184435 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed00031810 con 0x7fed1015a310 2024-03-18T16:30:51.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.130+0000 7fed0ea00640 1 -- 172.21.15.102:0/3274184435 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00031bc0 con 0x7fed1015a310 2024-03-18T16:30:51.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.131+0000 7fed15800640 1 -- 172.21.15.102:0/3274184435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed1015a310 msgr2=0x7fed1006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:51.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.131+0000 7fed15800640 1 --2- 172.21.15.102:0/3274184435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed1015a310 0x7fed1006d3e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fed000097a0 tx=0x7fed0002b8b0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.131+0000 7fed15800640 1 -- 172.21.15.102:0/3274184435 shutdown_connections 2024-03-18T16:30:51.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.131+0000 7fed15800640 1 --2- 172.21.15.102:0/3274184435 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a6f0 0x7fed1015ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.131+0000 7fed15800640 1 --2- 172.21.15.102:0/3274184435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed1015a310 0x7fed1006d3e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.131+0000 7fed15800640 1 -- 172.21.15.102:0/3274184435 >> 172.21.15.102:0/3274184435 conn(0x7fed1008c1a0 msgr2=0x7fed10066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:51.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.131+0000 7fed15800640 1 -- 172.21.15.102:0/3274184435 shutdown_connections 2024-03-18T16:30:51.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.131+0000 7fed15800640 1 -- 172.21.15.102:0/3274184435 wait complete. 2024-03-18T16:30:51.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.133+0000 7fed15800640 1 Processor -- start 2024-03-18T16:30:51.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.133+0000 7fed15800640 1 -- start start 2024-03-18T16:30:51.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.134+0000 7fed15800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a310 0x7fed100c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:51.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.134+0000 7fed15800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed1015a6f0 0x7fed101f06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:51.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.134+0000 7fed15800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1015baa0 0x7fed1014dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:51.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.134+0000 7fed15800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fed100cd130 con 0x7fed1015a6f0 2024-03-18T16:30:51.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.134+0000 7fed15800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed100cce50 con 0x7fed1015baa0 2024-03-18T16:30:51.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.134+0000 7fed15800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fed1006d920 con 0x7fed1015a310 2024-03-18T16:30:51.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.135+0000 7fed14e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1015baa0 0x7fed1014dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:51.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.135+0000 7fed0fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a310 0x7fed100c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:51.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.135+0000 7fed0f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed1015a6f0 0x7fed101f06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:51.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.135+0000 7fed14e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1015baa0 0x7fed1014dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:50702/0 (socket says 172.21.15.102:50702) 2024-03-18T16:30:51.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.135+0000 7fed14e00640 1 -- 172.21.15.102:0/1336764087 learned_addr learned my addr 172.21.15.102:0/1336764087 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:51.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.139+0000 7fed14e00640 1 -- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a310 msgr2=0x7fed100c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:51.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.139+0000 7fed14e00640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a310 0x7fed100c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.139+0000 7fed14e00640 1 -- 172.21.15.102:0/1336764087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed1015a6f0 msgr2=0x7fed101f06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:51.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.139+0000 7fed14e00640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed1015a6f0 0x7fed101f06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.139+0000 7fed14e00640 1 -- 172.21.15.102:0/1336764087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed101eff10 con 0x7fed1015baa0 2024-03-18T16:30:51.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.141+0000 7fed14e00640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1015baa0 0x7fed1014dde0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fed0800caa0 tx=0x7fed0800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:51.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.145+0000 7fed07e00640 1 -- 172.21.15.102:0/1336764087 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08017070 con 0x7fed1015baa0 2024-03-18T16:30:51.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.145+0000 7fed07e00640 1 -- 172.21.15.102:0/1336764087 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0800ac00 con 0x7fed1015baa0 2024-03-18T16:30:51.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.145+0000 7fed07e00640 1 -- 172.21.15.102:0/1336764087 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08012400 con 0x7fed1015baa0 2024-03-18T16:30:51.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.145+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed100c3ba0 con 0x7fed1015baa0 2024-03-18T16:30:51.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.145+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed100c8410 con 0x7fed1015baa0 2024-03-18T16:30:51.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.146+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed10074fd0 con 0x7fed1015baa0 2024-03-18T16:30:51.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.165+0000 7fed07e00640 1 -- 172.21.15.102:0/1336764087 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fed0801a070 con 0x7fed1015baa0 2024-03-18T16:30:51.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.166+0000 7fed07e00640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fecf006bc20 0x7fecf006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:51.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.167+0000 7fed0fe00640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fecf006bc20 0x7fecf006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:51.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.167+0000 7fed0fe00640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fecf006bc20 0x7fecf006e0e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fed00036000 tx=0x7fed00008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:51.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.169+0000 7fed07e00640 1 -- 172.21.15.102:0/1336764087 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fed08057100 con 0x7fed1015baa0 2024-03-18T16:30:51.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.429+0000 7fed07e00640 1 -- 172.21.15.102:0/1336764087 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fed08021020 con 0x7fed1015baa0 2024-03-18T16:30:51.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.674+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed100b5b00 con 0x7fed1015baa0 2024-03-18T16:30:51.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.802+0000 7fed07e00640 1 -- 172.21.15.102:0/1336764087 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fed08059060 con 0x7fed1015baa0 2024-03-18T16:30:51.804 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:51.804 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:51.804 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.805+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fecf006bc20 msgr2=0x7fecf006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.805+0000 7fed15800640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fecf006bc20 0x7fecf006e0e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fed00036000 tx=0x7fed00008030 comp rx=0 tx=0).stop 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.805+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1015baa0 msgr2=0x7fed1014dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.805+0000 7fed15800640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1015baa0 0x7fed1014dde0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fed0800caa0 tx=0x7fed0800cf60 comp rx=0 tx=0).stop 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.806+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 shutdown_connections 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.806+0000 7fed15800640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fecf006bc20 0x7fecf006e0e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.806+0000 7fed15800640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1015baa0 0x7fed1014dde0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.806+0000 7fed15800640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed1015a6f0 0x7fed101f06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.806+0000 7fed15800640 1 --2- 172.21.15.102:0/1336764087 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fed1015a310 0x7fed100c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.806+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 >> 172.21.15.102:0/1336764087 conn(0x7fed1008c1a0 msgr2=0x7fed101e3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.806+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 shutdown_connections 2024-03-18T16:30:51.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:51.806+0000 7fed15800640 1 -- 172.21.15.102:0/1336764087 wait complete. 2024-03-18T16:30:52.430 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:52.431 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:52.966 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:30:52.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:30:53.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.606+0000 7fa116a00640 1 Processor -- start 2024-03-18T16:30:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.606+0000 7fa116a00640 1 -- start start 2024-03-18T16:30:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.607+0000 7fa116a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa1100cdbd0 0x7fa1100c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.607+0000 7fa116a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1100ce0a0 con 0x7fa110154730 2024-03-18T16:30:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.607+0000 7fa116a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1100ce380 con 0x7fa11015d8c0 2024-03-18T16:30:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.607+0000 7fa116a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa11015a830 con 0x7fa1100cdbd0 2024-03-18T16:30:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.608+0000 7fa114a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa11015d8c0 0x7fa1100c8900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:37836/0 (socket says 172.21.15.102:37836) 2024-03-18T16:30:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.608+0000 7fa114a00640 1 -- 172.21.15.102:0/3016537872 learned_addr learned my addr 172.21.15.102:0/3016537872 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.608+0000 7fa10f400640 1 --2- 172.21.15.102:0/3016537872 >> v2:172.21.15.154:3300/0 conn(0x7fa1100cdbd0 0x7fa1100c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.612+0000 7fa10f400640 1 -- 172.21.15.102:0/3016537872 >> v1:172.21.15.154:6789/0 conn(0x7fa11015d8c0 legacy=0x7fa1100c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.612+0000 7fa10f400640 1 -- 172.21.15.102:0/3016537872 >> v1:172.21.15.102:6789/0 conn(0x7fa110154730 legacy=0x7fa11015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.612+0000 7fa10f400640 1 -- 172.21.15.102:0/3016537872 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1100ce660 con 0x7fa1100cdbd0 2024-03-18T16:30:53.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.617+0000 7fa10f400640 1 --2- 172.21.15.102:0/3016537872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1100cdbd0 0x7fa1100c96b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc0097a0 tx=0x7fa0fc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bafc5d9b30e7f8a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.693+0000 7fa10ea00640 1 -- 172.21.15.102:0/3016537872 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc0315a0 con 0x7fa1100cdbd0 2024-03-18T16:30:53.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.693+0000 7fa10ea00640 1 -- 172.21.15.102:0/3016537872 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0fc037070 con 0x7fa1100cdbd0 2024-03-18T16:30:53.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.693+0000 7fa10ea00640 1 -- 172.21.15.102:0/3016537872 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc031960 con 0x7fa1100cdbd0 2024-03-18T16:30:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.694+0000 7fa116a00640 1 -- 172.21.15.102:0/3016537872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1100cdbd0 msgr2=0x7fa1100c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.694+0000 7fa116a00640 1 --2- 172.21.15.102:0/3016537872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1100cdbd0 0x7fa1100c96b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc0097a0 tx=0x7fa0fc02b840 comp rx=0 tx=0).stop 2024-03-18T16:30:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.694+0000 7fa116a00640 1 -- 172.21.15.102:0/3016537872 shutdown_connections 2024-03-18T16:30:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.694+0000 7fa116a00640 1 --2- 172.21.15.102:0/3016537872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1100cdbd0 0x7fa1100c96b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.694+0000 7fa116a00640 1 -- 172.21.15.102:0/3016537872 >> 172.21.15.102:0/3016537872 conn(0x7fa110066f80 msgr2=0x7fa110067390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.694+0000 7fa116a00640 1 -- 172.21.15.102:0/3016537872 shutdown_connections 2024-03-18T16:30:53.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.694+0000 7fa116a00640 1 -- 172.21.15.102:0/3016537872 wait complete. 2024-03-18T16:30:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.696+0000 7fa116a00640 1 Processor -- start 2024-03-18T16:30:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.696+0000 7fa116a00640 1 -- start start 2024-03-18T16:30:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.697+0000 7fa116a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1100cdbd0 0x7fa1101f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.697+0000 7fa116a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa110154730 0x7fa11014e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.697+0000 7fa116a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa11015d8c0 0x7fa11015d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.697+0000 7fa116a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa1100c9190 con 0x7fa110154730 2024-03-18T16:30:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.697+0000 7fa116a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1100c9010 con 0x7fa1100cdbd0 2024-03-18T16:30:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.697+0000 7fa116a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa1100c9bf0 con 0x7fa11015d8c0 2024-03-18T16:30:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.698+0000 7fa114a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa11015d8c0 0x7fa11015d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.698+0000 7fa114a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa11015d8c0 0x7fa11015d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52402/0 (socket says 172.21.15.102:52402) 2024-03-18T16:30:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.698+0000 7fa114a00640 1 -- 172.21.15.102:0/900362450 learned_addr learned my addr 172.21.15.102:0/900362450 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.699+0000 7fa10f400640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa110154730 0x7fa11014e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.699+0000 7fa10fe00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1100cdbd0 0x7fa1101f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.703+0000 7fa114a00640 1 -- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1100cdbd0 msgr2=0x7fa1101f35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.703+0000 7fa114a00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1100cdbd0 0x7fa1101f35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.703+0000 7fa114a00640 1 -- 172.21.15.102:0/900362450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa110154730 msgr2=0x7fa11014e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.703+0000 7fa114a00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa110154730 0x7fa11014e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:53.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.703+0000 7fa114a00640 1 -- 172.21.15.102:0/900362450 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1101533c0 con 0x7fa11015d8c0 2024-03-18T16:30:53.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.706+0000 7fa114a00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa11015d8c0 0x7fa11015d1e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa10800ca20 tx=0x7fa10800cee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:53.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.710+0000 7fa107e00640 1 -- 172.21.15.102:0/900362450 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108017070 con 0x7fa11015d8c0 2024-03-18T16:30:53.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.711+0000 7fa107e00640 1 -- 172.21.15.102:0/900362450 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa10800ab70 con 0x7fa11015d8c0 2024-03-18T16:30:53.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.711+0000 7fa107e00640 1 -- 172.21.15.102:0/900362450 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108012400 con 0x7fa11015d8c0 2024-03-18T16:30:53.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.711+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa110152e00 con 0x7fa11015d8c0 2024-03-18T16:30:53.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.711+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa11015b5a0 con 0x7fa11015d8c0 2024-03-18T16:30:53.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.712+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1100b3e10 con 0x7fa11015d8c0 2024-03-18T16:30:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.743+0000 7fa107e00640 1 -- 172.21.15.102:0/900362450 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa10801a070 con 0x7fa11015d8c0 2024-03-18T16:30:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.745+0000 7fa107e00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0f006bc20 0x7fa0f006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.745+0000 7fa10fe00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0f006bc20 0x7fa0f006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.746+0000 7fa10fe00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0f006bc20 0x7fa0f006e0e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa1000017f0 tx=0x7fa100009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.748+0000 7fa107e00640 1 -- 172.21.15.102:0/900362450 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa108057080 con 0x7fa11015d8c0 2024-03-18T16:30:54.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:53.999+0000 7fa107e00640 1 -- 172.21.15.102:0/900362450 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa1100b3e10 con 0x7fa11015d8c0 2024-03-18T16:30:54.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.260+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa110074fd0 con 0x7fa11015d8c0 2024-03-18T16:30:54.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.273+0000 7fa107e00640 1 -- 172.21.15.102:0/900362450 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa108021020 con 0x7fa11015d8c0 2024-03-18T16:30:54.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:54.275 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:54.275 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0f006bc20 msgr2=0x7fa0f006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0f006bc20 0x7fa0f006e0e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa1000017f0 tx=0x7fa100009040 comp rx=0 tx=0).stop 2024-03-18T16:30:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa11015d8c0 msgr2=0x7fa11015d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa11015d8c0 0x7fa11015d1e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa10800ca20 tx=0x7fa10800cee0 comp rx=0 tx=0).stop 2024-03-18T16:30:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 shutdown_connections 2024-03-18T16:30:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0f006bc20 0x7fa0f006e0e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:54.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa11015d8c0 0x7fa11015d1e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa110154730 0x7fa11014e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 --2- 172.21.15.102:0/900362450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1100cdbd0 0x7fa1101f35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 >> 172.21.15.102:0/900362450 conn(0x7fa110066f80 msgr2=0x7fa11015dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 shutdown_connections 2024-03-18T16:30:54.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:54.276+0000 7fa116a00640 1 -- 172.21.15.102:0/900362450 wait complete. 2024-03-18T16:30:54.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:54.905 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:55.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:56.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.109+0000 7f712a800640 1 Processor -- start 2024-03-18T16:30:56.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.110+0000 7f712a800640 1 -- start start 2024-03-18T16:30:56.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.111+0000 7f712a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124071700 0x7f7124072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:56.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.111+0000 7f712a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7124161d30 0x7f7124072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:56.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.111+0000 7f712a800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71240c4190 con 0x7f7124158c00 2024-03-18T16:30:56.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.111+0000 7f712a800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7124158fe0 con 0x7f7124161d30 2024-03-18T16:30:56.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.111+0000 7f712a800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f712406f3b0 con 0x7f7124071700 2024-03-18T16:30:56.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.112+0000 7f7129e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7124161d30 0x7f7124072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:56.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.112+0000 7f7129400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f7124158c00 0x7f712406f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:37848/0 (socket says 172.21.15.102:37848) 2024-03-18T16:30:56.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.112+0000 7f7129400640 1 -- 172.21.15.102:0/857748032 learned_addr learned my addr 172.21.15.102:0/857748032 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:56.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.112+0000 7f7128a00640 1 --2- 172.21.15.102:0/857748032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124071700 0x7f7124072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.115+0000 7f7129e00640 1 -- 172.21.15.102:0/857748032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124071700 msgr2=0x7f7124072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:30:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.115+0000 7f7129e00640 1 --2- 172.21.15.102:0/857748032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124071700 0x7f7124072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.115+0000 7f7129e00640 1 -- 172.21.15.102:0/857748032 >> v1:172.21.15.154:6789/0 conn(0x7f7124158c00 legacy=0x7f712406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.115+0000 7f7129e00640 1 -- 172.21.15.102:0/857748032 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f712406f060 con 0x7f7124161d30 2024-03-18T16:30:56.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.117+0000 7f7129e00640 1 --2- 172.21.15.102:0/857748032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7124161d30 0x7f7124072fb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f711c009d90 tx=0x7f711c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6feef3f6a1eb63d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.122+0000 7f7123e00640 1 -- 172.21.15.102:0/857748032 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711c039070 con 0x7f7124161d30 2024-03-18T16:30:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.122+0000 7f7123e00640 1 -- 172.21.15.102:0/857748032 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711c02cc60 con 0x7f7124161d30 2024-03-18T16:30:56.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.122+0000 7f7123e00640 1 -- 172.21.15.102:0/857748032 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711c034660 con 0x7f7124161d30 2024-03-18T16:30:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.122+0000 7f712a800640 1 -- 172.21.15.102:0/857748032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7124161d30 msgr2=0x7f7124072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.122+0000 7f712a800640 1 --2- 172.21.15.102:0/857748032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7124161d30 0x7f7124072fb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f711c009d90 tx=0x7f711c02beb0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.123+0000 7f712a800640 1 -- 172.21.15.102:0/857748032 shutdown_connections 2024-03-18T16:30:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.123+0000 7f712a800640 1 --2- 172.21.15.102:0/857748032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7124161d30 0x7f7124072fb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.123+0000 7f712a800640 1 --2- 172.21.15.102:0/857748032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124071700 0x7f7124072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.123+0000 7f712a800640 1 -- 172.21.15.102:0/857748032 >> 172.21.15.102:0/857748032 conn(0x7f71240cb890 msgr2=0x7f71240cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.123+0000 7f712a800640 1 -- 172.21.15.102:0/857748032 shutdown_connections 2024-03-18T16:30:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.123+0000 7f712a800640 1 -- 172.21.15.102:0/857748032 wait complete. 2024-03-18T16:30:56.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.125+0000 7f712a800640 1 Processor -- start 2024-03-18T16:30:56.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.125+0000 7f712a800640 1 -- start start 2024-03-18T16:30:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.126+0000 7f712a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7124071700 0x7f7124160e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.126+0000 7f712a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124158c00 0x7f71241e8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.126+0000 7f712a800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7124161d30 0x7f71241e89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.126+0000 7f712a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7124072270 con 0x7f7124071700 2024-03-18T16:30:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.126+0000 7f712a800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7124071f90 con 0x7f7124161d30 2024-03-18T16:30:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.126+0000 7f712a800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7124072550 con 0x7f7124158c00 2024-03-18T16:30:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.126+0000 7f7128a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124158c00 0x7f71241e8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.127+0000 7f7129e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7124161d30 0x7f71241e89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.127+0000 7f7129400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7124071700 0x7f7124160e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.127+0000 7f7129e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7124161d30 0x7f71241e89c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:50738/0 (socket says 172.21.15.102:50738) 2024-03-18T16:30:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.127+0000 7f7129e00640 1 -- 172.21.15.102:0/651737807 learned_addr learned my addr 172.21.15.102:0/651737807 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:56.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.130+0000 7f7129e00640 1 -- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124158c00 msgr2=0x7f71241e8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:56.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.130+0000 7f7129e00640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124158c00 0x7f71241e8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.130+0000 7f7129e00640 1 -- 172.21.15.102:0/651737807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7124071700 msgr2=0x7f7124160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:56.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.130+0000 7f7129e00640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7124071700 0x7f7124160e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.130+0000 7f7129e00640 1 -- 172.21.15.102:0/651737807 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71241e7fa0 con 0x7f7124161d30 2024-03-18T16:30:56.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.133+0000 7f7129e00640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7124161d30 0x7f71241e89c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f711c009d90 tx=0x7f711c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.137+0000 7f7121600640 1 -- 172.21.15.102:0/651737807 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711c044020 con 0x7f7124161d30 2024-03-18T16:30:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.137+0000 7f7121600640 1 -- 172.21.15.102:0/651737807 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711c03f470 con 0x7f7124161d30 2024-03-18T16:30:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.137+0000 7f7121600640 1 -- 172.21.15.102:0/651737807 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711c039040 con 0x7f7124161d30 2024-03-18T16:30:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.137+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71240c5f80 con 0x7f7124161d30 2024-03-18T16:30:56.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.137+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71240c6520 con 0x7f7124161d30 2024-03-18T16:30:56.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.139+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7124074fd0 con 0x7f7124161d30 2024-03-18T16:30:56.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.165+0000 7f7121600640 1 -- 172.21.15.102:0/651737807 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f711c047020 con 0x7f7124161d30 2024-03-18T16:30:56.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.166+0000 7f7121600640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f710406bc20 0x7f710406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:56.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.167+0000 7f7121600640 1 -- 172.21.15.102:0/651737807 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f711c079d70 con 0x7f7124161d30 2024-03-18T16:30:56.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.167+0000 7f7129400640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f710406bc20 0x7f710406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:56.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.167+0000 7f7129400640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f710406bc20 0x7f710406e0e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7118007fd0 tx=0x7f7118007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:56.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.424+0000 7f7121600640 1 -- 172.21.15.102:0/651737807 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f711c04e030 con 0x7f7124161d30 2024-03-18T16:30:56.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.661+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7124072850 con 0x7f7124161d30 2024-03-18T16:30:56.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.683+0000 7f7121600640 1 -- 172.21.15.102:0/651737807 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f711c07bcd0 con 0x7f7124161d30 2024-03-18T16:30:56.685 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:56.685 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:56.686 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f710406bc20 msgr2=0x7f710406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f710406bc20 0x7f710406e0e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7118007fd0 tx=0x7f7118007e80 comp rx=0 tx=0).stop 2024-03-18T16:30:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7124161d30 msgr2=0x7f71241e89c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7124161d30 0x7f71241e89c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f711c009d90 tx=0x7f711c004a40 comp rx=0 tx=0).stop 2024-03-18T16:30:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 shutdown_connections 2024-03-18T16:30:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f710406bc20 0x7f710406e0e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7124161d30 0x7f71241e89c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7124158c00 0x7f71241e8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 --2- 172.21.15.102:0/651737807 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7124071700 0x7f7124160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 >> 172.21.15.102:0/651737807 conn(0x7f71240cb890 msgr2=0x7f71240c5510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 shutdown_connections 2024-03-18T16:30:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:56.686+0000 7f712a800640 1 -- 172.21.15.102:0/651737807 wait complete. 2024-03-18T16:30:57.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:57.308 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:30:58.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:30:58.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.483+0000 7f986f200640 1 Processor -- start 2024-03-18T16:30:58.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.483+0000 7f986f200640 1 -- start start 2024-03-18T16:30:58.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.484+0000 7f986f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f98680cacc0 0x7f98681589f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:58.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.484+0000 7f986f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98680c3150 con 0x7f9868152120 2024-03-18T16:30:58.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.484+0000 7f986f200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98680c3430 con 0x7f986815b4d0 2024-03-18T16:30:58.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.484+0000 7f986f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9868158560 con 0x7f98680cacc0 2024-03-18T16:30:58.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.485+0000 7f986d400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f986815b4d0 0x7f98680704c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:37862/0 (socket says 172.21.15.102:37862) 2024-03-18T16:30:58.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.485+0000 7f986d400640 1 -- 172.21.15.102:0/1304395846 learned_addr learned my addr 172.21.15.102:0/1304395846 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:30:58.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.485+0000 7f986e800640 1 --2- 172.21.15.102:0/1304395846 >> v2:172.21.15.102:3300/0 conn(0x7f98680cacc0 0x7f98681589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:58.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.488+0000 7f986ca00640 1 -- 172.21.15.102:0/1304395846 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4045690843 0 0) 0x7f98680c3430 con 0x7f986815b4d0 2024-03-18T16:30:58.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.488+0000 7f986ca00640 1 -- 172.21.15.102:0/1304395846 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9850003520 con 0x7f986815b4d0 2024-03-18T16:30:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.490+0000 7f986e800640 1 -- 172.21.15.102:0/1304395846 >> v1:172.21.15.154:6789/0 conn(0x7f986815b4d0 legacy=0x7f98680704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.490+0000 7f986e800640 1 -- 172.21.15.102:0/1304395846 >> v1:172.21.15.102:6789/0 conn(0x7f9868152120 legacy=0x7f98680c2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.490+0000 7f986e800640 1 -- 172.21.15.102:0/1304395846 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98680c3710 con 0x7f98680cacc0 2024-03-18T16:30:58.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.496+0000 7f986e800640 1 --2- 172.21.15.102:0/1304395846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98680cacc0 0x7f98681589f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9864009d90 tx=0x7f986402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ca46a2a6a0fa27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:58.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.501+0000 7f986ca00640 1 -- 172.21.15.102:0/1304395846 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864039070 con 0x7f98680cacc0 2024-03-18T16:30:58.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.501+0000 7f986ca00640 1 -- 172.21.15.102:0/1304395846 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f986402cd80 con 0x7f98680cacc0 2024-03-18T16:30:58.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.501+0000 7f986ca00640 1 -- 172.21.15.102:0/1304395846 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864034660 con 0x7f98680cacc0 2024-03-18T16:30:58.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.502+0000 7f986f200640 1 -- 172.21.15.102:0/1304395846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98680cacc0 msgr2=0x7f98681589f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:58.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.502+0000 7f986f200640 1 --2- 172.21.15.102:0/1304395846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98680cacc0 0x7f98681589f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9864009d90 tx=0x7f986402bfd0 comp rx=0 tx=0).stop 2024-03-18T16:30:58.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.502+0000 7f986f200640 1 -- 172.21.15.102:0/1304395846 shutdown_connections 2024-03-18T16:30:58.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.502+0000 7f986f200640 1 --2- 172.21.15.102:0/1304395846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98680cacc0 0x7f98681589f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:58.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.502+0000 7f986f200640 1 -- 172.21.15.102:0/1304395846 >> 172.21.15.102:0/1304395846 conn(0x7f9868067190 msgr2=0x7f98680675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:58.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.502+0000 7f986f200640 1 -- 172.21.15.102:0/1304395846 shutdown_connections 2024-03-18T16:30:58.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.502+0000 7f986f200640 1 -- 172.21.15.102:0/1304395846 wait complete. 2024-03-18T16:30:58.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.504+0000 7f986f200640 1 Processor -- start 2024-03-18T16:30:58.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.505+0000 7f986f200640 1 -- start start 2024-03-18T16:30:58.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.505+0000 7f986f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680cacc0 0x7f98680c7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:58.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.505+0000 7f986f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9868152120 0x7f98681f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:58.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.506+0000 7f986f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f986815b4d0 0x7f98680cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:58.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.506+0000 7f986f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9868157ee0 con 0x7f9868152120 2024-03-18T16:30:58.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.506+0000 7f986f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98680cb0d0 con 0x7f986815b4d0 2024-03-18T16:30:58.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.506+0000 7f986f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9868152500 con 0x7f98680cacc0 2024-03-18T16:30:58.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.506+0000 7f986d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9868152120 0x7f98681f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:58.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.506+0000 7f986e800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f986815b4d0 0x7f98680cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:58.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.506+0000 7f986de00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680cacc0 0x7f98680c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:58.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.507+0000 7f986d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9868152120 0x7f98681f3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59374/0 (socket says 172.21.15.102:59374) 2024-03-18T16:30:58.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.507+0000 7f986d400640 1 -- 172.21.15.102:0/4194166597 learned_addr learned my addr 172.21.15.102:0/4194166597 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:30:58.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.510+0000 7f986de00640 1 -- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f986815b4d0 msgr2=0x7f98680cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:58.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.510+0000 7f986de00640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f986815b4d0 0x7f98680cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:58.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.510+0000 7f986de00640 1 -- 172.21.15.102:0/4194166597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9868152120 msgr2=0x7f98681f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:58.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.510+0000 7f986de00640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9868152120 0x7f98681f3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:58.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.510+0000 7f986de00640 1 -- 172.21.15.102:0/4194166597 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986814a360 con 0x7f98680cacc0 2024-03-18T16:30:58.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.513+0000 7f986de00640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680cacc0 0x7f98680c7740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f985c00c340 tx=0x7f985c00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:58.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.517+0000 7f9862000640 1 -- 172.21.15.102:0/4194166597 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c0052d0 con 0x7f98680cacc0 2024-03-18T16:30:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.517+0000 7f9862000640 1 -- 172.21.15.102:0/4194166597 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f985c005490 con 0x7f98680cacc0 2024-03-18T16:30:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.517+0000 7f9862000640 1 -- 172.21.15.102:0/4194166597 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985c007260 con 0x7f98680cacc0 2024-03-18T16:30:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.517+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98681f3890 con 0x7f98680cacc0 2024-03-18T16:30:58.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.517+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9868149da0 con 0x7f98680cacc0 2024-03-18T16:30:58.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.518+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9868074fd0 con 0x7f98680cacc0 2024-03-18T16:30:58.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.776+0000 7f9862000640 1 -- 172.21.15.102:0/4194166597 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f985c007400 con 0x7f98680cacc0 2024-03-18T16:30:58.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.778+0000 7f9862000640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f984c06bc20 0x7f984c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:30:58.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.778+0000 7f9862000640 1 -- 172.21.15.102:0/4194166597 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f985c08a6e0 con 0x7f98680cacc0 2024-03-18T16:30:58.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.778+0000 7f986d400640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f984c06bc20 0x7f984c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:30:58.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.779+0000 7f986d400640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f984c06bc20 0x7f984c06e0e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f98680c85c0 tx=0x7f9858009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:30:58.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:58.788+0000 7f9862000640 1 -- 172.21.15.102:0/4194166597 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f985c058800 con 0x7f98680cacc0 2024-03-18T16:30:59.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.030+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98680b2ec0 con 0x7f98680cacc0 2024-03-18T16:30:59.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.043+0000 7f9862000640 1 -- 172.21.15.102:0/4194166597 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f985c05c8a0 con 0x7f98680cacc0 2024-03-18T16:30:59.045 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:30:59.045 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:30:59.045 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.045+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f984c06bc20 msgr2=0x7f984c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f984c06bc20 0x7f984c06e0e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f98680c85c0 tx=0x7f9858009040 comp rx=0 tx=0).stop 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680cacc0 msgr2=0x7f98680c7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680cacc0 0x7f98680c7740 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f985c00c340 tx=0x7f985c00c800 comp rx=0 tx=0).stop 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 shutdown_connections 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f984c06bc20 0x7f984c06e0e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f986815b4d0 0x7f98680cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9868152120 0x7f98681f3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 --2- 172.21.15.102:0/4194166597 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f98680cacc0 0x7f98680c7740 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 >> 172.21.15.102:0/4194166597 conn(0x7f9868067190 msgr2=0x7f98681574c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 shutdown_connections 2024-03-18T16:30:59.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:30:59.046+0000 7f986f200640 1 -- 172.21.15.102:0/4194166597 wait complete. 2024-03-18T16:30:59.671 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:30:59.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:00.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:00.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.845+0000 7f3bb3e00640 1 Processor -- start 2024-03-18T16:31:00.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.846+0000 7f3bb3e00640 1 -- start start 2024-03-18T16:31:00.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.847+0000 7f3bb3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40cca00 0x7f3bb415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:00.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.847+0000 7f3bb3e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bb40cabd0 con 0x7f3bb40d0250 2024-03-18T16:31:00.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.847+0000 7f3bb3e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bb40caea0 con 0x7f3bb40cfe70 2024-03-18T16:31:00.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.847+0000 7f3bb3e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb40cb180 con 0x7f3bb40cca00 2024-03-18T16:31:00.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.847+0000 7f3bb2a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3bb40cfe70 0x7f3bb40c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:37874/0 (socket says 172.21.15.102:37874) 2024-03-18T16:31:00.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.847+0000 7f3bb2a00640 1 -- 172.21.15.102:0/1817054649 learned_addr learned my addr 172.21.15.102:0/1817054649 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:00.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.849+0000 7f3bb3400640 1 --2- 172.21.15.102:0/1817054649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40cca00 0x7f3bb415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:00.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.850+0000 7f3bb1600640 1 -- 172.21.15.102:0/1817054649 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4119376041 0 0) 0x7f3bb40caea0 con 0x7f3bb40cfe70 2024-03-18T16:31:00.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.850+0000 7f3bb1600640 1 -- 172.21.15.102:0/1817054649 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b98003520 con 0x7f3bb40cfe70 2024-03-18T16:31:00.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.851+0000 7f3bb1600640 1 -- 172.21.15.102:0/1817054649 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340802707 0 0) 0x7f3bb40cabd0 con 0x7f3bb40d0250 2024-03-18T16:31:00.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.851+0000 7f3bb1600640 1 -- 172.21.15.102:0/1817054649 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bb40caea0 con 0x7f3bb40d0250 2024-03-18T16:31:00.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.853+0000 7f3bb3400640 1 -- 172.21.15.102:0/1817054649 >> v1:172.21.15.154:6789/0 conn(0x7f3bb40cfe70 legacy=0x7f3bb40c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:00.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.853+0000 7f3bb3400640 1 -- 172.21.15.102:0/1817054649 >> v1:172.21.15.102:6789/0 conn(0x7f3bb40d0250 legacy=0x7f3bb40cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:00.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.853+0000 7f3bb3400640 1 -- 172.21.15.102:0/1817054649 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb40cb460 con 0x7f3bb40cca00 2024-03-18T16:31:00.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.855+0000 7f3bb3400640 1 --2- 172.21.15.102:0/1817054649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40cca00 0x7f3bb415ae40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009dc0 tx=0x7f3bac02bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f2f6f7b49c2cdd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:00.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.859+0000 7f3bb1600640 1 -- 172.21.15.102:0/1817054649 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac039070 con 0x7f3bb40cca00 2024-03-18T16:31:00.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.859+0000 7f3bb1600640 1 -- 172.21.15.102:0/1817054649 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bac02cd60 con 0x7f3bb40cca00 2024-03-18T16:31:00.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.859+0000 7f3bb1600640 1 -- 172.21.15.102:0/1817054649 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac034660 con 0x7f3bb40cca00 2024-03-18T16:31:00.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.860+0000 7f3bb3e00640 1 -- 172.21.15.102:0/1817054649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40cca00 msgr2=0x7f3bb415ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:00.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.860+0000 7f3bb3e00640 1 --2- 172.21.15.102:0/1817054649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40cca00 0x7f3bb415ae40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3bac009dc0 tx=0x7f3bac02bfb0 comp rx=0 tx=0).stop 2024-03-18T16:31:00.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.861+0000 7f3bb3e00640 1 -- 172.21.15.102:0/1817054649 shutdown_connections 2024-03-18T16:31:00.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.861+0000 7f3bb3e00640 1 --2- 172.21.15.102:0/1817054649 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40cca00 0x7f3bb415ae40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:00.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.861+0000 7f3bb3e00640 1 -- 172.21.15.102:0/1817054649 >> 172.21.15.102:0/1817054649 conn(0x7f3bb408c3e0 msgr2=0x7f3bb406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:00.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.861+0000 7f3bb3e00640 1 -- 172.21.15.102:0/1817054649 shutdown_connections 2024-03-18T16:31:00.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.861+0000 7f3bb3e00640 1 -- 172.21.15.102:0/1817054649 wait complete. 2024-03-18T16:31:00.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.863+0000 7f3bb3e00640 1 Processor -- start 2024-03-18T16:31:00.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.863+0000 7f3bb3e00640 1 -- start start 2024-03-18T16:31:00.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.865+0000 7f3bb3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb40cca00 0x7f3bb41e1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:00.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.865+0000 7f3bb3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb40cfe70 0x7f3bb40c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:00.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.865+0000 7f3bb3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40d0250 0x7f3bb414f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:00.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.865+0000 7f3bb3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb40cbd00 con 0x7f3bb40cca00 2024-03-18T16:31:00.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.865+0000 7f3bb3e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb40cba20 con 0x7f3bb40cfe70 2024-03-18T16:31:00.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.865+0000 7f3bb3e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb40c8330 con 0x7f3bb40d0250 2024-03-18T16:31:00.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.866+0000 7f3bb2000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb40cfe70 0x7f3bb40c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:00.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.866+0000 7f3bb2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb40cca00 0x7f3bb41e1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:00.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.866+0000 7f3bb3400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40d0250 0x7f3bb414f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:00.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.866+0000 7f3bb2000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb40cfe70 0x7f3bb40c50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49614/0 (socket says 172.21.15.102:49614) 2024-03-18T16:31:00.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.866+0000 7f3bb2000640 1 -- 172.21.15.102:0/4134235662 learned_addr learned my addr 172.21.15.102:0/4134235662 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:00.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.869+0000 7f3bb2000640 1 -- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40d0250 msgr2=0x7f3bb414f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:00.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.869+0000 7f3bb2000640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40d0250 0x7f3bb414f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:00.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.869+0000 7f3bb2000640 1 -- 172.21.15.102:0/4134235662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb40cca00 msgr2=0x7f3bb41e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:00.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.869+0000 7f3bb2000640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb40cca00 0x7f3bb41e1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:00.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.869+0000 7f3bb2000640 1 -- 172.21.15.102:0/4134235662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb41f2fb0 con 0x7f3bb40cfe70 2024-03-18T16:31:00.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.872+0000 7f3bb2000640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb40cfe70 0x7f3bb40c50a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0001eb0 tx=0x7f3ba000c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:00.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.875+0000 7f3baaa00640 1 -- 172.21.15.102:0/4134235662 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0017070 con 0x7f3bb40cfe70 2024-03-18T16:31:00.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.875+0000 7f3baaa00640 1 -- 172.21.15.102:0/4134235662 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ba00052d0 con 0x7f3bb40cfe70 2024-03-18T16:31:00.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.875+0000 7f3baaa00640 1 -- 172.21.15.102:0/4134235662 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0003810 con 0x7f3bb40cfe70 2024-03-18T16:31:00.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.875+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bb4159470 con 0x7f3bb40cfe70 2024-03-18T16:31:00.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.875+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bb4159a10 con 0x7f3bb40cfe70 2024-03-18T16:31:00.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.876+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bb4074fd0 con 0x7f3bb40cfe70 2024-03-18T16:31:00.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.896+0000 7f3baaa00640 1 -- 172.21.15.102:0/4134235662 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3ba001a070 con 0x7f3bb40cfe70 2024-03-18T16:31:00.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.897+0000 7f3baaa00640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b9406bc20 0x7f3b9406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:00.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.898+0000 7f3bb2a00640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b9406bc20 0x7f3b9406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:00.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.898+0000 7f3bb2a00640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b9406bc20 0x7f3b9406e0e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4000f60 tx=0x7f3ba4007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:00.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:00.899+0000 7f3baaa00640 1 -- 172.21.15.102:0/4134235662 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3ba0056780 con 0x7f3bb40cfe70 2024-03-18T16:31:01.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.152+0000 7f3baaa00640 1 -- 172.21.15.102:0/4134235662 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3ba0021020 con 0x7f3bb40cfe70 2024-03-18T16:31:01.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.463+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bb40c64d0 con 0x7f3bb40cfe70 2024-03-18T16:31:01.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.551+0000 7f3baaa00640 1 -- 172.21.15.102:0/4134235662 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3ba00586e0 con 0x7f3bb40cfe70 2024-03-18T16:31:01.554 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:01.554 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:01.554 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b9406bc20 msgr2=0x7f3b9406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b9406bc20 0x7f3b9406e0e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4000f60 tx=0x7f3ba4007d30 comp rx=0 tx=0).stop 2024-03-18T16:31:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb40cfe70 msgr2=0x7f3bb40c50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb40cfe70 0x7f3bb40c50a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0001eb0 tx=0x7f3ba000c820 comp rx=0 tx=0).stop 2024-03-18T16:31:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 shutdown_connections 2024-03-18T16:31:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3b9406bc20 0x7f3b9406e0e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:01.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3bb40d0250 0x7f3bb414f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb40cfe70 0x7f3bb40c50a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 --2- 172.21.15.102:0/4134235662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bb40cca00 0x7f3bb41e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 >> 172.21.15.102:0/4134235662 conn(0x7f3bb408c3e0 msgr2=0x7f3bb40cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 shutdown_connections 2024-03-18T16:31:01.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:01.554+0000 7f3bb3e00640 1 -- 172.21.15.102:0/4134235662 wait complete. 2024-03-18T16:31:02.183 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:02.183 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:03.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:03.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.360+0000 7fd432a00640 1 Processor -- start 2024-03-18T16:31:03.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.360+0000 7fd432a00640 1 -- start start 2024-03-18T16:31:03.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.362+0000 7fd432a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c0c7830 0x7fd42c071e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:03.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.362+0000 7fd432a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42c15e950 con 0x7fd42c0c7c70 2024-03-18T16:31:03.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.362+0000 7fd432a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42c0c8190 con 0x7fd42c158bb0 2024-03-18T16:31:03.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.362+0000 7fd432a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd42c15f540 con 0x7fd42c0c7830 2024-03-18T16:31:03.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.362+0000 7fd42be00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd42c158bb0 0x7fd42c072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:50042/0 (socket says 172.21.15.102:50042) 2024-03-18T16:31:03.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.362+0000 7fd42be00640 1 -- 172.21.15.102:0/1885886238 learned_addr learned my addr 172.21.15.102:0/1885886238 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:03.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.363+0000 7fd42b400640 1 --2- 172.21.15.102:0/1885886238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c0c7830 0x7fd42c071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:03.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.365+0000 7fd42aa00640 1 -- 172.21.15.102:0/1885886238 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1968978423 0 0) 0x7fd42c0c8190 con 0x7fd42c158bb0 2024-03-18T16:31:03.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.365+0000 7fd42aa00640 1 -- 172.21.15.102:0/1885886238 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd410003520 con 0x7fd42c158bb0 2024-03-18T16:31:03.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.366+0000 7fd42aa00640 1 -- 172.21.15.102:0/1885886238 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036283402 0 0) 0x7fd42c15e950 con 0x7fd42c0c7c70 2024-03-18T16:31:03.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.366+0000 7fd42aa00640 1 -- 172.21.15.102:0/1885886238 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd42c0c8190 con 0x7fd42c0c7c70 2024-03-18T16:31:03.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.366+0000 7fd42b400640 1 -- 172.21.15.102:0/1885886238 >> v1:172.21.15.154:6789/0 conn(0x7fd42c158bb0 legacy=0x7fd42c072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:03.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.366+0000 7fd42b400640 1 -- 172.21.15.102:0/1885886238 >> v1:172.21.15.102:6789/0 conn(0x7fd42c0c7c70 legacy=0x7fd42c06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:03.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.366+0000 7fd42b400640 1 -- 172.21.15.102:0/1885886238 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42c15ec10 con 0x7fd42c0c7830 2024-03-18T16:31:03.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.369+0000 7fd42b400640 1 --2- 172.21.15.102:0/1885886238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c0c7830 0x7fd42c071e20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd42c15f6c0 tx=0x7fd41802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc51e35b97226dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:03.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.374+0000 7fd42aa00640 1 -- 172.21.15.102:0/1885886238 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418034070 con 0x7fd42c0c7830 2024-03-18T16:31:03.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.374+0000 7fd42aa00640 1 -- 172.21.15.102:0/1885886238 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd418030800 con 0x7fd42c0c7830 2024-03-18T16:31:03.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.374+0000 7fd42aa00640 1 -- 172.21.15.102:0/1885886238 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418030bd0 con 0x7fd42c0c7830 2024-03-18T16:31:03.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.374+0000 7fd432a00640 1 -- 172.21.15.102:0/1885886238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c0c7830 msgr2=0x7fd42c071e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:03.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.374+0000 7fd432a00640 1 --2- 172.21.15.102:0/1885886238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c0c7830 0x7fd42c071e20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd42c15f6c0 tx=0x7fd41802b940 comp rx=0 tx=0).stop 2024-03-18T16:31:03.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.375+0000 7fd432a00640 1 -- 172.21.15.102:0/1885886238 shutdown_connections 2024-03-18T16:31:03.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.375+0000 7fd432a00640 1 --2- 172.21.15.102:0/1885886238 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c0c7830 0x7fd42c071e20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:03.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.375+0000 7fd432a00640 1 -- 172.21.15.102:0/1885886238 >> 172.21.15.102:0/1885886238 conn(0x7fd42c0700d0 msgr2=0x7fd42c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:03.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.375+0000 7fd432a00640 1 -- 172.21.15.102:0/1885886238 shutdown_connections 2024-03-18T16:31:03.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.375+0000 7fd432a00640 1 -- 172.21.15.102:0/1885886238 wait complete. 2024-03-18T16:31:03.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.377+0000 7fd432a00640 1 Processor -- start 2024-03-18T16:31:03.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.377+0000 7fd432a00640 1 -- start start 2024-03-18T16:31:03.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.379+0000 7fd432a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd42c0c7830 0x7fd42c1e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:03.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.379+0000 7fd432a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd42c0c7c70 0x7fd42c156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:03.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.379+0000 7fd432a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c158bb0 0x7fd42c1f7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:03.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.379+0000 7fd432a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd42c158f90 con 0x7fd42c0c7c70 2024-03-18T16:31:03.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.379+0000 7fd432a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd42c15f1d0 con 0x7fd42c0c7830 2024-03-18T16:31:03.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.379+0000 7fd432a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd42c159110 con 0x7fd42c158bb0 2024-03-18T16:31:03.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.379+0000 7fd42be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd42c0c7830 0x7fd42c1e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:03.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.380+0000 7fd430a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c158bb0 0x7fd42c1f7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:03.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.380+0000 7fd42be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd42c0c7830 0x7fd42c1e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49626/0 (socket says 172.21.15.102:49626) 2024-03-18T16:31:03.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.380+0000 7fd42be00640 1 -- 172.21.15.102:0/473204784 learned_addr learned my addr 172.21.15.102:0/473204784 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:03.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.380+0000 7fd42b400640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd42c0c7c70 0x7fd42c156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.384+0000 7fd42be00640 1 -- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c158bb0 msgr2=0x7fd42c1f7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.384+0000 7fd42be00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c158bb0 0x7fd42c1f7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.384+0000 7fd42be00640 1 -- 172.21.15.102:0/473204784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd42c0c7c70 msgr2=0x7fd42c156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.384+0000 7fd42be00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd42c0c7c70 0x7fd42c156690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.384+0000 7fd42be00640 1 -- 172.21.15.102:0/473204784 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42c15bf60 con 0x7fd42c0c7830 2024-03-18T16:31:03.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.386+0000 7fd42be00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd42c0c7830 0x7fd42c1e8270 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00c410 tx=0x7fd41c00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:03.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.388+0000 7fd423e00640 1 -- 172.21.15.102:0/473204784 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c017070 con 0x7fd42c0c7830 2024-03-18T16:31:03.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.389+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd42c1e88d0 con 0x7fd42c0c7830 2024-03-18T16:31:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.389+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd42c1e5030 con 0x7fd42c0c7830 2024-03-18T16:31:03.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.390+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd42c074fd0 con 0x7fd42c0c7830 2024-03-18T16:31:03.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.414+0000 7fd423e00640 1 -- 172.21.15.102:0/473204784 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41c005ce0 con 0x7fd42c0c7830 2024-03-18T16:31:03.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.414+0000 7fd423e00640 1 -- 172.21.15.102:0/473204784 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c021060 con 0x7fd42c0c7830 2024-03-18T16:31:03.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.416+0000 7fd423e00640 1 -- 172.21.15.102:0/473204784 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd41c01a070 con 0x7fd42c0c7830 2024-03-18T16:31:03.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.418+0000 7fd423e00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd40c06bbd0 0x7fd40c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:03.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.418+0000 7fd423e00640 1 -- 172.21.15.102:0/473204784 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd41c056ca0 con 0x7fd42c0c7830 2024-03-18T16:31:03.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.418+0000 7fd42b400640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd40c06bbd0 0x7fd40c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:03.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.419+0000 7fd42b400640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd40c06bbd0 0x7fd40c06e090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd418036000 tx=0x7fd41803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:03.705+0000 7fd423e00640 1 -- 172.21.15.102:0/473204784 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd42c15bf60 con 0x7fd42c0c7830 2024-03-18T16:31:04.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.093+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd42c072960 con 0x7fd42c0c7830 2024-03-18T16:31:04.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.106+0000 7fd423e00640 1 -- 172.21.15.102:0/473204784 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd41c058c00 con 0x7fd42c0c7830 2024-03-18T16:31:04.109 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:04.109 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:04.109 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:04.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd40c06bbd0 msgr2=0x7fd40c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:04.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd40c06bbd0 0x7fd40c06e090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd418036000 tx=0x7fd41803b040 comp rx=0 tx=0).stop 2024-03-18T16:31:04.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd42c0c7830 msgr2=0x7fd42c1e8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd42c0c7830 0x7fd42c1e8270 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00c410 tx=0x7fd41c00c8d0 comp rx=0 tx=0).stop 2024-03-18T16:31:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 shutdown_connections 2024-03-18T16:31:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd40c06bbd0 0x7fd40c06e090 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd42c158bb0 0x7fd42c1f7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd42c0c7c70 0x7fd42c156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 --2- 172.21.15.102:0/473204784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd42c0c7830 0x7fd42c1e8270 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.108+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 >> 172.21.15.102:0/473204784 conn(0x7fd42c0700d0 msgr2=0x7fd42c06d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.109+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 shutdown_connections 2024-03-18T16:31:04.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:04.109+0000 7fd432a00640 1 -- 172.21.15.102:0/473204784 wait complete. 2024-03-18T16:31:04.732 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:04.732 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:05.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:05.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.910+0000 7efc03e00640 1 Processor -- start 2024-03-18T16:31:05.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.910+0000 7efc03e00640 1 -- start start 2024-03-18T16:31:05.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.911+0000 7efc03e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efbfc1511f0 0x7efbfc157b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:05.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.911+0000 7efc03e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc06f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:05.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.911+0000 7efc03e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efbfc0c3ff0 con 0x7efbfc0cd2b0 2024-03-18T16:31:05.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.911+0000 7efc03e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7efbfc1515d0 con 0x7efbfc1511f0 2024-03-18T16:31:05.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.911+0000 7efc03e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efbfc1580b0 con 0x7efbfc1585a0 2024-03-18T16:31:05.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.912+0000 7efc00a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc06f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:05.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.913+0000 7efc00a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc06f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46426/0 (socket says 172.21.15.102:46426) 2024-03-18T16:31:05.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.913+0000 7efc00a00640 1 -- 172.21.15.102:0/3014417046 learned_addr learned my addr 172.21.15.102:0/3014417046 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:05.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.914+0000 7efc01400640 1 --2- 172.21.15.102:0/3014417046 >> v2:172.21.15.102:3300/0 conn(0x7efbfc1511f0 0x7efbfc157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:05.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.919+0000 7efc00a00640 1 -- 172.21.15.102:0/3014417046 >> v2:172.21.15.102:3300/0 conn(0x7efbfc1511f0 msgr2=0x7efbfc157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:05.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.919+0000 7efc00a00640 1 --2- 172.21.15.102:0/3014417046 >> v2:172.21.15.102:3300/0 conn(0x7efbfc1511f0 0x7efbfc157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:05.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.919+0000 7efc00a00640 1 -- 172.21.15.102:0/3014417046 >> v1:172.21.15.102:6789/0 conn(0x7efbfc0cd2b0 legacy=0x7efbfc15a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:05.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.919+0000 7efc00a00640 1 -- 172.21.15.102:0/3014417046 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbfc1570b0 con 0x7efbfc1585a0 2024-03-18T16:31:05.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.925+0000 7efc00a00640 1 --2- 172.21.15.102:0/3014417046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc06f310 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009870 tx=0x7efbe802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=697cbd51d2f12281 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:05.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.976+0000 7efbfbe00640 1 -- 172.21.15.102:0/3014417046 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe802f830 con 0x7efbfc1585a0 2024-03-18T16:31:05.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.976+0000 7efbfbe00640 1 -- 172.21.15.102:0/3014417046 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbe802f9d0 con 0x7efbfc1585a0 2024-03-18T16:31:05.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.976+0000 7efbfbe00640 1 -- 172.21.15.102:0/3014417046 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbe802fda0 con 0x7efbfc1585a0 2024-03-18T16:31:05.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.977+0000 7efc03e00640 1 -- 172.21.15.102:0/3014417046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 msgr2=0x7efbfc06f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:05.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.977+0000 7efc03e00640 1 --2- 172.21.15.102:0/3014417046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc06f310 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009870 tx=0x7efbe802ba10 comp rx=0 tx=0).stop 2024-03-18T16:31:05.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.977+0000 7efc03e00640 1 -- 172.21.15.102:0/3014417046 shutdown_connections 2024-03-18T16:31:05.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.977+0000 7efc03e00640 1 --2- 172.21.15.102:0/3014417046 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc06f310 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:05.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.977+0000 7efc03e00640 1 --2- 172.21.15.102:0/3014417046 >> v2:172.21.15.102:3300/0 conn(0x7efbfc1511f0 0x7efbfc157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:05.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.977+0000 7efc03e00640 1 -- 172.21.15.102:0/3014417046 >> 172.21.15.102:0/3014417046 conn(0x7efbfc072e30 msgr2=0x7efbfc073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:05.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.978+0000 7efc03e00640 1 -- 172.21.15.102:0/3014417046 shutdown_connections 2024-03-18T16:31:05.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.978+0000 7efc03e00640 1 -- 172.21.15.102:0/3014417046 wait complete. 2024-03-18T16:31:05.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.980+0000 7efc03e00640 1 Processor -- start 2024-03-18T16:31:05.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.981+0000 7efc03e00640 1 -- start start 2024-03-18T16:31:05.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.982+0000 7efc03e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbfc0cd2b0 0x7efbfc1eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:05.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.982+0000 7efc03e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efbfc1511f0 0x7efbfc14c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:05.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.982+0000 7efc03e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:05.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.982+0000 7efc03e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc06ea40 con 0x7efbfc0cd2b0 2024-03-18T16:31:05.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.982+0000 7efc03e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efbfc1576e0 con 0x7efbfc1511f0 2024-03-18T16:31:05.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.982+0000 7efc03e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efbfc06ed20 con 0x7efbfc1585a0 2024-03-18T16:31:05.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.983+0000 7efc01e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:05.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.983+0000 7efc00a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efbfc1511f0 0x7efbfc14c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:05.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.983+0000 7efc01e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc149a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46442/0 (socket says 172.21.15.102:46442) 2024-03-18T16:31:05.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.983+0000 7efc01e00640 1 -- 172.21.15.102:0/30200633 learned_addr learned my addr 172.21.15.102:0/30200633 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:05.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.987+0000 7efc01e00640 1 -- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efbfc1511f0 msgr2=0x7efbfc14c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:05.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.987+0000 7efc01e00640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efbfc1511f0 0x7efbfc14c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:05.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.987+0000 7efc01e00640 1 -- 172.21.15.102:0/30200633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbfc0cd2b0 msgr2=0x7efbfc1eeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:05.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.987+0000 7efc01e00640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbfc0cd2b0 0x7efbfc1eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:05.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.987+0000 7efc01e00640 1 -- 172.21.15.102:0/30200633 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbfc0c4300 con 0x7efbfc1585a0 2024-03-18T16:31:05.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.990+0000 7efc01e00640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc149a50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c740 tx=0x7efbf400cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:05.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.994+0000 7efbf9600640 1 -- 172.21.15.102:0/30200633 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4016020 con 0x7efbfc1585a0 2024-03-18T16:31:05.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.994+0000 7efbf9600640 1 -- 172.21.15.102:0/30200633 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf400ce50 con 0x7efbfc1585a0 2024-03-18T16:31:05.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.994+0000 7efbf9600640 1 -- 172.21.15.102:0/30200633 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4003970 con 0x7efbfc1585a0 2024-03-18T16:31:05.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.994+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbfc0c45e0 con 0x7efbfc1585a0 2024-03-18T16:31:05.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.994+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efbfc06ca60 con 0x7efbfc1585a0 2024-03-18T16:31:05.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:05.996+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbfc074fd0 con 0x7efbfc1585a0 2024-03-18T16:31:06.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.019+0000 7efbf9600640 1 -- 172.21.15.102:0/30200633 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efbf4003b10 con 0x7efbfc1585a0 2024-03-18T16:31:06.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.020+0000 7efbf9600640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efbdc06bc20 0x7efbdc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:06.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.021+0000 7efc01400640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efbdc06bc20 0x7efbdc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:06.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.021+0000 7efc01400640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efbdc06bc20 0x7efbdc06e0e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efbf0006a90 tx=0x7efbf0006a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:06.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.271+0000 7efbf9600640 1 -- 172.21.15.102:0/30200633 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7efbf4057930 con 0x7efbfc1585a0 2024-03-18T16:31:06.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.283+0000 7efbf9600640 1 -- 172.21.15.102:0/30200633 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efbf4020020 con 0x7efbfc1585a0 2024-03-18T16:31:06.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.523+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efbfc0685b0 con 0x7efbfc1585a0 2024-03-18T16:31:06.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.537+0000 7efbf9600640 1 -- 172.21.15.102:0/30200633 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7efbfc0c3ff0 con 0x7efbfc1585a0 2024-03-18T16:31:06.540 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:06.540 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:06.540 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:06.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efbdc06bc20 msgr2=0x7efbdc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:06.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efbdc06bc20 0x7efbdc06e0e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efbf0006a90 tx=0x7efbf0006a20 comp rx=0 tx=0).stop 2024-03-18T16:31:06.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 msgr2=0x7efbfc149a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:06.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc149a50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efbf400c740 tx=0x7efbf400cc00 comp rx=0 tx=0).stop 2024-03-18T16:31:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 shutdown_connections 2024-03-18T16:31:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efbdc06bc20 0x7efbdc06e0e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efbfc1585a0 0x7efbfc149a50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efbfc1511f0 0x7efbfc14c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 --2- 172.21.15.102:0/30200633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efbfc0cd2b0 0x7efbfc1eeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 >> 172.21.15.102:0/30200633 conn(0x7efbfc072e30 msgr2=0x7efbfc073210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 shutdown_connections 2024-03-18T16:31:06.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:06.539+0000 7efc03e00640 1 -- 172.21.15.102:0/30200633 wait complete. 2024-03-18T16:31:07.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:07.163 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:08.164 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:08.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.338+0000 7eff9e000640 1 Processor -- start 2024-03-18T16:31:08.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.339+0000 7eff9e000640 1 -- start start 2024-03-18T16:31:08.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.340+0000 7eff9e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7eff98141650 0x7eff98158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:08.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.340+0000 7eff9e000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff980c72d0 con 0x7eff980c4fa0 2024-03-18T16:31:08.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.340+0000 7eff9e000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff98158ad0 con 0x7eff9815c3a0 2024-03-18T16:31:08.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.340+0000 7eff9e000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7eff98159370 con 0x7eff98141650 2024-03-18T16:31:08.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.340+0000 7eff97e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7eff9815c3a0 0x7eff980c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:50050/0 (socket says 172.21.15.102:50050) 2024-03-18T16:31:08.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.340+0000 7eff97e00640 1 -- 172.21.15.102:0/3629730368 learned_addr learned my addr 172.21.15.102:0/3629730368 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:08.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.341+0000 7eff9cc00640 1 --2- 172.21.15.102:0/3629730368 >> v2:172.21.15.102:3300/0 conn(0x7eff98141650 0x7eff98158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:08.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.343+0000 7eff97400640 1 -- 172.21.15.102:0/3629730368 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3759811317 0 0) 0x7eff98158ad0 con 0x7eff9815c3a0 2024-03-18T16:31:08.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.343+0000 7eff97400640 1 -- 172.21.15.102:0/3629730368 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff80003520 con 0x7eff9815c3a0 2024-03-18T16:31:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.345+0000 7eff9cc00640 1 -- 172.21.15.102:0/3629730368 >> v1:172.21.15.154:6789/0 conn(0x7eff9815c3a0 legacy=0x7eff980c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.345+0000 7eff9cc00640 1 -- 172.21.15.102:0/3629730368 >> v1:172.21.15.102:6789/0 conn(0x7eff980c4fa0 legacy=0x7eff98162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:08.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.345+0000 7eff9cc00640 1 -- 172.21.15.102:0/3629730368 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff98158db0 con 0x7eff98141650 2024-03-18T16:31:08.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.350+0000 7eff9cc00640 1 --2- 172.21.15.102:0/3629730368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff98141650 0x7eff98158420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097a0 tx=0x7eff8c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed760d860d2926f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.356+0000 7eff97400640 1 -- 172.21.15.102:0/3629730368 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c034070 con 0x7eff98141650 2024-03-18T16:31:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.356+0000 7eff97400640 1 -- 172.21.15.102:0/3629730368 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff8c030610 con 0x7eff98141650 2024-03-18T16:31:08.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.356+0000 7eff97400640 1 -- 172.21.15.102:0/3629730368 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0309e0 con 0x7eff98141650 2024-03-18T16:31:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.356+0000 7eff9e000640 1 -- 172.21.15.102:0/3629730368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff98141650 msgr2=0x7eff98158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.356+0000 7eff9e000640 1 --2- 172.21.15.102:0/3629730368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff98141650 0x7eff98158420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097a0 tx=0x7eff8c02b960 comp rx=0 tx=0).stop 2024-03-18T16:31:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.357+0000 7eff9e000640 1 -- 172.21.15.102:0/3629730368 shutdown_connections 2024-03-18T16:31:08.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.357+0000 7eff9e000640 1 --2- 172.21.15.102:0/3629730368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff98141650 0x7eff98158420 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.357+0000 7eff9e000640 1 -- 172.21.15.102:0/3629730368 >> 172.21.15.102:0/3629730368 conn(0x7eff980cb890 msgr2=0x7eff980cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.357+0000 7eff9e000640 1 -- 172.21.15.102:0/3629730368 shutdown_connections 2024-03-18T16:31:08.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.357+0000 7eff9e000640 1 -- 172.21.15.102:0/3629730368 wait complete. 2024-03-18T16:31:08.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.359+0000 7eff9e000640 1 Processor -- start 2024-03-18T16:31:08.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.360+0000 7eff9e000640 1 -- start start 2024-03-18T16:31:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.361+0000 7eff9e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff980c4fa0 0x7eff980c4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.361+0000 7eff9e000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98141650 0x7eff981e5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.361+0000 7eff9e000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff9815c3a0 0x7eff9813dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.361+0000 7eff9e000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7eff980c66a0 con 0x7eff980c4fa0 2024-03-18T16:31:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.361+0000 7eff9e000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff980c63c0 con 0x7eff98141650 2024-03-18T16:31:08.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.361+0000 7eff9e000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7eff9815c7c0 con 0x7eff9815c3a0 2024-03-18T16:31:08.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.362+0000 7eff97e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98141650 0x7eff981e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:08.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.362+0000 7eff9cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff980c4fa0 0x7eff980c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:08.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.362+0000 7eff97e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98141650 0x7eff981e5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49646/0 (socket says 172.21.15.102:49646) 2024-03-18T16:31:08.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.362+0000 7eff97e00640 1 -- 172.21.15.102:0/223448622 learned_addr learned my addr 172.21.15.102:0/223448622 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:08.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.362+0000 7eff9d600640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff9815c3a0 0x7eff9813dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:08.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.365+0000 7eff97e00640 1 -- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff9815c3a0 msgr2=0x7eff9813dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:08.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.365+0000 7eff97e00640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff9815c3a0 0x7eff9813dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:08.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.365+0000 7eff97e00640 1 -- 172.21.15.102:0/223448622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff980c4fa0 msgr2=0x7eff980c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:08.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.365+0000 7eff97e00640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff980c4fa0 0x7eff980c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:08.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.365+0000 7eff97e00640 1 -- 172.21.15.102:0/223448622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff981f0030 con 0x7eff98141650 2024-03-18T16:31:08.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.367+0000 7eff97e00640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98141650 0x7eff981e5190 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c410 tx=0x7eff8400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:08.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.371+0000 7eff94c00640 1 -- 172.21.15.102:0/223448622 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff84017070 con 0x7eff98141650 2024-03-18T16:31:08.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.371+0000 7eff94c00640 1 -- 172.21.15.102:0/223448622 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff84005ce0 con 0x7eff98141650 2024-03-18T16:31:08.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.371+0000 7eff94c00640 1 -- 172.21.15.102:0/223448622 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff840072f0 con 0x7eff98141650 2024-03-18T16:31:08.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.372+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9813cc60 con 0x7eff98141650 2024-03-18T16:31:08.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.372+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9813d200 con 0x7eff98141650 2024-03-18T16:31:08.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.373+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff98074fd0 con 0x7eff98141650 2024-03-18T16:31:08.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.394+0000 7eff94c00640 1 -- 172.21.15.102:0/223448622 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7eff8401a070 con 0x7eff98141650 2024-03-18T16:31:08.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.395+0000 7eff94c00640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff7806bc20 0x7eff7806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:08.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.395+0000 7eff9cc00640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff7806bc20 0x7eff7806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:08.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.396+0000 7eff9cc00640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff7806bc20 0x7eff7806e0e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7eff8c036000 tx=0x7eff8c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:08.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.634+0000 7eff94c00640 1 -- 172.21.15.102:0/223448622 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7eff840568c0 con 0x7eff98141650 2024-03-18T16:31:08.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.645+0000 7eff94c00640 1 -- 172.21.15.102:0/223448622 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7eff84021020 con 0x7eff98141650 2024-03-18T16:31:08.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.881+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff980b7f00 con 0x7eff98141650 2024-03-18T16:31:08.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.894+0000 7eff94c00640 1 -- 172.21.15.102:0/223448622 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7eff84058820 con 0x7eff98141650 2024-03-18T16:31:08.897 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:08.897 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:08.897 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff7806bc20 msgr2=0x7eff7806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff7806bc20 0x7eff7806e0e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7eff8c036000 tx=0x7eff8c03b040 comp rx=0 tx=0).stop 2024-03-18T16:31:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98141650 msgr2=0x7eff981e5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98141650 0x7eff981e5190 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7eff8400c410 tx=0x7eff8400c8d0 comp rx=0 tx=0).stop 2024-03-18T16:31:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 shutdown_connections 2024-03-18T16:31:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7eff7806bc20 0x7eff7806e0e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7eff9815c3a0 0x7eff9813dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:08.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff98141650 0x7eff981e5190 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 --2- 172.21.15.102:0/223448622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff980c4fa0 0x7eff980c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 >> 172.21.15.102:0/223448622 conn(0x7eff980cb890 msgr2=0x7eff98072d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 shutdown_connections 2024-03-18T16:31:08.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:08.896+0000 7eff9e000640 1 -- 172.21.15.102:0/223448622 wait complete. 2024-03-18T16:31:09.518 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:09.518 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:10.519 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:10.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.689+0000 7f2e9fe00640 1 Processor -- start 2024-03-18T16:31:10.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.689+0000 7f2e9fe00640 1 -- start start 2024-03-18T16:31:10.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.690+0000 7f2e9fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2ea00cfe70 0x7f2ea015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:10.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.690+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea00cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:10.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.690+0000 7f2e9fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea00cabd0 con 0x7f2ea00d0250 2024-03-18T16:31:10.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.690+0000 7f2e9fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2ea00caea0 con 0x7f2ea00cfe70 2024-03-18T16:31:10.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.690+0000 7f2e9fe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea00cb180 con 0x7f2ea00cc2f0 2024-03-18T16:31:10.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.691+0000 7f2e9f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea00cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:10.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.691+0000 7f2e9ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2ea00cfe70 0x7f2ea015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:10.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.691+0000 7f2e9f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea00cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46452/0 (socket says 172.21.15.102:46452) 2024-03-18T16:31:10.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.691+0000 7f2e9f400640 1 -- 172.21.15.102:0/1718215286 learned_addr learned my addr 172.21.15.102:0/1718215286 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.695+0000 7f2e9f400640 1 -- 172.21.15.102:0/1718215286 >> v2:172.21.15.102:3300/0 conn(0x7f2ea00cfe70 msgr2=0x7f2ea015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.695+0000 7f2e9f400640 1 --2- 172.21.15.102:0/1718215286 >> v2:172.21.15.102:3300/0 conn(0x7f2ea00cfe70 0x7f2ea015ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.695+0000 7f2e9f400640 1 -- 172.21.15.102:0/1718215286 >> v1:172.21.15.102:6789/0 conn(0x7f2ea00d0250 legacy=0x7f2ea00c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:10.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.695+0000 7f2e9f400640 1 -- 172.21.15.102:0/1718215286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea00cb460 con 0x7f2ea00cc2f0 2024-03-18T16:31:10.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.698+0000 7f2e9f400640 1 --2- 172.21.15.102:0/1718215286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea00cc710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009e50 tx=0x7f2e9802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=240c76c1cc6000b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:10.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.701+0000 7f2e9d600640 1 -- 172.21.15.102:0/1718215286 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9802cb90 con 0x7f2ea00cc2f0 2024-03-18T16:31:10.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.703+0000 7f2e9d600640 1 -- 172.21.15.102:0/1718215286 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e9802cd30 con 0x7f2ea00cc2f0 2024-03-18T16:31:10.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.703+0000 7f2e9d600640 1 -- 172.21.15.102:0/1718215286 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e98034610 con 0x7f2ea00cc2f0 2024-03-18T16:31:10.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.704+0000 7f2e9fe00640 1 -- 172.21.15.102:0/1718215286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 msgr2=0x7f2ea00cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:10.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.704+0000 7f2e9fe00640 1 --2- 172.21.15.102:0/1718215286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea00cc710 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009e50 tx=0x7f2e9802bff0 comp rx=0 tx=0).stop 2024-03-18T16:31:10.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.704+0000 7f2e9fe00640 1 -- 172.21.15.102:0/1718215286 shutdown_connections 2024-03-18T16:31:10.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.704+0000 7f2e9fe00640 1 --2- 172.21.15.102:0/1718215286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea00cc710 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:10.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.704+0000 7f2e9fe00640 1 --2- 172.21.15.102:0/1718215286 >> v2:172.21.15.102:3300/0 conn(0x7f2ea00cfe70 0x7f2ea015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:10.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.704+0000 7f2e9fe00640 1 -- 172.21.15.102:0/1718215286 >> 172.21.15.102:0/1718215286 conn(0x7f2ea008c3e0 msgr2=0x7f2ea006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:10.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.704+0000 7f2e9fe00640 1 -- 172.21.15.102:0/1718215286 shutdown_connections 2024-03-18T16:31:10.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.704+0000 7f2e9fe00640 1 -- 172.21.15.102:0/1718215286 wait complete. 2024-03-18T16:31:10.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.707+0000 7f2e9fe00640 1 Processor -- start 2024-03-18T16:31:10.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.707+0000 7f2e9fe00640 1 -- start start 2024-03-18T16:31:10.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.709+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea01e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:10.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.709+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ea00cfe70 0x7f2ea0158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:10.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.709+0000 7f2e9fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea00d0250 0x7f2ea0159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:10.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.709+0000 7f2e9fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea00cbd00 con 0x7f2ea00d0250 2024-03-18T16:31:10.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.709+0000 7f2e9fe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea00cba20 con 0x7f2ea00cfe70 2024-03-18T16:31:10.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.709+0000 7f2e9fe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea00ccc50 con 0x7f2ea00cc2f0 2024-03-18T16:31:10.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.709+0000 7f2e9ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea01e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:10.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.710+0000 7f2e9f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea00d0250 0x7f2ea0159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:10.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.710+0000 7f2e9ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea01e03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46462/0 (socket says 172.21.15.102:46462) 2024-03-18T16:31:10.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.710+0000 7f2e9ea00640 1 -- 172.21.15.102:0/2673250420 learned_addr learned my addr 172.21.15.102:0/2673250420 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:10.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.711+0000 7f2e9e000640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ea00cfe70 0x7f2ea0158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:10.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.714+0000 7f2e9f400640 1 -- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 msgr2=0x7f2ea01e03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:10.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.714+0000 7f2e9f400640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea01e03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:10.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.714+0000 7f2e9f400640 1 -- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ea00cfe70 msgr2=0x7f2ea0158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:10.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.714+0000 7f2e9f400640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ea00cfe70 0x7f2ea0158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:10.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.714+0000 7f2e9f400640 1 -- 172.21.15.102:0/2673250420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea01e3250 con 0x7f2ea00d0250 2024-03-18T16:31:10.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.717+0000 7f2e9f400640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea00d0250 0x7f2ea0159470 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00cb300 tx=0x7f2e98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:10.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.722+0000 7f2e96a00640 1 -- 172.21.15.102:0/2673250420 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e98043020 con 0x7f2ea00d0250 2024-03-18T16:31:10.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.722+0000 7f2e96a00640 1 -- 172.21.15.102:0/2673250420 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e9803e450 con 0x7f2ea00d0250 2024-03-18T16:31:10.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.722+0000 7f2e96a00640 1 -- 172.21.15.102:0/2673250420 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e98009510 con 0x7f2ea00d0250 2024-03-18T16:31:10.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.722+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea01efea0 con 0x7f2ea00d0250 2024-03-18T16:31:10.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.722+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea01f0460 con 0x7f2ea00d0250 2024-03-18T16:31:10.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.724+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea0074fd0 con 0x7f2ea00d0250 2024-03-18T16:31:10.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.749+0000 7f2e96a00640 1 -- 172.21.15.102:0/2673250420 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2e98046020 con 0x7f2ea00d0250 2024-03-18T16:31:10.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.751+0000 7f2e96a00640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2e8006bbd0 0x7f2e8006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:10.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.751+0000 7f2e9ea00640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2e8006bbd0 0x7f2e8006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:10.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.752+0000 7f2e9ea00640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2e8006bbd0 0x7f2e8006e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009c00 tx=0x7f2e90009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:10.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:10.753+0000 7f2e96a00640 1 -- 172.21.15.102:0/2673250420 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f2e98079850 con 0x7f2ea00d0250 2024-03-18T16:31:11.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.032+0000 7f2e96a00640 1 -- 172.21.15.102:0/2673250420 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2e9804d030 con 0x7f2ea00d0250 2024-03-18T16:31:11.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.397+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ea00c6930 con 0x7f2ea00d0250 2024-03-18T16:31:11.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.415+0000 7f2e96a00640 1 -- 172.21.15.102:0/2673250420 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f2e9807b7b0 con 0x7f2ea00d0250 2024-03-18T16:31:11.418 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:11.418 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:11.418 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2e8006bbd0 msgr2=0x7f2e8006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2e8006bbd0 0x7f2e8006e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009c00 tx=0x7f2e90009190 comp rx=0 tx=0).stop 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea00d0250 msgr2=0x7f2ea0159470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea00d0250 0x7f2ea0159470 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00cb300 tx=0x7f2e98004a40 comp rx=0 tx=0).stop 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 shutdown_connections 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2e8006bbd0 0x7f2e8006e090 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea00d0250 0x7f2ea0159470 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ea00cfe70 0x7f2ea0158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 --2- 172.21.15.102:0/2673250420 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2ea00cc2f0 0x7f2ea01e03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:11.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 >> 172.21.15.102:0/2673250420 conn(0x7f2ea008c3e0 msgr2=0x7f2ea00cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 shutdown_connections 2024-03-18T16:31:11.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:11.418+0000 7f2e9fe00640 1 -- 172.21.15.102:0/2673250420 wait complete. 2024-03-18T16:31:12.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:12.049 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:13.050 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:13.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.224+0000 7f8facc00640 1 Processor -- start 2024-03-18T16:31:13.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.224+0000 7f8facc00640 1 -- start start 2024-03-18T16:31:13.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.225+0000 7f8facc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8fa80cfe70 0x7f8fa815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:13.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.225+0000 7f8facc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8fa80d0250 0x7f8fa80c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:13.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.225+0000 7f8facc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fa80cabd0 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.225+0000 7f8facc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8fa80caea0 con 0x7f8fa80d0250 2024-03-18T16:31:13.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.225+0000 7f8facc00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8fa80cb180 con 0x7f8fa80cfe70 2024-03-18T16:31:13.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.226+0000 7f8fa7400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8fa80cfe70 0x7f8fa815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:13.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.226+0000 7f8fa6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8fa80d0250 0x7f8fa80c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:13.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.226+0000 7f8fa7400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8fa80cfe70 0x7f8fa815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56744/0 (socket says 172.21.15.102:56744) 2024-03-18T16:31:13.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.226+0000 7f8fa7400640 1 -- 172.21.15.102:0/708132865 learned_addr learned my addr 172.21.15.102:0/708132865 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:13.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.230+0000 7f8fa7400640 1 -- 172.21.15.102:0/708132865 >> v2:172.21.15.102:3300/0 conn(0x7f8fa80d0250 msgr2=0x7f8fa80c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:13.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.230+0000 7f8fa7400640 1 --2- 172.21.15.102:0/708132865 >> v2:172.21.15.102:3300/0 conn(0x7f8fa80d0250 0x7f8fa80c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.230+0000 7f8fa7400640 1 -- 172.21.15.102:0/708132865 >> v1:172.21.15.102:6789/0 conn(0x7f8fa80cc2f0 legacy=0x7f8fa80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:13.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.230+0000 7f8fa7400640 1 -- 172.21.15.102:0/708132865 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa80cb460 con 0x7f8fa80cfe70 2024-03-18T16:31:13.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.233+0000 7f8fa7400640 1 --2- 172.21.15.102:0/708132865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cfe70 0x7f8fa815ae40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80cd800 tx=0x7f8f9802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cb04cb00236a724 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:13.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.237+0000 7f8fa6000640 1 -- 172.21.15.102:0/708132865 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9802f850 con 0x7f8fa80cfe70 2024-03-18T16:31:13.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.237+0000 7f8fa6000640 1 -- 172.21.15.102:0/708132865 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f9802f9f0 con 0x7f8fa80cfe70 2024-03-18T16:31:13.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.237+0000 7f8fa6000640 1 -- 172.21.15.102:0/708132865 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9802fdc0 con 0x7f8fa80cfe70 2024-03-18T16:31:13.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.238+0000 7f8facc00640 1 -- 172.21.15.102:0/708132865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cfe70 msgr2=0x7f8fa815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:13.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.238+0000 7f8facc00640 1 --2- 172.21.15.102:0/708132865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cfe70 0x7f8fa815ae40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80cd800 tx=0x7f8f9802ba30 comp rx=0 tx=0).stop 2024-03-18T16:31:13.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.238+0000 7f8facc00640 1 -- 172.21.15.102:0/708132865 shutdown_connections 2024-03-18T16:31:13.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.238+0000 7f8facc00640 1 --2- 172.21.15.102:0/708132865 >> v2:172.21.15.102:3300/0 conn(0x7f8fa80d0250 0x7f8fa80c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.238+0000 7f8facc00640 1 --2- 172.21.15.102:0/708132865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cfe70 0x7f8fa815ae40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.238+0000 7f8facc00640 1 -- 172.21.15.102:0/708132865 >> 172.21.15.102:0/708132865 conn(0x7f8fa808c3e0 msgr2=0x7f8fa806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:13.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.238+0000 7f8facc00640 1 -- 172.21.15.102:0/708132865 shutdown_connections 2024-03-18T16:31:13.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.238+0000 7f8facc00640 1 -- 172.21.15.102:0/708132865 wait complete. 2024-03-18T16:31:13.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.241+0000 7f8facc00640 1 Processor -- start 2024-03-18T16:31:13.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.241+0000 7f8facc00640 1 -- start start 2024-03-18T16:31:13.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.243+0000 7f8facc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cc2f0 0x7f8fa8155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:13.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.243+0000 7f8facc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa80cfe70 0x7f8fa8158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:13.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.243+0000 7f8facc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fa80d0250 0x7f8fa8159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:13.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.243+0000 7f8facc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa80cbd00 con 0x7f8fa80cfe70 2024-03-18T16:31:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.243+0000 7f8facc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa80cba20 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.243+0000 7f8facc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa80c6b20 con 0x7f8fa80d0250 2024-03-18T16:31:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.243+0000 7f8fa7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cc2f0 0x7f8fa8155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.244+0000 7f8fa7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fa80d0250 0x7f8fa8159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.244+0000 7f8fa7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cc2f0 0x7f8fa8155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56746/0 (socket says 172.21.15.102:56746) 2024-03-18T16:31:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.244+0000 7f8fa7400640 1 -- 172.21.15.102:0/303396718 learned_addr learned my addr 172.21.15.102:0/303396718 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:13.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.245+0000 7f8fa6a00640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa80cfe70 0x7f8fa8158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:13.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.249+0000 7f8fa7400640 1 -- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fa80d0250 msgr2=0x7f8fa8159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:13.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.249+0000 7f8fa7400640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fa80d0250 0x7f8fa8159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.249+0000 7f8fa7400640 1 -- 172.21.15.102:0/303396718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa80cfe70 msgr2=0x7f8fa8158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:13.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.249+0000 7f8fa7400640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa80cfe70 0x7f8fa8158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.249+0000 7f8fa7400640 1 -- 172.21.15.102:0/303396718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa81580b0 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.252+0000 7f8fa7400640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cc2f0 0x7f8fa8155c50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8f980090b0 tx=0x7f8f980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:13.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.256+0000 7f8f9f400640 1 -- 172.21.15.102:0/303396718 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98037c80 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.257+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa80c48c0 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.257+0000 7f8f9f400640 1 -- 172.21.15.102:0/303396718 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f98047070 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.257+0000 7f8f9f400640 1 -- 172.21.15.102:0/303396718 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98037e60 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.257+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa8158320 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.258+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa8074fd0 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.534+0000 7f8f9f400640 1 -- 172.21.15.102:0/303396718 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8f98040050 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.535+0000 7f8f9f400640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f8806bc50 0x7f8f8806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:13.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.535+0000 7f8fa6a00640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f8806bc50 0x7f8f8806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:13.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.536+0000 7f8fa6a00640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f8806bc50 0x7f8f8806e110 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8157e10 tx=0x7f8f94009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:13.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.538+0000 7f8f9f400640 1 -- 172.21.15.102:0/303396718 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8f9807a600 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.556+0000 7f8f9f400640 1 -- 172.21.15.102:0/303396718 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8f98033030 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.802+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fa8067c20 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.844+0000 7f8f9f400640 1 -- 172.21.15.102:0/303396718 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8f9807c560 con 0x7f8fa80cc2f0 2024-03-18T16:31:13.847 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:13.847 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:13.847 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:13.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.848+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f8806bc50 msgr2=0x7f8f8806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:13.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.848+0000 7f8facc00640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f8806bc50 0x7f8f8806e110 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8157e10 tx=0x7f8f94009190 comp rx=0 tx=0).stop 2024-03-18T16:31:13.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.848+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cc2f0 msgr2=0x7f8fa8155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:13.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.848+0000 7f8facc00640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cc2f0 0x7f8fa8155c50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8f980090b0 tx=0x7f8f980046c0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.849+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 shutdown_connections 2024-03-18T16:31:13.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.849+0000 7f8facc00640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8f8806bc50 0x7f8f8806e110 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.849+0000 7f8facc00640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8fa80d0250 0x7f8fa8159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.849+0000 7f8facc00640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa80cfe70 0x7f8fa8158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.849+0000 7f8facc00640 1 --2- 172.21.15.102:0/303396718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8fa80cc2f0 0x7f8fa8155c50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:13.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.849+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 >> 172.21.15.102:0/303396718 conn(0x7f8fa808c3e0 msgr2=0x7f8fa80c3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:13.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.849+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 shutdown_connections 2024-03-18T16:31:13.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:13.849+0000 7f8facc00640 1 -- 172.21.15.102:0/303396718 wait complete. 2024-03-18T16:31:14.479 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:14.479 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:15.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:15.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.649+0000 7f5c43e00640 1 Processor -- start 2024-03-18T16:31:15.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.649+0000 7f5c43e00640 1 -- start start 2024-03-18T16:31:15.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.651+0000 7f5c43e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5c440cfe70 0x7f5c4415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:15.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.651+0000 7f5c43e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c440cabd0 con 0x7f5c440cc2f0 2024-03-18T16:31:15.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.651+0000 7f5c43e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c440caea0 con 0x7f5c440d0250 2024-03-18T16:31:15.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.651+0000 7f5c43e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5c440cb180 con 0x7f5c440cfe70 2024-03-18T16:31:15.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.652+0000 7f5c42000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f5c440d0250 0x7f5c440c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:53092/0 (socket says 172.21.15.102:53092) 2024-03-18T16:31:15.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.652+0000 7f5c42000640 1 -- 172.21.15.102:0/635629221 learned_addr learned my addr 172.21.15.102:0/635629221 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:15.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.652+0000 7f5c42a00640 1 --2- 172.21.15.102:0/635629221 >> v2:172.21.15.102:3300/0 conn(0x7f5c440cfe70 0x7f5c4415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:15.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.656+0000 7f5c41600640 1 -- 172.21.15.102:0/635629221 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3651379660 0 0) 0x7f5c440caea0 con 0x7f5c440d0250 2024-03-18T16:31:15.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.656+0000 7f5c41600640 1 -- 172.21.15.102:0/635629221 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c28003520 con 0x7f5c440d0250 2024-03-18T16:31:15.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.657+0000 7f5c42a00640 1 -- 172.21.15.102:0/635629221 >> v1:172.21.15.154:6789/0 conn(0x7f5c440d0250 legacy=0x7f5c440c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:15.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.657+0000 7f5c42a00640 1 -- 172.21.15.102:0/635629221 >> v1:172.21.15.102:6789/0 conn(0x7f5c440cc2f0 legacy=0x7f5c440cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:15.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.657+0000 7f5c42a00640 1 -- 172.21.15.102:0/635629221 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c440cb460 con 0x7f5c440cfe70 2024-03-18T16:31:15.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.664+0000 7f5c42a00640 1 --2- 172.21.15.102:0/635629221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c440cfe70 0x7f5c4415ae40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5c340097a0 tx=0x7f5c3402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47481aca808eadc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:15.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.669+0000 7f5c41600640 1 -- 172.21.15.102:0/635629221 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3402f810 con 0x7f5c440cfe70 2024-03-18T16:31:15.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.670+0000 7f5c41600640 1 -- 172.21.15.102:0/635629221 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c3402f9b0 con 0x7f5c440cfe70 2024-03-18T16:31:15.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.670+0000 7f5c41600640 1 -- 172.21.15.102:0/635629221 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3402fd80 con 0x7f5c440cfe70 2024-03-18T16:31:15.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.670+0000 7f5c43e00640 1 -- 172.21.15.102:0/635629221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c440cfe70 msgr2=0x7f5c4415ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:15.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.670+0000 7f5c43e00640 1 --2- 172.21.15.102:0/635629221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c440cfe70 0x7f5c4415ae40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5c340097a0 tx=0x7f5c3402b9f0 comp rx=0 tx=0).stop 2024-03-18T16:31:15.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.670+0000 7f5c43e00640 1 -- 172.21.15.102:0/635629221 shutdown_connections 2024-03-18T16:31:15.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.670+0000 7f5c43e00640 1 --2- 172.21.15.102:0/635629221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c440cfe70 0x7f5c4415ae40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:15.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.670+0000 7f5c43e00640 1 -- 172.21.15.102:0/635629221 >> 172.21.15.102:0/635629221 conn(0x7f5c4408c3e0 msgr2=0x7f5c4406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.671+0000 7f5c43e00640 1 -- 172.21.15.102:0/635629221 shutdown_connections 2024-03-18T16:31:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.671+0000 7f5c43e00640 1 -- 172.21.15.102:0/635629221 wait complete. 2024-03-18T16:31:15.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.672+0000 7f5c43e00640 1 Processor -- start 2024-03-18T16:31:15.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.673+0000 7f5c43e00640 1 -- start start 2024-03-18T16:31:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.673+0000 7f5c43e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c440cc2f0 0x7f5c440cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.673+0000 7f5c43e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c440cfe70 0x7f5c441562f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.673+0000 7f5c43e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c440d0250 0x7f5c441e0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.673+0000 7f5c43e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5c440cbd00 con 0x7f5c440d0250 2024-03-18T16:31:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.673+0000 7f5c43e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5c440cba20 con 0x7f5c440cc2f0 2024-03-18T16:31:15.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.673+0000 7f5c43e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5c440c8330 con 0x7f5c440cfe70 2024-03-18T16:31:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.674+0000 7f5c42000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c440cfe70 0x7f5c441562f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.674+0000 7f5c42a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c440cc2f0 0x7f5c440cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.674+0000 7f5c43400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c440d0250 0x7f5c441e0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.675+0000 7f5c42000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c440cfe70 0x7f5c441562f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:51612/0 (socket says 172.21.15.102:51612) 2024-03-18T16:31:15.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.675+0000 7f5c42000640 1 -- 172.21.15.102:0/1234110879 learned_addr learned my addr 172.21.15.102:0/1234110879 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:15.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.678+0000 7f5c42000640 1 -- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c440cc2f0 msgr2=0x7f5c440cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:15.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.678+0000 7f5c42000640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c440cc2f0 0x7f5c440cedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:15.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.678+0000 7f5c42000640 1 -- 172.21.15.102:0/1234110879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c440d0250 msgr2=0x7f5c441e0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:15.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.678+0000 7f5c42000640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c440d0250 0x7f5c441e0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:15.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.678+0000 7f5c42000640 1 -- 172.21.15.102:0/1234110879 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c44156c10 con 0x7f5c440cfe70 2024-03-18T16:31:15.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.681+0000 7f5c42000640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c440cfe70 0x7f5c441562f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000c3e0 tx=0x7f5c3000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:15.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.685+0000 7f5c3aa00640 1 -- 172.21.15.102:0/1234110879 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30017070 con 0x7f5c440cfe70 2024-03-18T16:31:15.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.685+0000 7f5c3aa00640 1 -- 172.21.15.102:0/1234110879 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c30005ce0 con 0x7f5c440cfe70 2024-03-18T16:31:15.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.685+0000 7f5c3aa00640 1 -- 172.21.15.102:0/1234110879 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c300072d0 con 0x7f5c440cfe70 2024-03-18T16:31:15.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.685+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c441f31b0 con 0x7f5c440cfe70 2024-03-18T16:31:15.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.685+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c441f3770 con 0x7f5c440cfe70 2024-03-18T16:31:15.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.686+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c44074fd0 con 0x7f5c440cfe70 2024-03-18T16:31:15.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.705+0000 7f5c3aa00640 1 -- 172.21.15.102:0/1234110879 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5c3001a070 con 0x7f5c440cfe70 2024-03-18T16:31:15.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.706+0000 7f5c3aa00640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5c2406bc50 0x7f5c2406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:15.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.707+0000 7f5c42a00640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5c2406bc50 0x7f5c2406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:15.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.707+0000 7f5c42a00640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5c2406bc50 0x7f5c2406e110 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5c34036000 tx=0x7f5c34008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:15.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.709+0000 7f5c3aa00640 1 -- 172.21.15.102:0/1234110879 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5c30056830 con 0x7f5c440cfe70 2024-03-18T16:31:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:15.961+0000 7f5c3aa00640 1 -- 172.21.15.102:0/1234110879 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5c30058790 con 0x7f5c440cfe70 2024-03-18T16:31:16.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.252+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c440ce660 con 0x7f5c440cfe70 2024-03-18T16:31:16.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.276+0000 7f5c3aa00640 1 -- 172.21.15.102:0/1234110879 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f5c3005c830 con 0x7f5c440cfe70 2024-03-18T16:31:16.278 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:16.278 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:16.279 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5c2406bc50 msgr2=0x7f5c2406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5c2406bc50 0x7f5c2406e110 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5c34036000 tx=0x7f5c34008030 comp rx=0 tx=0).stop 2024-03-18T16:31:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c440cfe70 msgr2=0x7f5c441562f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c440cfe70 0x7f5c441562f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000c3e0 tx=0x7f5c3000c8a0 comp rx=0 tx=0).stop 2024-03-18T16:31:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 shutdown_connections 2024-03-18T16:31:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5c2406bc50 0x7f5c2406e110 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:16.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c440d0250 0x7f5c441e0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:16.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5c440cfe70 0x7f5c441562f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:16.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 --2- 172.21.15.102:0/1234110879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c440cc2f0 0x7f5c440cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:16.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 >> 172.21.15.102:0/1234110879 conn(0x7f5c4408c3e0 msgr2=0x7f5c4415a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:16.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 shutdown_connections 2024-03-18T16:31:16.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:16.280+0000 7f5c43e00640 1 -- 172.21.15.102:0/1234110879 wait complete. 2024-03-18T16:31:16.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:16.911 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:17.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:18.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.087+0000 7f7ba2a00640 1 Processor -- start 2024-03-18T16:31:18.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.088+0000 7f7ba2a00640 1 -- start start 2024-03-18T16:31:18.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.089+0000 7f7ba2a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7b9c15a310 0x7f7b9c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:18.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.089+0000 7f7ba2a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9c0ca500 con 0x7f7b9c15a6f0 2024-03-18T16:31:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.089+0000 7f7ba2a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9c0ca7a0 con 0x7f7b9c15baa0 2024-03-18T16:31:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.089+0000 7f7ba2a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7b9c0cd410 con 0x7f7b9c15a310 2024-03-18T16:31:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.090+0000 7f7ba1600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7b9c15a310 0x7f7b9c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.090+0000 7f7ba0c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7b9c15a6f0 0x7f7b9c15ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41034/0 (socket says 172.21.15.102:41034) 2024-03-18T16:31:18.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.090+0000 7f7ba0c00640 1 -- 172.21.15.102:0/3878962122 learned_addr learned my addr 172.21.15.102:0/3878962122 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:18.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.093+0000 7f7b9be00640 1 -- 172.21.15.102:0/3878962122 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021887166 0 0) 0x7f7b9c0ca500 con 0x7f7b9c15a6f0 2024-03-18T16:31:18.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.093+0000 7f7b9be00640 1 -- 172.21.15.102:0/3878962122 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b84003520 con 0x7f7b9c15a6f0 2024-03-18T16:31:18.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.096+0000 7f7ba1600640 1 -- 172.21.15.102:0/3878962122 >> v1:172.21.15.154:6789/0 conn(0x7f7b9c15baa0 legacy=0x7f7b9c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:18.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.096+0000 7f7ba1600640 1 -- 172.21.15.102:0/3878962122 >> v1:172.21.15.102:6789/0 conn(0x7f7b9c15a6f0 legacy=0x7f7b9c15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:18.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.096+0000 7f7ba1600640 1 -- 172.21.15.102:0/3878962122 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9c0caa80 con 0x7f7b9c15a310 2024-03-18T16:31:18.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.100+0000 7f7ba1600640 1 --2- 172.21.15.102:0/3878962122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a310 0x7f7b9c06d3e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7b900097a0 tx=0x7f7b9002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3096d3cf39fcdbf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:18.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.104+0000 7f7b9be00640 1 -- 172.21.15.102:0/3878962122 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90034070 con 0x7f7b9c15a310 2024-03-18T16:31:18.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.105+0000 7f7b9be00640 1 -- 172.21.15.102:0/3878962122 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b90030610 con 0x7f7b9c15a310 2024-03-18T16:31:18.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.105+0000 7f7b9be00640 1 -- 172.21.15.102:0/3878962122 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b900309e0 con 0x7f7b9c15a310 2024-03-18T16:31:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.105+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3878962122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a310 msgr2=0x7f7b9c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.105+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3878962122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a310 0x7f7b9c06d3e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7b900097a0 tx=0x7f7b9002b960 comp rx=0 tx=0).stop 2024-03-18T16:31:18.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.105+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3878962122 shutdown_connections 2024-03-18T16:31:18.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.105+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3878962122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a310 0x7f7b9c06d3e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:18.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.105+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3878962122 >> 172.21.15.102:0/3878962122 conn(0x7f7b9c08c1a0 msgr2=0x7f7b9c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:18.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.106+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3878962122 shutdown_connections 2024-03-18T16:31:18.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.106+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3878962122 wait complete. 2024-03-18T16:31:18.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.107+0000 7f7ba2a00640 1 Processor -- start 2024-03-18T16:31:18.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.107+0000 7f7ba2a00640 1 -- start start 2024-03-18T16:31:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.108+0000 7f7ba2a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b9c15a310 0x7f7b9c0ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.108+0000 7f7ba2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a6f0 0x7f7b9c157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.108+0000 7f7ba2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c15baa0 0x7f7b9c1f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.108+0000 7f7ba2a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c06d920 con 0x7f7b9c15baa0 2024-03-18T16:31:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.108+0000 7f7ba2a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c0cd130 con 0x7f7b9c15a6f0 2024-03-18T16:31:18.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.108+0000 7f7ba2a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7b9c1548c0 con 0x7f7b9c15a310 2024-03-18T16:31:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.109+0000 7f7ba0c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a6f0 0x7f7b9c157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.109+0000 7f7ba1600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b9c15a310 0x7f7b9c0ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.109+0000 7f7ba2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c15baa0 0x7f7b9c1f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.109+0000 7f7ba0c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a6f0 0x7f7b9c157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56774/0 (socket says 172.21.15.102:56774) 2024-03-18T16:31:18.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.109+0000 7f7ba0c00640 1 -- 172.21.15.102:0/3145911376 learned_addr learned my addr 172.21.15.102:0/3145911376 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:18.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.113+0000 7f7ba1600640 1 -- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a6f0 msgr2=0x7f7b9c157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:18.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.113+0000 7f7ba1600640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a6f0 0x7f7b9c157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:18.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.113+0000 7f7ba1600640 1 -- 172.21.15.102:0/3145911376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c15baa0 msgr2=0x7f7b9c1f02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:18.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.113+0000 7f7ba1600640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c15baa0 0x7f7b9c1f02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:18.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.113+0000 7f7ba1600640 1 -- 172.21.15.102:0/3145911376 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9c1f37c0 con 0x7f7b9c15a310 2024-03-18T16:31:18.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.116+0000 7f7ba1600640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b9c15a310 0x7f7b9c0ce420 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7b90033000 tx=0x7f7b900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:18.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.120+0000 7f7b99600640 1 -- 172.21.15.102:0/3145911376 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90047070 con 0x7f7b9c15a310 2024-03-18T16:31:18.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.120+0000 7f7b99600640 1 -- 172.21.15.102:0/3145911376 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b90039990 con 0x7f7b9c15a310 2024-03-18T16:31:18.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.120+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b9c1f3aa0 con 0x7f7b9c15a310 2024-03-18T16:31:18.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.120+0000 7f7b99600640 1 -- 172.21.15.102:0/3145911376 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90034040 con 0x7f7b9c15a310 2024-03-18T16:31:18.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.121+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b9c1f4060 con 0x7f7b9c15a310 2024-03-18T16:31:18.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.122+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b9c074fd0 con 0x7f7b9c15a310 2024-03-18T16:31:18.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.142+0000 7f7b99600640 1 -- 172.21.15.102:0/3145911376 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7b9004a070 con 0x7f7b9c15a310 2024-03-18T16:31:18.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.143+0000 7f7b99600640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7b7c06bc20 0x7f7b7c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:18.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.144+0000 7f7ba0c00640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7b7c06bc20 0x7f7b7c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:18.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.144+0000 7f7ba0c00640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7b7c06bc20 0x7f7b7c06e0e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0cef20 tx=0x7f7b88009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:18.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.384+0000 7f7b99600640 1 -- 172.21.15.102:0/3145911376 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7b90078c80 con 0x7f7b9c15a310 2024-03-18T16:31:18.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.400+0000 7f7b99600640 1 -- 172.21.15.102:0/3145911376 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7b90051020 con 0x7f7b9c15a310 2024-03-18T16:31:18.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.658+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b9c153df0 con 0x7f7b9c15a310 2024-03-18T16:31:18.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.671+0000 7f7b99600640 1 -- 172.21.15.102:0/3145911376 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7b9007abe0 con 0x7f7b9c15a310 2024-03-18T16:31:18.673 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:18.673 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:18.673 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.674+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7b7c06bc20 msgr2=0x7f7b7c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.674+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7b7c06bc20 0x7f7b7c06e0e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c0cef20 tx=0x7f7b88009040 comp rx=0 tx=0).stop 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.674+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b9c15a310 msgr2=0x7f7b9c0ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.674+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b9c15a310 0x7f7b9c0ce420 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7b90033000 tx=0x7f7b900046c0 comp rx=0 tx=0).stop 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.675+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 shutdown_connections 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.675+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7b7c06bc20 0x7f7b7c06e0e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.675+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c15baa0 0x7f7b9c1f02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.675+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c15a6f0 0x7f7b9c157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.675+0000 7f7ba2a00640 1 --2- 172.21.15.102:0/3145911376 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7b9c15a310 0x7f7b9c0ce420 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:18.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.675+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 >> 172.21.15.102:0/3145911376 conn(0x7f7b9c08c1a0 msgr2=0x7f7b9c0676d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:18.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.675+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 shutdown_connections 2024-03-18T16:31:18.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:18.676+0000 7f7ba2a00640 1 -- 172.21.15.102:0/3145911376 wait complete. 2024-03-18T16:31:19.298 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:19.298 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:20.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:20.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.472+0000 7f5a7be00640 1 Processor -- start 2024-03-18T16:31:20.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.472+0000 7f5a7be00640 1 -- start start 2024-03-18T16:31:20.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.473+0000 7f5a7be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5a7c15a310 0x7f5a7c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:20.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.473+0000 7f5a7be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 0x7f5a7c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:20.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.473+0000 7f5a7be00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5a7c15baa0 0x7f5a7c15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:20.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.473+0000 7f5a7be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5a7c0cd410 con 0x7f5a7c15a310 2024-03-18T16:31:20.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.473+0000 7f5a7be00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5a7c15c470 con 0x7f5a7c15baa0 2024-03-18T16:31:20.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.473+0000 7f5a7be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5a7c0ca500 con 0x7f5a7c15a6f0 2024-03-18T16:31:20.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.474+0000 7f5a7b400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5a7c15baa0 0x7f5a7c15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:20.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.474+0000 7f5a7aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5a7c15a310 0x7f5a7c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:20.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.474+0000 7f5a7a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 0x7f5a7c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:20.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.474+0000 7f5a7b400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5a7c15baa0 0x7f5a7c15bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56788/0 (socket says 172.21.15.102:56788) 2024-03-18T16:31:20.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.474+0000 7f5a7b400640 1 -- 172.21.15.102:0/1562906393 learned_addr learned my addr 172.21.15.102:0/1562906393 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:20.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.476+0000 7f5a7b400640 1 -- 172.21.15.102:0/1562906393 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 msgr2=0x7f5a7c15ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:20.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.476+0000 7f5a7b400640 1 --2- 172.21.15.102:0/1562906393 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 0x7f5a7c15ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:20.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.476+0000 7f5a7b400640 1 -- 172.21.15.102:0/1562906393 >> v2:172.21.15.102:3300/0 conn(0x7f5a7c15a310 msgr2=0x7f5a7c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:20.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.476+0000 7f5a7b400640 1 --2- 172.21.15.102:0/1562906393 >> v2:172.21.15.102:3300/0 conn(0x7f5a7c15a310 0x7f5a7c06d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:20.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.476+0000 7f5a7b400640 1 -- 172.21.15.102:0/1562906393 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7c15b050 con 0x7f5a7c15baa0 2024-03-18T16:31:20.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.479+0000 7f5a7b400640 1 --2- 172.21.15.102:0/1562906393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15baa0 0x7f5a7c15bf30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0d0790 tx=0x7f5a7402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daff90e295077bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:20.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.483+0000 7f5a79600640 1 -- 172.21.15.102:0/1562906393 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7402ca80 con 0x7f5a7c15baa0 2024-03-18T16:31:20.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.483+0000 7f5a79600640 1 -- 172.21.15.102:0/1562906393 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a7402cc20 con 0x7f5a7c15baa0 2024-03-18T16:31:20.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.483+0000 7f5a79600640 1 -- 172.21.15.102:0/1562906393 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a740343f0 con 0x7f5a7c15baa0 2024-03-18T16:31:20.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.484+0000 7f5a7be00640 1 -- 172.21.15.102:0/1562906393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15baa0 msgr2=0x7f5a7c15bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.484+0000 7f5a7be00640 1 --2- 172.21.15.102:0/1562906393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15baa0 0x7f5a7c15bf30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0d0790 tx=0x7f5a7402bee0 comp rx=0 tx=0).stop 2024-03-18T16:31:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.484+0000 7f5a7be00640 1 -- 172.21.15.102:0/1562906393 shutdown_connections 2024-03-18T16:31:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.484+0000 7f5a7be00640 1 --2- 172.21.15.102:0/1562906393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15baa0 0x7f5a7c15bf30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.484+0000 7f5a7be00640 1 --2- 172.21.15.102:0/1562906393 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 0x7f5a7c15ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.484+0000 7f5a7be00640 1 --2- 172.21.15.102:0/1562906393 >> v2:172.21.15.102:3300/0 conn(0x7f5a7c15a310 0x7f5a7c06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.484+0000 7f5a7be00640 1 -- 172.21.15.102:0/1562906393 >> 172.21.15.102:0/1562906393 conn(0x7f5a7c08c1a0 msgr2=0x7f5a7c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.484+0000 7f5a7be00640 1 -- 172.21.15.102:0/1562906393 shutdown_connections 2024-03-18T16:31:20.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.484+0000 7f5a7be00640 1 -- 172.21.15.102:0/1562906393 wait complete. 2024-03-18T16:31:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.486+0000 7f5a7be00640 1 Processor -- start 2024-03-18T16:31:20.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.487+0000 7f5a7be00640 1 -- start start 2024-03-18T16:31:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.488+0000 7f5a7be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15a310 0x7f5a7c1e5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.488+0000 7f5a7be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 0x7f5a7c14bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.488+0000 7f5a7be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a7c15baa0 0x7f5a7c14c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.488+0000 7f5a7be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7c0cce50 con 0x7f5a7c15baa0 2024-03-18T16:31:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.488+0000 7f5a7be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7c0caa80 con 0x7f5a7c15a310 2024-03-18T16:31:20.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.488+0000 7f5a7be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5a7c0cd130 con 0x7f5a7c15a6f0 2024-03-18T16:31:20.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.489+0000 7f5a7aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15a310 0x7f5a7c1e5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:20.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.489+0000 7f5a7aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15a310 0x7f5a7c1e5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56792/0 (socket says 172.21.15.102:56792) 2024-03-18T16:31:20.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.489+0000 7f5a7aa00640 1 -- 172.21.15.102:0/4224971504 learned_addr learned my addr 172.21.15.102:0/4224971504 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:20.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.489+0000 7f5a7a000640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 0x7f5a7c14bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:20.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.489+0000 7f5a7b400640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a7c15baa0 0x7f5a7c14c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:20.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.492+0000 7f5a7aa00640 1 -- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 msgr2=0x7f5a7c14bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:20.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.492+0000 7f5a7aa00640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 0x7f5a7c14bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:20.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.492+0000 7f5a7aa00640 1 -- 172.21.15.102:0/4224971504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a7c15baa0 msgr2=0x7f5a7c14c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:20.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.492+0000 7f5a7aa00640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a7c15baa0 0x7f5a7c14c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:20.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.492+0000 7f5a7aa00640 1 -- 172.21.15.102:0/4224971504 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7c159020 con 0x7f5a7c15a310 2024-03-18T16:31:20.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.495+0000 7f5a7aa00640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15a310 0x7f5a7c1e5ed0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00e870 tx=0x7f5a6c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:20.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.499+0000 7f5a73400640 1 -- 172.21.15.102:0/4224971504 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c017070 con 0x7f5a7c15a310 2024-03-18T16:31:20.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.499+0000 7f5a73400640 1 -- 172.21.15.102:0/4224971504 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a6c00ee60 con 0x7f5a7c15a310 2024-03-18T16:31:20.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.499+0000 7f5a73400640 1 -- 172.21.15.102:0/4224971504 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c00cb80 con 0x7f5a7c15a310 2024-03-18T16:31:20.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.499+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7c159470 con 0x7f5a7c15a310 2024-03-18T16:31:20.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.499+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a7c159a30 con 0x7f5a7c15a310 2024-03-18T16:31:20.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.500+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a7c074fd0 con 0x7f5a7c15a310 2024-03-18T16:31:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.759+0000 7f5a73400640 1 -- 172.21.15.102:0/4224971504 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5a6c01a070 con 0x7f5a7c15a310 2024-03-18T16:31:20.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.760+0000 7f5a73400640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a5c06bc50 0x7f5a5c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:20.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.761+0000 7f5a7a000640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a5c06bc50 0x7f5a5c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:20.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.762+0000 7f5a7a000640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a5c06bc50 0x7f5a5c06e110 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c158ac0 tx=0x7f5a68009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.767+0000 7f5a73400640 1 -- 172.21.15.102:0/4224971504 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5a6c057520 con 0x7f5a7c15a310 2024-03-18T16:31:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:20.778+0000 7f5a73400640 1 -- 172.21.15.102:0/4224971504 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5a6c059480 con 0x7f5a7c15a310 2024-03-18T16:31:21.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.072+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a7c153a00 con 0x7f5a7c15a310 2024-03-18T16:31:21.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.208+0000 7f5a73400640 1 -- 172.21.15.102:0/4224971504 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f5a6c05d520 con 0x7f5a7c15a310 2024-03-18T16:31:21.210 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:21.210 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:21.210 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.211+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a5c06bc50 msgr2=0x7f5a5c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a5c06bc50 0x7f5a5c06e110 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c158ac0 tx=0x7f5a68009190 comp rx=0 tx=0).stop 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15a310 msgr2=0x7f5a7c1e5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15a310 0x7f5a7c1e5ed0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c00e870 tx=0x7f5a6c00ed30 comp rx=0 tx=0).stop 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 shutdown_connections 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5a5c06bc50 0x7f5a5c06e110 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a7c15baa0 0x7f5a7c14c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5a7c15a6f0 0x7f5a7c14bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 --2- 172.21.15.102:0/4224971504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a7c15a310 0x7f5a7c1e5ed0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 >> 172.21.15.102:0/4224971504 conn(0x7f5a7c08c1a0 msgr2=0x7f5a7c14ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:21.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 shutdown_connections 2024-03-18T16:31:21.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:21.212+0000 7f5a7be00640 1 -- 172.21.15.102:0/4224971504 wait complete. 2024-03-18T16:31:21.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:21.845 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:22.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:23.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.033+0000 7f9952a00640 1 Processor -- start 2024-03-18T16:31:23.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.034+0000 7f9952a00640 1 -- start start 2024-03-18T16:31:23.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.035+0000 7f9952a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f994c0cfe70 0x7f994c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:23.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.035+0000 7f9952a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0cc2f0 0x7f994c0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:23.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.035+0000 7f9952a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f994c0cabd0 con 0x7f994c0d0250 2024-03-18T16:31:23.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.035+0000 7f9952a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f994c0caea0 con 0x7f994c0cfe70 2024-03-18T16:31:23.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.035+0000 7f9952a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f994c0cb180 con 0x7f994c0cc2f0 2024-03-18T16:31:23.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.035+0000 7f9952000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0cc2f0 0x7f994c0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:23.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.036+0000 7f9952000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0cc2f0 0x7f994c0cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56492/0 (socket says 172.21.15.102:56492) 2024-03-18T16:31:23.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.036+0000 7f9952000640 1 -- 172.21.15.102:0/648234469 learned_addr learned my addr 172.21.15.102:0/648234469 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:23.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.036+0000 7f9951600640 1 --2- 172.21.15.102:0/648234469 >> v2:172.21.15.154:3300/0 conn(0x7f994c0cfe70 0x7f994c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:23.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.038+0000 7f9952000640 1 -- 172.21.15.102:0/648234469 >> v2:172.21.15.154:3300/0 conn(0x7f994c0cfe70 msgr2=0x7f994c15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:23.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.038+0000 7f9952000640 1 --2- 172.21.15.102:0/648234469 >> v2:172.21.15.154:3300/0 conn(0x7f994c0cfe70 0x7f994c15ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.038+0000 7f9952000640 1 -- 172.21.15.102:0/648234469 >> v1:172.21.15.154:6789/0 conn(0x7f994c0d0250 legacy=0x7f994c0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:23.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.038+0000 7f9952000640 1 -- 172.21.15.102:0/648234469 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994c0cb460 con 0x7f994c0cc2f0 2024-03-18T16:31:23.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.041+0000 7f9952000640 1 --2- 172.21.15.102:0/648234469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0cc2f0 0x7f994c0cc710 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9944009e50 tx=0x7f994402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36e2a4d3d7dcdce1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:23.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.045+0000 7f994be00640 1 -- 172.21.15.102:0/648234469 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994402cb90 con 0x7f994c0cc2f0 2024-03-18T16:31:23.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.045+0000 7f994be00640 1 -- 172.21.15.102:0/648234469 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f994402cd30 con 0x7f994c0cc2f0 2024-03-18T16:31:23.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.045+0000 7f994be00640 1 -- 172.21.15.102:0/648234469 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944034610 con 0x7f994c0cc2f0 2024-03-18T16:31:23.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.045+0000 7f9952a00640 1 -- 172.21.15.102:0/648234469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0cc2f0 msgr2=0x7f994c0cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:23.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.045+0000 7f9952a00640 1 --2- 172.21.15.102:0/648234469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0cc2f0 0x7f994c0cc710 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9944009e50 tx=0x7f994402bff0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.046+0000 7f9952a00640 1 -- 172.21.15.102:0/648234469 shutdown_connections 2024-03-18T16:31:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.046+0000 7f9952a00640 1 --2- 172.21.15.102:0/648234469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0cc2f0 0x7f994c0cc710 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.046+0000 7f9952a00640 1 --2- 172.21.15.102:0/648234469 >> v2:172.21.15.154:3300/0 conn(0x7f994c0cfe70 0x7f994c15ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.046+0000 7f9952a00640 1 -- 172.21.15.102:0/648234469 >> 172.21.15.102:0/648234469 conn(0x7f994c08c3e0 msgr2=0x7f994c06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.046+0000 7f9952a00640 1 -- 172.21.15.102:0/648234469 shutdown_connections 2024-03-18T16:31:23.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.046+0000 7f9952a00640 1 -- 172.21.15.102:0/648234469 wait complete. 2024-03-18T16:31:23.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.048+0000 7f9952a00640 1 Processor -- start 2024-03-18T16:31:23.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.048+0000 7f9952a00640 1 -- start start 2024-03-18T16:31:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.049+0000 7f9952a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c0cc2f0 0x7f994c1e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.049+0000 7f9952a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f994c0cfe70 0x7f994c158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.049+0000 7f9952a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0d0250 0x7f994c159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.049+0000 7f9952a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f994c0cbd00 con 0x7f994c0cc2f0 2024-03-18T16:31:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.049+0000 7f9952a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f994c0cba20 con 0x7f994c0cfe70 2024-03-18T16:31:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.049+0000 7f9952a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f994c0ccc50 con 0x7f994c0d0250 2024-03-18T16:31:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.049+0000 7f9952000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0d0250 0x7f994c159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.049+0000 7f9950c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f994c0cfe70 0x7f994c158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.050+0000 7f9951600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c0cc2f0 0x7f994c1e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.050+0000 7f9952000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0d0250 0x7f994c159470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56496/0 (socket says 172.21.15.102:56496) 2024-03-18T16:31:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.050+0000 7f9952000640 1 -- 172.21.15.102:0/634496881 learned_addr learned my addr 172.21.15.102:0/634496881 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.053+0000 7f9952000640 1 -- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f994c0cfe70 msgr2=0x7f994c158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.053+0000 7f9952000640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f994c0cfe70 0x7f994c158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.053+0000 7f9952000640 1 -- 172.21.15.102:0/634496881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c0cc2f0 msgr2=0x7f994c1e03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.053+0000 7f9952000640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c0cc2f0 0x7f994c1e03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.053+0000 7f9952000640 1 -- 172.21.15.102:0/634496881 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994c1e3250 con 0x7f994c0d0250 2024-03-18T16:31:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.056+0000 7f9952000640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0d0250 0x7f994c159470 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9944036040 tx=0x7f9944004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.060+0000 7f9949600640 1 -- 172.21.15.102:0/634496881 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944043020 con 0x7f994c0d0250 2024-03-18T16:31:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.060+0000 7f9949600640 1 -- 172.21.15.102:0/634496881 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f994403e450 con 0x7f994c0d0250 2024-03-18T16:31:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.060+0000 7f9949600640 1 -- 172.21.15.102:0/634496881 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944009510 con 0x7f994c0d0250 2024-03-18T16:31:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.060+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f994c1efea0 con 0x7f994c0d0250 2024-03-18T16:31:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.060+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f994c1f0460 con 0x7f994c0d0250 2024-03-18T16:31:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.062+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f994c074fd0 con 0x7f994c0d0250 2024-03-18T16:31:23.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:31:23.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:31:23.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.083+0000 7f9949600640 1 -- 172.21.15.102:0/634496881 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9944046020 con 0x7f994c0d0250 2024-03-18T16:31:23.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.084+0000 7f9949600640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f992c06bbd0 0x7f992c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:23.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.084+0000 7f9949600640 1 -- 172.21.15.102:0/634496881 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9944079860 con 0x7f994c0d0250 2024-03-18T16:31:23.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.085+0000 7f9951600640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f992c06bbd0 0x7f992c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:23.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.086+0000 7f9951600640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f992c06bbd0 0x7f992c06e090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9940009b40 tx=0x7f9940009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:23.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.334+0000 7f9949600640 1 -- 172.21.15.102:0/634496881 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f994404d030 con 0x7f994c0d0250 2024-03-18T16:31:23.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.621+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f994c0c6930 con 0x7f994c0d0250 2024-03-18T16:31:23.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.700+0000 7f9949600640 1 -- 172.21.15.102:0/634496881 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f994407b7c0 con 0x7f994c0d0250 2024-03-18T16:31:23.703 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:23.703 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:23.703 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:23.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f992c06bbd0 msgr2=0x7f992c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:23.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f992c06bbd0 0x7f992c06e090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9940009b40 tx=0x7f9940009190 comp rx=0 tx=0).stop 2024-03-18T16:31:23.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0d0250 msgr2=0x7f994c159470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:23.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0d0250 0x7f994c159470 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9944036040 tx=0x7f9944004a40 comp rx=0 tx=0).stop 2024-03-18T16:31:23.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 shutdown_connections 2024-03-18T16:31:23.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f992c06bbd0 0x7f992c06e090 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f994c0d0250 0x7f994c159470 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f994c0cfe70 0x7f994c158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 --2- 172.21.15.102:0/634496881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c0cc2f0 0x7f994c1e03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:23.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 >> 172.21.15.102:0/634496881 conn(0x7f994c08c3e0 msgr2=0x7f994c0cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:23.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.704+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 shutdown_connections 2024-03-18T16:31:23.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:23.705+0000 7f9952a00640 1 -- 172.21.15.102:0/634496881 wait complete. 2024-03-18T16:31:24.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:24.334 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:25.335 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:25.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.508+0000 7f2c45e00640 1 Processor -- start 2024-03-18T16:31:25.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.509+0000 7f2c45e00640 1 -- start start 2024-03-18T16:31:25.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.510+0000 7f2c45e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2c400cad40 0x7f2c4006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:25.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.510+0000 7f2c45e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2c4015a1a0 0x7f2c400705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:25.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.510+0000 7f2c45e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4006c9c0 con 0x7f2c400ca960 2024-03-18T16:31:25.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.510+0000 7f2c45e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f2c400c8680 con 0x7f2c4015a1a0 2024-03-18T16:31:25.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.510+0000 7f2c45e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2c4006cca0 con 0x7f2c400cad40 2024-03-18T16:31:25.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.511+0000 7f2c3fe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2c400cad40 0x7f2c4006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:25.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.511+0000 7f2c45400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2c4015a1a0 0x7f2c400705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:25.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.511+0000 7f2c3fe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2c400cad40 0x7f2c4006c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:50254/0 (socket says 172.21.15.102:50254) 2024-03-18T16:31:25.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.511+0000 7f2c3fe00640 1 -- 172.21.15.102:0/1107915020 learned_addr learned my addr 172.21.15.102:0/1107915020 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:25.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.513+0000 7f2c3fe00640 1 -- 172.21.15.102:0/1107915020 >> v2:172.21.15.102:3300/0 conn(0x7f2c4015a1a0 msgr2=0x7f2c400705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:25.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.513+0000 7f2c3fe00640 1 --2- 172.21.15.102:0/1107915020 >> v2:172.21.15.102:3300/0 conn(0x7f2c4015a1a0 0x7f2c400705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:25.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.513+0000 7f2c3fe00640 1 -- 172.21.15.102:0/1107915020 >> v1:172.21.15.102:6789/0 conn(0x7f2c400ca960 legacy=0x7f2c4015e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:25.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.513+0000 7f2c3fe00640 1 -- 172.21.15.102:0/1107915020 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4006cf80 con 0x7f2c400cad40 2024-03-18T16:31:25.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.516+0000 7f2c3fe00640 1 --2- 172.21.15.102:0/1107915020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400cad40 0x7f2c4006c310 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2c400cb1f0 tx=0x7f2c2c02ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61f2b4d582b332ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:25.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.520+0000 7f2c3f400640 1 -- 172.21.15.102:0/1107915020 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c02f850 con 0x7f2c400cad40 2024-03-18T16:31:25.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.520+0000 7f2c3f400640 1 -- 172.21.15.102:0/1107915020 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c2c02f9f0 con 0x7f2c400cad40 2024-03-18T16:31:25.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.520+0000 7f2c3f400640 1 -- 172.21.15.102:0/1107915020 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c02fdc0 con 0x7f2c400cad40 2024-03-18T16:31:25.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.521+0000 7f2c45e00640 1 -- 172.21.15.102:0/1107915020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400cad40 msgr2=0x7f2c4006c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:25.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.521+0000 7f2c45e00640 1 --2- 172.21.15.102:0/1107915020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400cad40 0x7f2c4006c310 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2c400cb1f0 tx=0x7f2c2c02ba30 comp rx=0 tx=0).stop 2024-03-18T16:31:25.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.521+0000 7f2c45e00640 1 -- 172.21.15.102:0/1107915020 shutdown_connections 2024-03-18T16:31:25.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.521+0000 7f2c45e00640 1 --2- 172.21.15.102:0/1107915020 >> v2:172.21.15.102:3300/0 conn(0x7f2c4015a1a0 0x7f2c400705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:25.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.521+0000 7f2c45e00640 1 --2- 172.21.15.102:0/1107915020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400cad40 0x7f2c4006c310 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:25.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.521+0000 7f2c45e00640 1 -- 172.21.15.102:0/1107915020 >> 172.21.15.102:0/1107915020 conn(0x7f2c40067190 msgr2=0x7f2c400675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:25.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.521+0000 7f2c45e00640 1 -- 172.21.15.102:0/1107915020 shutdown_connections 2024-03-18T16:31:25.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.521+0000 7f2c45e00640 1 -- 172.21.15.102:0/1107915020 wait complete. 2024-03-18T16:31:25.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.523+0000 7f2c45e00640 1 Processor -- start 2024-03-18T16:31:25.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.523+0000 7f2c45e00640 1 -- start start 2024-03-18T16:31:25.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.524+0000 7f2c45e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ca960 0x7f2c40159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.524+0000 7f2c45e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c400cad40 0x7f2c401e5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.524+0000 7f2c45e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c4015a1a0 0x7f2c400ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.524+0000 7f2c45e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2c40070ae0 con 0x7f2c4015a1a0 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.524+0000 7f2c45e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4015dc10 con 0x7f2c400ca960 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.524+0000 7f2c45e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2c40070c60 con 0x7f2c400cad40 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.525+0000 7f2c44a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ca960 0x7f2c40159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.525+0000 7f2c3fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c400cad40 0x7f2c401e5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.525+0000 7f2c45400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c4015a1a0 0x7f2c400ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.525+0000 7f2c44a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ca960 0x7f2c40159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:50262/0 (socket says 172.21.15.102:50262) 2024-03-18T16:31:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.525+0000 7f2c44a00640 1 -- 172.21.15.102:0/4023933551 learned_addr learned my addr 172.21.15.102:0/4023933551 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.529+0000 7f2c3fe00640 1 -- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ca960 msgr2=0x7f2c40159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.529+0000 7f2c3fe00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ca960 0x7f2c40159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.529+0000 7f2c3fe00640 1 -- 172.21.15.102:0/4023933551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c4015a1a0 msgr2=0x7f2c400ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.529+0000 7f2c3fe00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c4015a1a0 0x7f2c400ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.529+0000 7f2c3fe00640 1 -- 172.21.15.102:0/4023933551 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c40154ec0 con 0x7f2c400cad40 2024-03-18T16:31:25.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.532+0000 7f2c3fe00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c400cad40 0x7f2c401e5420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2c400c8dc0 tx=0x7f2c2c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:25.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.536+0000 7f2c3cc00640 1 -- 172.21.15.102:0/4023933551 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c047070 con 0x7f2c400cad40 2024-03-18T16:31:25.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.536+0000 7f2c3cc00640 1 -- 172.21.15.102:0/4023933551 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c2c037be0 con 0x7f2c400cad40 2024-03-18T16:31:25.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.536+0000 7f2c3cc00640 1 -- 172.21.15.102:0/4023933551 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c042480 con 0x7f2c400cad40 2024-03-18T16:31:25.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.536+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c401f3d40 con 0x7f2c400cad40 2024-03-18T16:31:25.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.536+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c401e4940 con 0x7f2c400cad40 2024-03-18T16:31:25.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.537+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c400678a0 con 0x7f2c400cad40 2024-03-18T16:31:25.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.562+0000 7f2c3cc00640 1 -- 172.21.15.102:0/4023933551 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2c2c04a070 con 0x7f2c400cad40 2024-03-18T16:31:25.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.563+0000 7f2c3cc00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2c2006bbd0 0x7f2c2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:25.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.563+0000 7f2c44a00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2c2006bbd0 0x7f2c2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:25.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.563+0000 7f2c3cc00640 1 -- 172.21.15.102:0/4023933551 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f2c2c0799d0 con 0x7f2c400cad40 2024-03-18T16:31:25.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.564+0000 7f2c44a00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2c2006bbd0 0x7f2c2006e090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2c34001840 tx=0x7f2c34009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:25.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:25.841+0000 7f2c3cc00640 1 -- 172.21.15.102:0/4023933551 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2c400678a0 con 0x7f2c400cad40 2024-03-18T16:31:26.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.087+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c40074fd0 con 0x7f2c400cad40 2024-03-18T16:31:26.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.101+0000 7f2c3cc00640 1 -- 172.21.15.102:0/4023933551 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f2c2c033030 con 0x7f2c400cad40 2024-03-18T16:31:26.104 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:26.104 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:26.104 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:26.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2c2006bbd0 msgr2=0x7f2c2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:26.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2c2006bbd0 0x7f2c2006e090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2c34001840 tx=0x7f2c34009040 comp rx=0 tx=0).stop 2024-03-18T16:31:26.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c400cad40 msgr2=0x7f2c401e5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:26.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c400cad40 0x7f2c401e5420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2c400c8dc0 tx=0x7f2c2c004730 comp rx=0 tx=0).stop 2024-03-18T16:31:26.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 shutdown_connections 2024-03-18T16:31:26.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2c2006bbd0 0x7f2c2006e090 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:26.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c4015a1a0 0x7f2c400ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:26.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2c400cad40 0x7f2c401e5420 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:26.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 --2- 172.21.15.102:0/4023933551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ca960 0x7f2c40159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:26.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 >> 172.21.15.102:0/4023933551 conn(0x7f2c40067190 msgr2=0x7f2c40153f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:26.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 shutdown_connections 2024-03-18T16:31:26.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:26.105+0000 7f2c45e00640 1 -- 172.21.15.102:0/4023933551 wait complete. 2024-03-18T16:31:26.727 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:26.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:27.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:27.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.896+0000 7fd143e00640 1 Processor -- start 2024-03-18T16:31:27.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.896+0000 7fd143e00640 1 -- start start 2024-03-18T16:31:27.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.897+0000 7fd143e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd144141650 0x7fd144158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.897+0000 7fd143e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1440c72d0 con 0x7fd14415c3a0 2024-03-18T16:31:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.897+0000 7fd143e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd144158ad0 con 0x7fd1440c4fa0 2024-03-18T16:31:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.897+0000 7fd143e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd144159370 con 0x7fd144141650 2024-03-18T16:31:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.898+0000 7fd142a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd144141650 0x7fd144158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:27.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.898+0000 7fd142000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd14415c3a0 0x7fd1440c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40480/0 (socket says 172.21.15.102:40480) 2024-03-18T16:31:27.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.898+0000 7fd142000640 1 -- 172.21.15.102:0/3032740327 learned_addr learned my addr 172.21.15.102:0/3032740327 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:27.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.901+0000 7fd142a00640 1 -- 172.21.15.102:0/3032740327 >> v1:172.21.15.154:6789/0 conn(0x7fd1440c4fa0 legacy=0x7fd144162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:27.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.901+0000 7fd142a00640 1 -- 172.21.15.102:0/3032740327 >> v1:172.21.15.102:6789/0 conn(0x7fd14415c3a0 legacy=0x7fd1440c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:27.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.901+0000 7fd142a00640 1 -- 172.21.15.102:0/3032740327 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd144158db0 con 0x7fd144141650 2024-03-18T16:31:27.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.901+0000 7fd141600640 1 -- 172.21.15.102:0/3032740327 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988432076 0 0) 0x7fd1440c72d0 con 0x7fd14415c3a0 2024-03-18T16:31:27.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.906+0000 7fd142a00640 1 --2- 172.21.15.102:0/3032740327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd144141650 0x7fd144158420 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd1340097a0 tx=0x7fd13402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75cd55ea2055205 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:27.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.910+0000 7fd141600640 1 -- 172.21.15.102:0/3032740327 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd134034070 con 0x7fd144141650 2024-03-18T16:31:27.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.910+0000 7fd141600640 1 -- 172.21.15.102:0/3032740327 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd134030610 con 0x7fd144141650 2024-03-18T16:31:27.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.910+0000 7fd141600640 1 -- 172.21.15.102:0/3032740327 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1340309e0 con 0x7fd144141650 2024-03-18T16:31:27.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.910+0000 7fd143e00640 1 -- 172.21.15.102:0/3032740327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd144141650 msgr2=0x7fd144158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:27.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.910+0000 7fd143e00640 1 --2- 172.21.15.102:0/3032740327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd144141650 0x7fd144158420 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd1340097a0 tx=0x7fd13402b960 comp rx=0 tx=0).stop 2024-03-18T16:31:27.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.910+0000 7fd143e00640 1 -- 172.21.15.102:0/3032740327 shutdown_connections 2024-03-18T16:31:27.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.910+0000 7fd143e00640 1 --2- 172.21.15.102:0/3032740327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd144141650 0x7fd144158420 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:27.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.910+0000 7fd143e00640 1 -- 172.21.15.102:0/3032740327 >> 172.21.15.102:0/3032740327 conn(0x7fd1440cb890 msgr2=0x7fd1440cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:27.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.911+0000 7fd143e00640 1 -- 172.21.15.102:0/3032740327 shutdown_connections 2024-03-18T16:31:27.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.911+0000 7fd143e00640 1 -- 172.21.15.102:0/3032740327 wait complete. 2024-03-18T16:31:27.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.912+0000 7fd143e00640 1 Processor -- start 2024-03-18T16:31:27.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.912+0000 7fd143e00640 1 -- start start 2024-03-18T16:31:27.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.913+0000 7fd143e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1440c4fa0 0x7fd1441ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:27.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.913+0000 7fd143e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd144141650 0x7fd1441f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:27.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.913+0000 7fd143e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd14415c3a0 0x7fd1441e5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:27.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.913+0000 7fd143e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd1440c66a0 con 0x7fd1440c4fa0 2024-03-18T16:31:27.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.913+0000 7fd143e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1440c63c0 con 0x7fd14415c3a0 2024-03-18T16:31:27.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.913+0000 7fd143e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd14415c7c0 con 0x7fd144141650 2024-03-18T16:31:27.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.914+0000 7fd143400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd14415c3a0 0x7fd1441e5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:27.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.914+0000 7fd142a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1440c4fa0 0x7fd1441ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:27.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.914+0000 7fd142000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd144141650 0x7fd1441f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:27.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.914+0000 7fd143400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd14415c3a0 0x7fd1441e5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:50274/0 (socket says 172.21.15.102:50274) 2024-03-18T16:31:27.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.914+0000 7fd143400640 1 -- 172.21.15.102:0/1395626991 learned_addr learned my addr 172.21.15.102:0/1395626991 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:27.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.918+0000 7fd143400640 1 -- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd144141650 msgr2=0x7fd1441f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:27.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.918+0000 7fd143400640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd144141650 0x7fd1441f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:27.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.918+0000 7fd143400640 1 -- 172.21.15.102:0/1395626991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1440c4fa0 msgr2=0x7fd1441ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:27.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.918+0000 7fd143400640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1440c4fa0 0x7fd1441ee220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:27.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.918+0000 7fd143400640 1 -- 172.21.15.102:0/1395626991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1441567e0 con 0x7fd14415c3a0 2024-03-18T16:31:27.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.920+0000 7fd143400640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd14415c3a0 0x7fd1441e5240 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00c7a0 tx=0x7fd13c00cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:27.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.924+0000 7fd13aa00640 1 -- 172.21.15.102:0/1395626991 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c003b70 con 0x7fd14415c3a0 2024-03-18T16:31:27.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.925+0000 7fd13aa00640 1 -- 172.21.15.102:0/1395626991 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd13c003d30 con 0x7fd14415c3a0 2024-03-18T16:31:27.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.925+0000 7fd13aa00640 1 -- 172.21.15.102:0/1395626991 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c00aa90 con 0x7fd14415c3a0 2024-03-18T16:31:27.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.925+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd144156ac0 con 0x7fd14415c3a0 2024-03-18T16:31:27.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.925+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1441e3c10 con 0x7fd14415c3a0 2024-03-18T16:31:27.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:27.926+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd144074fd0 con 0x7fd14415c3a0 2024-03-18T16:31:28.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.198+0000 7fd13aa00640 1 -- 172.21.15.102:0/1395626991 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd13c00ac30 con 0x7fd14415c3a0 2024-03-18T16:31:28.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.201+0000 7fd13aa00640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd12406bc20 0x7fd12406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:28.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.201+0000 7fd142a00640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd12406bc20 0x7fd12406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:28.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.202+0000 7fd13aa00640 1 -- 172.21.15.102:0/1395626991 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd13c0578b0 con 0x7fd14415c3a0 2024-03-18T16:31:28.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.202+0000 7fd142a00640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd12406bc20 0x7fd12406e0e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd134036000 tx=0x7fd13403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:28.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.219+0000 7fd13aa00640 1 -- 172.21.15.102:0/1395626991 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd13c059810 con 0x7fd14415c3a0 2024-03-18T16:31:28.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.459+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd144156c90 con 0x7fd14415c3a0 2024-03-18T16:31:28.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.494+0000 7fd13aa00640 1 -- 172.21.15.102:0/1395626991 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd13c05d8b0 con 0x7fd14415c3a0 2024-03-18T16:31:28.496 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:28.496 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:28.496 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:28.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd12406bc20 msgr2=0x7fd12406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:28.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd12406bc20 0x7fd12406e0e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd134036000 tx=0x7fd13403b040 comp rx=0 tx=0).stop 2024-03-18T16:31:28.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd14415c3a0 msgr2=0x7fd1441e5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:28.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd14415c3a0 0x7fd1441e5240 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00c7a0 tx=0x7fd13c00cc60 comp rx=0 tx=0).stop 2024-03-18T16:31:28.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 shutdown_connections 2024-03-18T16:31:28.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd12406bc20 0x7fd12406e0e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:28.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd14415c3a0 0x7fd1441e5240 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:28.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd144141650 0x7fd1441f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:28.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 --2- 172.21.15.102:0/1395626991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1440c4fa0 0x7fd1441ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:28.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 >> 172.21.15.102:0/1395626991 conn(0x7fd1440cb890 msgr2=0x7fd1440cc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:28.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 shutdown_connections 2024-03-18T16:31:28.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:28.497+0000 7fd143e00640 1 -- 172.21.15.102:0/1395626991 wait complete. 2024-03-18T16:31:29.126 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:29.126 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:30.127 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:30.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.296+0000 7fbff6400640 1 Processor -- start 2024-03-18T16:31:30.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.296+0000 7fbff6400640 1 -- start start 2024-03-18T16:31:30.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.297+0000 7fbff6400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbff00cad40 0x7fbff006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:30.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.297+0000 7fbff6400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbff015a1a0 0x7fbff00705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:30.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.297+0000 7fbff6400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff006c9c0 con 0x7fbff00ca960 2024-03-18T16:31:30.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.297+0000 7fbff6400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbff00c8680 con 0x7fbff00cad40 2024-03-18T16:31:30.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.297+0000 7fbff6400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbff006cca0 con 0x7fbff015a1a0 2024-03-18T16:31:30.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.298+0000 7fbff5a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbff015a1a0 0x7fbff00705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:30.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.298+0000 7fbfefe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbff00cad40 0x7fbff006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:30.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.299+0000 7fbff5a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbff015a1a0 0x7fbff00705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:50284/0 (socket says 172.21.15.102:50284) 2024-03-18T16:31:30.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.299+0000 7fbff5a00640 1 -- 172.21.15.102:0/3181437628 learned_addr learned my addr 172.21.15.102:0/3181437628 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:30.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.301+0000 7fbff5a00640 1 -- 172.21.15.102:0/3181437628 >> v2:172.21.15.102:3300/0 conn(0x7fbff00cad40 msgr2=0x7fbff006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:30.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.301+0000 7fbff5a00640 1 --2- 172.21.15.102:0/3181437628 >> v2:172.21.15.102:3300/0 conn(0x7fbff00cad40 0x7fbff006c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:30.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.301+0000 7fbff5a00640 1 -- 172.21.15.102:0/3181437628 >> v1:172.21.15.102:6789/0 conn(0x7fbff00ca960 legacy=0x7fbff015e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:30.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.301+0000 7fbff5a00640 1 -- 172.21.15.102:0/3181437628 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff006cf80 con 0x7fbff015a1a0 2024-03-18T16:31:30.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.303+0000 7fbff5a00640 1 --2- 172.21.15.102:0/3181437628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff015a1a0 0x7fbff00705a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbff00b2f30 tx=0x7fbfe802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a64c02baf77740a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:30.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.307+0000 7fbfef400640 1 -- 172.21.15.102:0/3181437628 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe802cb20 con 0x7fbff015a1a0 2024-03-18T16:31:30.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.307+0000 7fbfef400640 1 -- 172.21.15.102:0/3181437628 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe802ccc0 con 0x7fbff015a1a0 2024-03-18T16:31:30.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.307+0000 7fbfef400640 1 -- 172.21.15.102:0/3181437628 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe8034610 con 0x7fbff015a1a0 2024-03-18T16:31:30.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.308+0000 7fbff6400640 1 -- 172.21.15.102:0/3181437628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff015a1a0 msgr2=0x7fbff00705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:30.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.308+0000 7fbff6400640 1 --2- 172.21.15.102:0/3181437628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff015a1a0 0x7fbff00705a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fbff00b2f30 tx=0x7fbfe802bf80 comp rx=0 tx=0).stop 2024-03-18T16:31:30.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.308+0000 7fbff6400640 1 -- 172.21.15.102:0/3181437628 shutdown_connections 2024-03-18T16:31:30.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.308+0000 7fbff6400640 1 --2- 172.21.15.102:0/3181437628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff015a1a0 0x7fbff00705a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:30.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.308+0000 7fbff6400640 1 --2- 172.21.15.102:0/3181437628 >> v2:172.21.15.102:3300/0 conn(0x7fbff00cad40 0x7fbff006c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:30.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.308+0000 7fbff6400640 1 -- 172.21.15.102:0/3181437628 >> 172.21.15.102:0/3181437628 conn(0x7fbff0067190 msgr2=0x7fbff00675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:30.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.308+0000 7fbff6400640 1 -- 172.21.15.102:0/3181437628 shutdown_connections 2024-03-18T16:31:30.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.308+0000 7fbff6400640 1 -- 172.21.15.102:0/3181437628 wait complete. 2024-03-18T16:31:30.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.311+0000 7fbff6400640 1 Processor -- start 2024-03-18T16:31:30.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.311+0000 7fbff6400640 1 -- start start 2024-03-18T16:31:30.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.313+0000 7fbff6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbff00ca960 0x7fbff0158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:30.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.313+0000 7fbff6400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00cad40 0x7fbff0155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:30.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.313+0000 7fbff6400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbff015a1a0 0x7fbff0140160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:30.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.313+0000 7fbff6400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbff0070ae0 con 0x7fbff00ca960 2024-03-18T16:31:30.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.313+0000 7fbff6400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbff015dc10 con 0x7fbff00cad40 2024-03-18T16:31:30.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.313+0000 7fbff6400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbff0070c60 con 0x7fbff015a1a0 2024-03-18T16:31:30.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.313+0000 7fbfefe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00cad40 0x7fbff0155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:30.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.314+0000 7fbff5a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbff015a1a0 0x7fbff0140160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:30.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.314+0000 7fbfefe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00cad40 0x7fbff0155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:50288/0 (socket says 172.21.15.102:50288) 2024-03-18T16:31:30.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.314+0000 7fbfefe00640 1 -- 172.21.15.102:0/1843952104 learned_addr learned my addr 172.21.15.102:0/1843952104 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:30.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.314+0000 7fbff5000640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbff00ca960 0x7fbff0158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:30.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.317+0000 7fbfefe00640 1 -- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbff015a1a0 msgr2=0x7fbff0140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:30.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.317+0000 7fbfefe00640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbff015a1a0 0x7fbff0140160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:30.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.317+0000 7fbfefe00640 1 -- 172.21.15.102:0/1843952104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbff00ca960 msgr2=0x7fbff0158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:30.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.317+0000 7fbfefe00640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbff00ca960 0x7fbff0158d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:30.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.318+0000 7fbfefe00640 1 -- 172.21.15.102:0/1843952104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff00cd120 con 0x7fbff00cad40 2024-03-18T16:31:30.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.318+0000 7fbff5a00640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbff015a1a0 0x7fbff0140160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:31:30.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.320+0000 7fbfefe00640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00cad40 0x7fbff0155c90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00e980 tx=0x7fbfdc00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:30.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.325+0000 7fbfecc00640 1 -- 172.21.15.102:0/1843952104 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc017070 con 0x7fbff00cad40 2024-03-18T16:31:30.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.325+0000 7fbfecc00640 1 -- 172.21.15.102:0/1843952104 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfdc00ca00 con 0x7fbff00cad40 2024-03-18T16:31:30.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.325+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff01efbe0 con 0x7fbff00cad40 2024-03-18T16:31:30.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.325+0000 7fbfecc00640 1 -- 172.21.15.102:0/1843952104 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc00cdd0 con 0x7fbff00cad40 2024-03-18T16:31:30.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.325+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff01f01a0 con 0x7fbff00cad40 2024-03-18T16:31:30.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.327+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbff0074fd0 con 0x7fbff00cad40 2024-03-18T16:31:30.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.352+0000 7fbfecc00640 1 -- 172.21.15.102:0/1843952104 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbfdc01a070 con 0x7fbff00cad40 2024-03-18T16:31:30.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.354+0000 7fbfecc00640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbfd006bc20 0x7fbfd006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:30.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.354+0000 7fbff5000640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbfd006bc20 0x7fbfd006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:30.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.355+0000 7fbff5000640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbfd006bc20 0x7fbfd006e0e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4004f60 tx=0x7fbfe4007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:30.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.355+0000 7fbfecc00640 1 -- 172.21.15.102:0/1843952104 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fbfdc057700 con 0x7fbff00cad40 2024-03-18T16:31:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.628+0000 7fbfecc00640 1 -- 172.21.15.102:0/1843952104 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbfdc059660 con 0x7fbff00cad40 2024-03-18T16:31:30.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.885+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbff0067710 con 0x7fbff00cad40 2024-03-18T16:31:30.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.901+0000 7fbfecc00640 1 -- 172.21.15.102:0/1843952104 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fbfdc05d700 con 0x7fbff00cad40 2024-03-18T16:31:30.902 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:30.902 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:30.903 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbfd006bc20 msgr2=0x7fbfd006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbfd006bc20 0x7fbfd006e0e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4004f60 tx=0x7fbfe4007e30 comp rx=0 tx=0).stop 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00cad40 msgr2=0x7fbff0155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00cad40 0x7fbff0155c90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00e980 tx=0x7fbfdc00ee40 comp rx=0 tx=0).stop 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 shutdown_connections 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbfd006bc20 0x7fbfd006e0e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbff015a1a0 0x7fbff0140160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00cad40 0x7fbff0155c90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 --2- 172.21.15.102:0/1843952104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbff00ca960 0x7fbff0158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 >> 172.21.15.102:0/1843952104 conn(0x7fbff0067190 msgr2=0x7fbff015b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:30.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 shutdown_connections 2024-03-18T16:31:30.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:30.903+0000 7fbff6400640 1 -- 172.21.15.102:0/1843952104 wait complete. 2024-03-18T16:31:31.526 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:31.526 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:32.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:32.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.701+0000 7f2b75400640 1 Processor -- start 2024-03-18T16:31:32.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.701+0000 7f2b75400640 1 -- start start 2024-03-18T16:31:32.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.702+0000 7f2b75400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b7015a310 0x7f2b7006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:32.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.702+0000 7f2b75400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a6f0 0x7f2b7015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:32.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.702+0000 7f2b75400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b700ca500 con 0x7f2b7015baa0 2024-03-18T16:31:32.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.702+0000 7f2b75400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2b700cd410 con 0x7f2b7015a310 2024-03-18T16:31:32.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.702+0000 7f2b75400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b7015b050 con 0x7f2b7015a6f0 2024-03-18T16:31:32.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.703+0000 7f2b6fe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2b7015a310 0x7f2b7006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:32.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.703+0000 7f2b6f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a6f0 0x7f2b7015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:32.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.703+0000 7f2b74a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2b7015baa0 0x7f2b7015bf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43354/0 (socket says 172.21.15.102:43354) 2024-03-18T16:31:32.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.703+0000 7f2b74a00640 1 -- 172.21.15.102:0/1918365226 learned_addr learned my addr 172.21.15.102:0/1918365226 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:32.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.706+0000 7f2b6f400640 1 -- 172.21.15.102:0/1918365226 >> v2:172.21.15.154:3300/0 conn(0x7f2b7015a310 msgr2=0x7f2b7006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:32.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.706+0000 7f2b6f400640 1 --2- 172.21.15.102:0/1918365226 >> v2:172.21.15.154:3300/0 conn(0x7f2b7015a310 0x7f2b7006d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:32.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.706+0000 7f2b6f400640 1 -- 172.21.15.102:0/1918365226 >> v1:172.21.15.102:6789/0 conn(0x7f2b7015baa0 legacy=0x7f2b7015bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:32.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.706+0000 7f2b6f400640 1 -- 172.21.15.102:0/1918365226 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b700ca7a0 con 0x7f2b7015a6f0 2024-03-18T16:31:32.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.708+0000 7f2b6f400640 1 --2- 172.21.15.102:0/1918365226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a6f0 0x7f2b7015ab10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0097a0 tx=0x7f2b5c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7302b19f9fe82cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:32.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.712+0000 7f2b6ea00640 1 -- 172.21.15.102:0/1918365226 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c035020 con 0x7f2b7015a6f0 2024-03-18T16:31:32.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.712+0000 7f2b6ea00640 1 -- 172.21.15.102:0/1918365226 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b5c031810 con 0x7f2b7015a6f0 2024-03-18T16:31:32.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.712+0000 7f2b6ea00640 1 -- 172.21.15.102:0/1918365226 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c031be0 con 0x7f2b7015a6f0 2024-03-18T16:31:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.713+0000 7f2b75400640 1 -- 172.21.15.102:0/1918365226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a6f0 msgr2=0x7f2b7015ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.713+0000 7f2b75400640 1 --2- 172.21.15.102:0/1918365226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a6f0 0x7f2b7015ab10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0097a0 tx=0x7f2b5c02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:31:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.713+0000 7f2b75400640 1 -- 172.21.15.102:0/1918365226 shutdown_connections 2024-03-18T16:31:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.713+0000 7f2b75400640 1 --2- 172.21.15.102:0/1918365226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a6f0 0x7f2b7015ab10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.713+0000 7f2b75400640 1 --2- 172.21.15.102:0/1918365226 >> v2:172.21.15.154:3300/0 conn(0x7f2b7015a310 0x7f2b7006d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.713+0000 7f2b75400640 1 -- 172.21.15.102:0/1918365226 >> 172.21.15.102:0/1918365226 conn(0x7f2b7008c1a0 msgr2=0x7f2b70066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.713+0000 7f2b75400640 1 -- 172.21.15.102:0/1918365226 shutdown_connections 2024-03-18T16:31:32.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.713+0000 7f2b75400640 1 -- 172.21.15.102:0/1918365226 wait complete. 2024-03-18T16:31:32.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.715+0000 7f2b75400640 1 Processor -- start 2024-03-18T16:31:32.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.715+0000 7f2b75400640 1 -- start start 2024-03-18T16:31:32.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.716+0000 7f2b75400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a310 0x7f2b700ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:32.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.716+0000 7f2b75400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b7015a6f0 0x7f2b700c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:32.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.716+0000 7f2b75400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b7015baa0 0x7f2b7014d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:32.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.716+0000 7f2b75400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700cd130 con 0x7f2b7015baa0 2024-03-18T16:31:32.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.716+0000 7f2b75400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700cce50 con 0x7f2b7015a6f0 2024-03-18T16:31:32.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.716+0000 7f2b75400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2b7006d920 con 0x7f2b7015a310 2024-03-18T16:31:32.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.716+0000 7f2b6fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a310 0x7f2b700ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:32.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.716+0000 7f2b6f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b7015a6f0 0x7f2b700c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:32.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.717+0000 7f2b74a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b7015baa0 0x7f2b7014d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:32.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.717+0000 7f2b6fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a310 0x7f2b700ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46394/0 (socket says 172.21.15.102:46394) 2024-03-18T16:31:32.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.717+0000 7f2b6fe00640 1 -- 172.21.15.102:0/2364386429 learned_addr learned my addr 172.21.15.102:0/2364386429 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.720+0000 7f2b6fe00640 1 -- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b7015a6f0 msgr2=0x7f2b700c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.720+0000 7f2b6fe00640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b7015a6f0 0x7f2b700c8a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.720+0000 7f2b6fe00640 1 -- 172.21.15.102:0/2364386429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b7015baa0 msgr2=0x7f2b7014d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.720+0000 7f2b6fe00640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b7015baa0 0x7f2b7014d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:32.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.720+0000 7f2b6fe00640 1 -- 172.21.15.102:0/2364386429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b700c95a0 con 0x7f2b7015a310 2024-03-18T16:31:32.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.723+0000 7f2b6fe00640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a310 0x7f2b700ceb70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000e9b0 tx=0x7f2b6000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:32.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.727+0000 7f2b67e00640 1 -- 172.21.15.102:0/2364386429 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60017070 con 0x7f2b7015a310 2024-03-18T16:31:32.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.727+0000 7f2b67e00640 1 -- 172.21.15.102:0/2364386429 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6000c9a0 con 0x7f2b7015a310 2024-03-18T16:31:32.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.727+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b70150150 con 0x7f2b7015a310 2024-03-18T16:31:32.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.727+0000 7f2b67e00640 1 -- 172.21.15.102:0/2364386429 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6000cd90 con 0x7f2b7015a310 2024-03-18T16:31:32.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.727+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b701f4400 con 0x7f2b7015a310 2024-03-18T16:31:32.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.729+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b70074fd0 con 0x7f2b7015a310 2024-03-18T16:31:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.748+0000 7f2b67e00640 1 -- 172.21.15.102:0/2364386429 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2b6001a070 con 0x7f2b7015a310 2024-03-18T16:31:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.749+0000 7f2b67e00640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2b5006bbd0 0x7f2b5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.750+0000 7f2b6f400640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2b5006bbd0 0x7f2b5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:32.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.750+0000 7f2b6f400640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2b5006bbd0 0x7f2b5006e090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c036000 tx=0x7f2b5c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:32.751+0000 7f2b67e00640 1 -- 172.21.15.102:0/2364386429 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f2b60057750 con 0x7f2b7015a310 2024-03-18T16:31:33.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.003+0000 7f2b67e00640 1 -- 172.21.15.102:0/2364386429 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2b600596b0 con 0x7f2b7015a310 2024-03-18T16:31:33.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.318+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b7006df20 con 0x7f2b7015a310 2024-03-18T16:31:33.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.364+0000 7f2b67e00640 1 -- 172.21.15.102:0/2364386429 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f2b6005d750 con 0x7f2b7015a310 2024-03-18T16:31:33.366 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:33.366 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:33.367 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:33.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.367+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2b5006bbd0 msgr2=0x7f2b5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.367+0000 7f2b75400640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2b5006bbd0 0x7f2b5006e090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c036000 tx=0x7f2b5c008030 comp rx=0 tx=0).stop 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.367+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a310 msgr2=0x7f2b700ceb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.367+0000 7f2b75400640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a310 0x7f2b700ceb70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000e9b0 tx=0x7f2b6000ee70 comp rx=0 tx=0).stop 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.368+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 shutdown_connections 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.368+0000 7f2b75400640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2b5006bbd0 0x7f2b5006e090 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.368+0000 7f2b75400640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b7015baa0 0x7f2b7014d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.368+0000 7f2b75400640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b7015a6f0 0x7f2b700c8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.368+0000 7f2b75400640 1 --2- 172.21.15.102:0/2364386429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2b7015a310 0x7f2b700ceb70 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000e9b0 tx=0x7f2b6000ee70 comp rx=0 tx=0).stop 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.368+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 >> 172.21.15.102:0/2364386429 conn(0x7f2b7008c1a0 msgr2=0x7f2b70153850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.368+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 shutdown_connections 2024-03-18T16:31:33.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:33.368+0000 7f2b75400640 1 -- 172.21.15.102:0/2364386429 wait complete. 2024-03-18T16:31:33.988 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:33.989 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:34.990 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.173+0000 7f8217200640 1 Processor -- start 2024-03-18T16:31:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.173+0000 7f8217200640 1 -- start start 2024-03-18T16:31:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.175+0000 7f8217200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8218158420 0x7f821815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.175+0000 7f8217200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82180c4fa0 0x7f82180c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.175+0000 7f8217200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82180c72d0 con 0x7f8218141650 2024-03-18T16:31:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.175+0000 7f8217200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f821815c8e0 con 0x7f8218158420 2024-03-18T16:31:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.175+0000 7f8217200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8218159370 con 0x7f82180c4fa0 2024-03-18T16:31:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.176+0000 7f8216800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82180c4fa0 0x7f82180c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.176+0000 7f8215400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8218158420 0x7f821815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:35.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.176+0000 7f8215e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8218141650 0x7f82180c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43360/0 (socket says 172.21.15.102:43360) 2024-03-18T16:31:35.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.176+0000 7f8215e00640 1 -- 172.21.15.102:0/1657218606 learned_addr learned my addr 172.21.15.102:0/1657218606 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:35.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.178+0000 7f8216800640 1 -- 172.21.15.102:0/1657218606 >> v2:172.21.15.102:3300/0 conn(0x7f8218158420 msgr2=0x7f821815c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:35.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.178+0000 7f8216800640 1 --2- 172.21.15.102:0/1657218606 >> v2:172.21.15.102:3300/0 conn(0x7f8218158420 0x7f821815c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.178+0000 7f8216800640 1 -- 172.21.15.102:0/1657218606 >> v1:172.21.15.102:6789/0 conn(0x7f8218141650 legacy=0x7f82180c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:35.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.179+0000 7f8216800640 1 -- 172.21.15.102:0/1657218606 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8218158840 con 0x7f82180c4fa0 2024-03-18T16:31:35.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.181+0000 7f8216800640 1 --2- 172.21.15.102:0/1657218606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82180c4fa0 0x7f82180c5430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8210009d90 tx=0x7f821002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f27c98f9da4ff122 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:35.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.185+0000 7f8214a00640 1 -- 172.21.15.102:0/1657218606 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8210039070 con 0x7f82180c4fa0 2024-03-18T16:31:35.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.185+0000 7f8214a00640 1 -- 172.21.15.102:0/1657218606 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821002cc60 con 0x7f82180c4fa0 2024-03-18T16:31:35.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.185+0000 7f8214a00640 1 -- 172.21.15.102:0/1657218606 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8210034660 con 0x7f82180c4fa0 2024-03-18T16:31:35.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.186+0000 7f8217200640 1 -- 172.21.15.102:0/1657218606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82180c4fa0 msgr2=0x7f82180c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:35.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.186+0000 7f8217200640 1 --2- 172.21.15.102:0/1657218606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82180c4fa0 0x7f82180c5430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8210009d90 tx=0x7f821002beb0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.186+0000 7f8217200640 1 -- 172.21.15.102:0/1657218606 shutdown_connections 2024-03-18T16:31:35.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.186+0000 7f8217200640 1 --2- 172.21.15.102:0/1657218606 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f82180c4fa0 0x7f82180c5430 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.186+0000 7f8217200640 1 --2- 172.21.15.102:0/1657218606 >> v2:172.21.15.102:3300/0 conn(0x7f8218158420 0x7f821815c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.186+0000 7f8217200640 1 -- 172.21.15.102:0/1657218606 >> 172.21.15.102:0/1657218606 conn(0x7f82180cb890 msgr2=0x7f82180cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:35.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.186+0000 7f8217200640 1 -- 172.21.15.102:0/1657218606 shutdown_connections 2024-03-18T16:31:35.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.186+0000 7f8217200640 1 -- 172.21.15.102:0/1657218606 wait complete. 2024-03-18T16:31:35.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.188+0000 7f8217200640 1 Processor -- start 2024-03-18T16:31:35.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.188+0000 7f8217200640 1 -- start start 2024-03-18T16:31:35.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.189+0000 7f8217200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82180c4fa0 0x7f821813e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:35.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.189+0000 7f8217200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8218141650 0x7f82181f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:35.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.189+0000 7f8217200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8218158420 0x7f82181f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:35.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.189+0000 7f8217200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8218159090 con 0x7f8218141650 2024-03-18T16:31:35.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.189+0000 7f8217200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8218158db0 con 0x7f82180c4fa0 2024-03-18T16:31:35.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.189+0000 7f8217200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f82180c63c0 con 0x7f8218158420 2024-03-18T16:31:35.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.190+0000 7f8215e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82180c4fa0 0x7f821813e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:35.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.190+0000 7f8215400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8218141650 0x7f82181f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:35.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.190+0000 7f8216800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8218158420 0x7f82181f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:35.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.190+0000 7f8215e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82180c4fa0 0x7f821813e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34470/0 (socket says 172.21.15.102:34470) 2024-03-18T16:31:35.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.190+0000 7f8215e00640 1 -- 172.21.15.102:0/3006807808 learned_addr learned my addr 172.21.15.102:0/3006807808 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:35.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.193+0000 7f8215e00640 1 -- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8218158420 msgr2=0x7f82181f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:35.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.193+0000 7f8215e00640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8218158420 0x7f82181f42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.193+0000 7f8215e00640 1 -- 172.21.15.102:0/3006807808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8218141650 msgr2=0x7f82181f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:35.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.193+0000 7f8215e00640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8218141650 0x7f82181f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.194+0000 7f8215e00640 1 -- 172.21.15.102:0/3006807808 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821813d130 con 0x7f82180c4fa0 2024-03-18T16:31:35.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.196+0000 7f8215e00640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82180c4fa0 0x7f821813e350 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f820800c400 tx=0x7f820800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:35.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.200+0000 7f820e000640 1 -- 172.21.15.102:0/3006807808 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208017070 con 0x7f82180c4fa0 2024-03-18T16:31:35.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.200+0000 7f820e000640 1 -- 172.21.15.102:0/3006807808 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8208005ce0 con 0x7f82180c4fa0 2024-03-18T16:31:35.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.200+0000 7f820e000640 1 -- 172.21.15.102:0/3006807808 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080072f0 con 0x7f82180c4fa0 2024-03-18T16:31:35.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.200+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f821813d3f0 con 0x7f82180c4fa0 2024-03-18T16:31:35.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.200+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82181f3ab0 con 0x7f82180c4fa0 2024-03-18T16:31:35.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.201+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82181569a0 con 0x7f82180c4fa0 2024-03-18T16:31:35.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.221+0000 7f820e000640 1 -- 172.21.15.102:0/3006807808 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f820801a070 con 0x7f82180c4fa0 2024-03-18T16:31:35.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.222+0000 7f820e000640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81f806bc20 0x7f81f806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:35.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.222+0000 7f8215400640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81f806bc20 0x7f81f806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:35.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.223+0000 7f8215400640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81f806bc20 0x7f81f806e0e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8218140410 tx=0x7f8204009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:35.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.467+0000 7f820e000640 1 -- 172.21.15.102:0/3006807808 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8208056760 con 0x7f82180c4fa0 2024-03-18T16:31:35.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.480+0000 7f820e000640 1 -- 172.21.15.102:0/3006807808 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f82181569a0 con 0x7f82180c4fa0 2024-03-18T16:31:35.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.754+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8218074fd0 con 0x7f82180c4fa0 2024-03-18T16:31:35.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.769+0000 7f820e000640 1 -- 172.21.15.102:0/3006807808 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8208021020 con 0x7f82180c4fa0 2024-03-18T16:31:35.771 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:35.771 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:35.771 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:35.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.771+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81f806bc20 msgr2=0x7f81f806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:35.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.771+0000 7f8217200640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81f806bc20 0x7f81f806e0e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8218140410 tx=0x7f8204009190 comp rx=0 tx=0).stop 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.771+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82180c4fa0 msgr2=0x7f821813e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.771+0000 7f8217200640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82180c4fa0 0x7f821813e350 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f820800c400 tx=0x7f820800c8c0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.771+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 shutdown_connections 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.772+0000 7f8217200640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f81f806bc20 0x7f81f806e0e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.772+0000 7f8217200640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8218158420 0x7f82181f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.772+0000 7f8217200640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8218141650 0x7f82181f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.772+0000 7f8217200640 1 --2- 172.21.15.102:0/3006807808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82180c4fa0 0x7f821813e350 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.772+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 >> 172.21.15.102:0/3006807808 conn(0x7f82180cb890 msgr2=0x7f8218156290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.772+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 shutdown_connections 2024-03-18T16:31:35.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:35.773+0000 7f8217200640 1 -- 172.21.15.102:0/3006807808 wait complete. 2024-03-18T16:31:36.396 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:36.396 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:37.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:37.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.570+0000 7f2de5e00640 1 Processor -- start 2024-03-18T16:31:37.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.571+0000 7f2de5e00640 1 -- start start 2024-03-18T16:31:37.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.572+0000 7f2de5e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cfe70 0x7f2de015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:37.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.572+0000 7f2de5e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2de00d0250 0x7f2de00c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.572+0000 7f2de5e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de00cabd0 con 0x7f2de00cc2f0 2024-03-18T16:31:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.572+0000 7f2de5e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2de00caea0 con 0x7f2de00d0250 2024-03-18T16:31:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.572+0000 7f2de5e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2de00cb180 con 0x7f2de00cfe70 2024-03-18T16:31:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.573+0000 7f2de5400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2de00cc2f0 0x7f2de00cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43372/0 (socket says 172.21.15.102:43372) 2024-03-18T16:31:37.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.573+0000 7f2de5400640 1 -- 172.21.15.102:0/2445272977 learned_addr learned my addr 172.21.15.102:0/2445272977 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:37.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.573+0000 7f2de4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cfe70 0x7f2de015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:37.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.573+0000 7f2ddfe00640 1 --2- 172.21.15.102:0/2445272977 >> v2:172.21.15.154:3300/0 conn(0x7f2de00d0250 0x7f2de00c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.576+0000 7f2ddf400640 1 -- 172.21.15.102:0/2445272977 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3200979181 0 0) 0x7f2de00cabd0 con 0x7f2de00cc2f0 2024-03-18T16:31:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.576+0000 7f2ddf400640 1 -- 172.21.15.102:0/2445272977 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dc8003520 con 0x7f2de00cc2f0 2024-03-18T16:31:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.576+0000 7f2ddfe00640 1 -- 172.21.15.102:0/2445272977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cfe70 msgr2=0x7f2de015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.576+0000 7f2ddfe00640 1 --2- 172.21.15.102:0/2445272977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cfe70 0x7f2de015ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:37.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.576+0000 7f2ddfe00640 1 -- 172.21.15.102:0/2445272977 >> v1:172.21.15.102:6789/0 conn(0x7f2de00cc2f0 legacy=0x7f2de00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:37.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.576+0000 7f2ddfe00640 1 -- 172.21.15.102:0/2445272977 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de00cb460 con 0x7f2de00d0250 2024-03-18T16:31:37.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.577+0000 7f2de4a00640 1 --2- 172.21.15.102:0/2445272977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cfe70 0x7f2de015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:31:37.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.579+0000 7f2ddfe00640 1 --2- 172.21.15.102:0/2445272977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 0x7f2de00c65e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0097a0 tx=0x7f2dcc02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2f2ce28f9b25e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:37.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.583+0000 7f2ddf400640 1 -- 172.21.15.102:0/2445272977 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc034070 con 0x7f2de00d0250 2024-03-18T16:31:37.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.583+0000 7f2ddf400640 1 -- 172.21.15.102:0/2445272977 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dcc030800 con 0x7f2de00d0250 2024-03-18T16:31:37.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.584+0000 7f2ddf400640 1 -- 172.21.15.102:0/2445272977 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc030bd0 con 0x7f2de00d0250 2024-03-18T16:31:37.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.584+0000 7f2de5e00640 1 -- 172.21.15.102:0/2445272977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 msgr2=0x7f2de00c65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:37.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.584+0000 7f2de5e00640 1 --2- 172.21.15.102:0/2445272977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 0x7f2de00c65e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc0097a0 tx=0x7f2dcc02b940 comp rx=0 tx=0).stop 2024-03-18T16:31:37.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.585+0000 7f2de5e00640 1 -- 172.21.15.102:0/2445272977 shutdown_connections 2024-03-18T16:31:37.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.585+0000 7f2de5e00640 1 --2- 172.21.15.102:0/2445272977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 0x7f2de00c65e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:37.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.585+0000 7f2de5e00640 1 --2- 172.21.15.102:0/2445272977 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cfe70 0x7f2de015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:37.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.585+0000 7f2de5e00640 1 -- 172.21.15.102:0/2445272977 >> 172.21.15.102:0/2445272977 conn(0x7f2de008c3e0 msgr2=0x7f2de006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:37.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.585+0000 7f2de5e00640 1 -- 172.21.15.102:0/2445272977 shutdown_connections 2024-03-18T16:31:37.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.585+0000 7f2de5e00640 1 -- 172.21.15.102:0/2445272977 wait complete. 2024-03-18T16:31:37.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.587+0000 7f2de5e00640 1 Processor -- start 2024-03-18T16:31:37.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.588+0000 7f2de5e00640 1 -- start start 2024-03-18T16:31:37.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.589+0000 7f2de5e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cc2f0 0x7f2de0156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:37.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.589+0000 7f2de5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de00cfe70 0x7f2de0156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:37.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.589+0000 7f2de5e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 0x7f2de0159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:37.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.589+0000 7f2de5e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f2de00cbd00 con 0x7f2de00cfe70 2024-03-18T16:31:37.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.589+0000 7f2de5e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2de00cba20 con 0x7f2de00d0250 2024-03-18T16:31:37.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.589+0000 7f2de5e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2de00c6b20 con 0x7f2de00cc2f0 2024-03-18T16:31:37.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.590+0000 7f2de5400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 0x7f2de0159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:37.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.590+0000 7f2ddfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de00cfe70 0x7f2de0156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:37.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.590+0000 7f2de4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cc2f0 0x7f2de0156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:37.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.590+0000 7f2ddfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de00cfe70 0x7f2de0156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60998/0 (socket says 172.21.15.102:60998) 2024-03-18T16:31:37.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.590+0000 7f2ddfe00640 1 -- 172.21.15.102:0/755175133 learned_addr learned my addr 172.21.15.102:0/755175133 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.595+0000 7f2de5400640 1 -- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cc2f0 msgr2=0x7f2de0156840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.595+0000 7f2de5400640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cc2f0 0x7f2de0156840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.595+0000 7f2de5400640 1 -- 172.21.15.102:0/755175133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de00cfe70 msgr2=0x7f2de0156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.595+0000 7f2de5400640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de00cfe70 0x7f2de0156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.595+0000 7f2de5400640 1 -- 172.21.15.102:0/755175133 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de0158000 con 0x7f2de00d0250 2024-03-18T16:31:37.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.598+0000 7f2de5400640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 0x7f2de0159660 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800cb00 tx=0x7f2dd800cf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:37.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.603+0000 7f2ddcc00640 1 -- 172.21.15.102:0/755175133 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8017070 con 0x7f2de00d0250 2024-03-18T16:31:37.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.603+0000 7f2ddcc00640 1 -- 172.21.15.102:0/755175133 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd800abd0 con 0x7f2de00d0250 2024-03-18T16:31:37.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.603+0000 7f2ddcc00640 1 -- 172.21.15.102:0/755175133 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8012400 con 0x7f2de00d0250 2024-03-18T16:31:37.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.603+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de00cf0d0 con 0x7f2de00d0250 2024-03-18T16:31:37.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.603+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de01f0370 con 0x7f2de00d0250 2024-03-18T16:31:37.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.604+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de0074fd0 con 0x7f2de00d0250 2024-03-18T16:31:37.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.629+0000 7f2ddcc00640 1 -- 172.21.15.102:0/755175133 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2dd801a070 con 0x7f2de00d0250 2024-03-18T16:31:37.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.630+0000 7f2ddcc00640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc006bc20 0x7f2dc006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:37.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.630+0000 7f2de4a00640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc006bc20 0x7f2dc006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:37.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.631+0000 7f2de4a00640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc006bc20 0x7f2dc006e0e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009560 tx=0x7f2dd4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:37.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.888+0000 7f2ddcc00640 1 -- 172.21.15.102:0/755175133 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f2dd8057080 con 0x7f2de00d0250 2024-03-18T16:31:37.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:37.900+0000 7f2ddcc00640 1 -- 172.21.15.102:0/755175133 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2dd8058fe0 con 0x7f2de00d0250 2024-03-18T16:31:38.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.206+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2de00c8ab0 con 0x7f2de00d0250 2024-03-18T16:31:38.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.219+0000 7f2ddcc00640 1 -- 172.21.15.102:0/755175133 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f2dd805d080 con 0x7f2de00d0250 2024-03-18T16:31:38.221 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:38.221 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:38.221 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:38.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc006bc20 msgr2=0x7f2dc006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:38.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc006bc20 0x7f2dc006e0e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009560 tx=0x7f2dd4009190 comp rx=0 tx=0).stop 2024-03-18T16:31:38.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 msgr2=0x7f2de0159660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:38.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 0x7f2de0159660 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800cb00 tx=0x7f2dd800cf00 comp rx=0 tx=0).stop 2024-03-18T16:31:38.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 shutdown_connections 2024-03-18T16:31:38.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f2dc006bc20 0x7f2dc006e0e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:38.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2de00d0250 0x7f2de0159660 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:38.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2de00cfe70 0x7f2de0156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:38.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 --2- 172.21.15.102:0/755175133 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2de00cc2f0 0x7f2de0156840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:38.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.221+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 >> 172.21.15.102:0/755175133 conn(0x7f2de008c3e0 msgr2=0x7f2de00cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:38.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.222+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 shutdown_connections 2024-03-18T16:31:38.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:38.222+0000 7f2de5e00640 1 -- 172.21.15.102:0/755175133 wait complete. 2024-03-18T16:31:38.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:38.845 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:39.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.019+0000 7f43cf200640 1 Processor -- start 2024-03-18T16:31:40.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.019+0000 7f43cf200640 1 -- start start 2024-03-18T16:31:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.020+0000 7f43cf200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f43c814dad0 0x7f43c8157a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.020+0000 7f43cf200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80733b0 0x7f43c806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.020+0000 7f43cf200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43c80c75b0 con 0x7f43c815b1c0 2024-03-18T16:31:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.021+0000 7f43cf200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f43c814deb0 con 0x7f43c814dad0 2024-03-18T16:31:40.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.021+0000 7f43cf200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f43c815ad30 con 0x7f43c80733b0 2024-03-18T16:31:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.021+0000 7f43cde00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f43c814dad0 0x7f43c8157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.021+0000 7f43ce800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f43c815b1c0 0x7f43c80c6230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44028/0 (socket says 172.21.15.102:44028) 2024-03-18T16:31:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.021+0000 7f43ce800640 1 -- 172.21.15.102:0/2283617359 learned_addr learned my addr 172.21.15.102:0/2283617359 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:40.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.022+0000 7f43cd400640 1 --2- 172.21.15.102:0/2283617359 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80733b0 0x7f43c806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:40.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.024+0000 7f43cca00640 1 -- 172.21.15.102:0/2283617359 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3537881872 0 0) 0x7f43c80c75b0 con 0x7f43c815b1c0 2024-03-18T16:31:40.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.024+0000 7f43cca00640 1 -- 172.21.15.102:0/2283617359 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43b0003520 con 0x7f43c815b1c0 2024-03-18T16:31:40.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.025+0000 7f43cd400640 1 -- 172.21.15.102:0/2283617359 >> v2:172.21.15.102:3300/0 conn(0x7f43c814dad0 msgr2=0x7f43c8157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:40.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.025+0000 7f43cd400640 1 --2- 172.21.15.102:0/2283617359 >> v2:172.21.15.102:3300/0 conn(0x7f43c814dad0 0x7f43c8157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.025+0000 7f43cd400640 1 -- 172.21.15.102:0/2283617359 >> v1:172.21.15.154:6789/0 conn(0x7f43c815b1c0 legacy=0x7f43c80c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:40.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.025+0000 7f43cd400640 1 -- 172.21.15.102:0/2283617359 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c80c7850 con 0x7f43c80733b0 2024-03-18T16:31:40.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.028+0000 7f43cd400640 1 --2- 172.21.15.102:0/2283617359 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80733b0 0x7f43c806de50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f43b80097a0 tx=0x7f43b802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2e3f561fc796cba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.032+0000 7f43cca00640 1 -- 172.21.15.102:0/2283617359 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8035020 con 0x7f43c80733b0 2024-03-18T16:31:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.033+0000 7f43cca00640 1 -- 172.21.15.102:0/2283617359 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b8031810 con 0x7f43c80733b0 2024-03-18T16:31:40.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.033+0000 7f43cca00640 1 -- 172.21.15.102:0/2283617359 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8031be0 con 0x7f43c80733b0 2024-03-18T16:31:40.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.033+0000 7f43cf200640 1 -- 172.21.15.102:0/2283617359 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80733b0 msgr2=0x7f43c806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:40.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.033+0000 7f43cf200640 1 --2- 172.21.15.102:0/2283617359 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80733b0 0x7f43c806de50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f43b80097a0 tx=0x7f43b802b8b0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.033+0000 7f43cf200640 1 -- 172.21.15.102:0/2283617359 shutdown_connections 2024-03-18T16:31:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.033+0000 7f43cf200640 1 --2- 172.21.15.102:0/2283617359 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c80733b0 0x7f43c806de50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.033+0000 7f43cf200640 1 --2- 172.21.15.102:0/2283617359 >> v2:172.21.15.102:3300/0 conn(0x7f43c814dad0 0x7f43c8157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.033+0000 7f43cf200640 1 -- 172.21.15.102:0/2283617359 >> 172.21.15.102:0/2283617359 conn(0x7f43c80727e0 msgr2=0x7f43c8072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.034+0000 7f43cf200640 1 -- 172.21.15.102:0/2283617359 shutdown_connections 2024-03-18T16:31:40.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.034+0000 7f43cf200640 1 -- 172.21.15.102:0/2283617359 wait complete. 2024-03-18T16:31:40.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.035+0000 7f43cf200640 1 Processor -- start 2024-03-18T16:31:40.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.035+0000 7f43cf200640 1 -- start start 2024-03-18T16:31:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.036+0000 7f43cf200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43c80733b0 0x7f43c813d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.036+0000 7f43cf200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43c814dad0 0x7f43c81f3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.036+0000 7f43cf200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c815b1c0 0x7f43c80c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.036+0000 7f43cf200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f43c815a6e0 con 0x7f43c80733b0 2024-03-18T16:31:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.036+0000 7f43cf200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f43c806e390 con 0x7f43c814dad0 2024-03-18T16:31:40.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.036+0000 7f43cf200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f43c815a9c0 con 0x7f43c815b1c0 2024-03-18T16:31:40.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.037+0000 7f43ce800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c815b1c0 0x7f43c80c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:40.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.037+0000 7f43cde00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43c80733b0 0x7f43c813d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:40.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.038+0000 7f43ce800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c815b1c0 0x7f43c80c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46434/0 (socket says 172.21.15.102:46434) 2024-03-18T16:31:40.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.038+0000 7f43ce800640 1 -- 172.21.15.102:0/108555085 learned_addr learned my addr 172.21.15.102:0/108555085 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:40.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.038+0000 7f43cde00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43c80733b0 0x7f43c813d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:32774/0 (socket says 172.21.15.102:32774) 2024-03-18T16:31:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.042+0000 7f43cde00640 1 -- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c815b1c0 msgr2=0x7f43c80c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.042+0000 7f43cde00640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c815b1c0 0x7f43c80c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.042+0000 7f43cde00640 1 -- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43c814dad0 msgr2=0x7f43c81f3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:40.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.042+0000 7f43cde00640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43c814dad0 0x7f43c81f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.042+0000 7f43cde00640 1 -- 172.21.15.102:0/108555085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c81f2f50 con 0x7f43c80733b0 2024-03-18T16:31:40.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.044+0000 7f43cde00640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43c80733b0 0x7f43c813d4f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00e850 tx=0x7f43bc00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:40.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.049+0000 7f43c2000640 1 -- 172.21.15.102:0/108555085 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43bc017070 con 0x7f43c80733b0 2024-03-18T16:31:40.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.049+0000 7f43c2000640 1 -- 172.21.15.102:0/108555085 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43bc00c8e0 con 0x7f43c80733b0 2024-03-18T16:31:40.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.049+0000 7f43c2000640 1 -- 172.21.15.102:0/108555085 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43bc00ccd0 con 0x7f43c80733b0 2024-03-18T16:31:40.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.049+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c81646d0 con 0x7f43c80733b0 2024-03-18T16:31:40.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.049+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c814d030 con 0x7f43c80733b0 2024-03-18T16:31:40.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.050+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c8074fd0 con 0x7f43c80733b0 2024-03-18T16:31:40.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.086+0000 7f43c2000640 1 -- 172.21.15.102:0/108555085 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f43bc01a070 con 0x7f43c80733b0 2024-03-18T16:31:40.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.087+0000 7f43c2000640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43ac06bc20 0x7f43ac06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:40.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.087+0000 7f43cd400640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43ac06bc20 0x7f43ac06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:40.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.088+0000 7f43cd400640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43ac06bc20 0x7f43ac06e0e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f43b8036000 tx=0x7f43b8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.384+0000 7f43c2000640 1 -- 172.21.15.102:0/108555085 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f43bc0575d0 con 0x7f43c80733b0 2024-03-18T16:31:40.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.385+0000 7f43c2000640 1 -- 172.21.15.102:0/108555085 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f43bc059530 con 0x7f43c80733b0 2024-03-18T16:31:40.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.648+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43c8157eb0 con 0x7f43c80733b0 2024-03-18T16:31:40.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.664+0000 7f43c2000640 1 -- 172.21.15.102:0/108555085 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f43bc05d5d0 con 0x7f43c80733b0 2024-03-18T16:31:40.667 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:40.667 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:40.667 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:40.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43ac06bc20 msgr2=0x7f43ac06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:40.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43ac06bc20 0x7f43ac06e0e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f43b8036000 tx=0x7f43b8030000 comp rx=0 tx=0).stop 2024-03-18T16:31:40.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43c80733b0 msgr2=0x7f43c813d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:40.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43c80733b0 0x7f43c813d4f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00e850 tx=0x7f43bc00ed10 comp rx=0 tx=0).stop 2024-03-18T16:31:40.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 shutdown_connections 2024-03-18T16:31:40.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43ac06bc20 0x7f43ac06e0e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43c815b1c0 0x7f43c80c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43c814dad0 0x7f43c81f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 --2- 172.21.15.102:0/108555085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43c80733b0 0x7f43c813d4f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:40.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 >> 172.21.15.102:0/108555085 conn(0x7f43c80727e0 msgr2=0x7f43c814bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:40.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 shutdown_connections 2024-03-18T16:31:40.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:40.668+0000 7f43cf200640 1 -- 172.21.15.102:0/108555085 wait complete. 2024-03-18T16:31:41.298 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:41.298 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:42.298 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:42.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.475+0000 7f6ab0400640 1 Processor -- start 2024-03-18T16:31:42.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.476+0000 7f6ab0400640 1 -- start start 2024-03-18T16:31:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.477+0000 7f6ab0400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 0x7f6aa815a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.477+0000 7f6ab0400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6aa80cc1f0 0x7f6aa80caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.477+0000 7f6ab0400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa815daa0 con 0x7f6aa80c5050 2024-03-18T16:31:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.477+0000 7f6ab0400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6aa815e970 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.477+0000 7f6ab0400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa80cc600 con 0x7f6aa8154780 2024-03-18T16:31:42.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.478+0000 7f6aada00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 0x7f6aa815a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:42.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.479+0000 7f6aada00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 0x7f6aa815a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52236/0 (socket says 172.21.15.102:52236) 2024-03-18T16:31:42.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.479+0000 7f6aada00640 1 -- 172.21.15.102:0/1589303528 learned_addr learned my addr 172.21.15.102:0/1589303528 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:42.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.480+0000 7f6aae400640 1 --2- 172.21.15.102:0/1589303528 >> v2:172.21.15.102:3300/0 conn(0x7f6aa80cc1f0 0x7f6aa80caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.483+0000 7f6aa7e00640 1 -- 172.21.15.102:0/1589303528 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365340576 0 0) 0x7f6aa815daa0 con 0x7f6aa80c5050 2024-03-18T16:31:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.483+0000 7f6aa7e00640 1 -- 172.21.15.102:0/1589303528 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a90003520 con 0x7f6aa80c5050 2024-03-18T16:31:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.483+0000 7f6aae400640 1 -- 172.21.15.102:0/1589303528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 msgr2=0x7f6aa815a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.483+0000 7f6aae400640 1 --2- 172.21.15.102:0/1589303528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 0x7f6aa815a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.483+0000 7f6aae400640 1 -- 172.21.15.102:0/1589303528 >> v1:172.21.15.154:6789/0 conn(0x7f6aa80c5050 legacy=0x7f6aa815e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:42.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.483+0000 7f6aae400640 1 -- 172.21.15.102:0/1589303528 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa815aad0 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.487+0000 7f6aae400640 1 --2- 172.21.15.102:0/1589303528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80cc1f0 0x7f6aa80caca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009d90 tx=0x7f6aa002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5656fc53f8454b71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:42.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.493+0000 7f6aa7e00640 1 -- 172.21.15.102:0/1589303528 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0039070 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.493+0000 7f6aa7e00640 1 -- 172.21.15.102:0/1589303528 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa002cce0 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.493+0000 7f6aa7e00640 1 -- 172.21.15.102:0/1589303528 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0034660 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.494+0000 7f6ab0400640 1 -- 172.21.15.102:0/1589303528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80cc1f0 msgr2=0x7f6aa80caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:42.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.494+0000 7f6ab0400640 1 --2- 172.21.15.102:0/1589303528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80cc1f0 0x7f6aa80caca0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009d90 tx=0x7f6aa002bf30 comp rx=0 tx=0).stop 2024-03-18T16:31:42.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.494+0000 7f6ab0400640 1 -- 172.21.15.102:0/1589303528 shutdown_connections 2024-03-18T16:31:42.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.494+0000 7f6ab0400640 1 --2- 172.21.15.102:0/1589303528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80cc1f0 0x7f6aa80caca0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:42.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.494+0000 7f6ab0400640 1 --2- 172.21.15.102:0/1589303528 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 0x7f6aa815a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:42.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.494+0000 7f6ab0400640 1 -- 172.21.15.102:0/1589303528 >> 172.21.15.102:0/1589303528 conn(0x7f6aa8067160 msgr2=0x7f6aa8067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:42.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.495+0000 7f6ab0400640 1 -- 172.21.15.102:0/1589303528 shutdown_connections 2024-03-18T16:31:42.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.495+0000 7f6ab0400640 1 -- 172.21.15.102:0/1589303528 wait complete. 2024-03-18T16:31:42.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.497+0000 7f6ab0400640 1 Processor -- start 2024-03-18T16:31:42.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.497+0000 7f6ab0400640 1 -- start start 2024-03-18T16:31:42.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.499+0000 7f6ab0400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80c5050 0x7f6aa80c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:42.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.499+0000 7f6ab0400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa80cc1f0 0x7f6aa814a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:42.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.499+0000 7f6ab0400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 0x7f6aa81efbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:42.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.499+0000 7f6ab0400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa815dd40 con 0x7f6aa80c5050 2024-03-18T16:31:42.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.499+0000 7f6ab0400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa815b090 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.499+0000 7f6ab0400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa815b370 con 0x7f6aa8154780 2024-03-18T16:31:42.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.500+0000 7f6aada00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80c5050 0x7f6aa80c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:42.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.500+0000 7f6aad000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa80cc1f0 0x7f6aa814a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:42.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.500+0000 7f6aae400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 0x7f6aa81efbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:42.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.500+0000 7f6aada00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80c5050 0x7f6aa80c95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50090/0 (socket says 172.21.15.102:50090) 2024-03-18T16:31:42.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.500+0000 7f6aada00640 1 -- 172.21.15.102:0/1110837083 learned_addr learned my addr 172.21.15.102:0/1110837083 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:42.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.503+0000 7f6aad000640 1 -- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 msgr2=0x7f6aa81efbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:42.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.503+0000 7f6aad000640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 0x7f6aa81efbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:42.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.503+0000 7f6aad000640 1 -- 172.21.15.102:0/1110837083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80c5050 msgr2=0x7f6aa80c95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:42.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.503+0000 7f6aad000640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80c5050 0x7f6aa80c95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:42.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.504+0000 7f6aad000640 1 -- 172.21.15.102:0/1110837083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa80c67f0 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.506+0000 7f6aad000640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa80cc1f0 0x7f6aa814a140 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6aa815dc70 tx=0x7f6a9400c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:42.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.511+0000 7f6aa5600640 1 -- 172.21.15.102:0/1110837083 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94017070 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.511+0000 7f6aa5600640 1 -- 172.21.15.102:0/1110837083 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a940052d0 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.511+0000 7f6aa5600640 1 -- 172.21.15.102:0/1110837083 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94003830 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.511+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa8150ba0 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.511+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa81f35c0 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.512+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa8074fd0 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.535+0000 7f6aa5600640 1 -- 172.21.15.102:0/1110837083 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6a9401a070 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.536+0000 7f6aa5600640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:42.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.537+0000 7f6aada00640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:42.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.538+0000 7f6aada00640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c006f40 tx=0x7f6a9c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:42.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.776+0000 7f6aa5600640 1 -- 172.21.15.102:0/1110837083 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6a94056770 con 0x7f6aa80cc1f0 2024-03-18T16:31:42.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:42.788+0000 7f6aa5600640 1 -- 172.21.15.102:0/1110837083 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6a94021020 con 0x7f6aa80cc1f0 2024-03-18T16:31:43.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.082+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6aa80cb850 con 0x7f6aa80cc1f0 2024-03-18T16:31:43.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.210+0000 7f6aa5600640 1 -- 172.21.15.102:0/1110837083 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f6a940586d0 con 0x7f6aa80cc1f0 2024-03-18T16:31:43.212 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:43.212 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:43.212 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:43.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 msgr2=0x7f6a8806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c006f40 tx=0x7f6a9c009240 comp rx=0 tx=0).stop 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa80cc1f0 msgr2=0x7f6aa814a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa80cc1f0 0x7f6aa814a140 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f6aa815dc70 tx=0x7f6a9400c890 comp rx=0 tx=0).stop 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 shutdown_connections 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa8154780 0x7f6aa81efbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa80cc1f0 0x7f6aa814a140 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 --2- 172.21.15.102:0/1110837083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa80c5050 0x7f6aa80c95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.213+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 >> 172.21.15.102:0/1110837083 conn(0x7f6aa8067160 msgr2=0x7f6aa80b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.214+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 shutdown_connections 2024-03-18T16:31:43.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:43.214+0000 7f6ab0400640 1 -- 172.21.15.102:0/1110837083 wait complete. 2024-03-18T16:31:43.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:43.847 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:44.848 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.020+0000 7fa48fe00640 1 Processor -- start 2024-03-18T16:31:45.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.020+0000 7fa48fe00640 1 -- start start 2024-03-18T16:31:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.022+0000 7fa48fe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa490141650 0x7fa490158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.022+0000 7fa48fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4900c72d0 con 0x7fa49015c3a0 2024-03-18T16:31:45.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.022+0000 7fa48fe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa490158ad0 con 0x7fa4900c4fa0 2024-03-18T16:31:45.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.022+0000 7fa48fe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa490159370 con 0x7fa490141650 2024-03-18T16:31:45.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.022+0000 7fa48ea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa490141650 0x7fa490158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:45.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.023+0000 7fa48e000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa49015c3a0 0x7fa4900c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53116/0 (socket says 172.21.15.102:53116) 2024-03-18T16:31:45.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.023+0000 7fa48e000640 1 -- 172.21.15.102:0/2236893570 learned_addr learned my addr 172.21.15.102:0/2236893570 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.026+0000 7fa48ea00640 1 -- 172.21.15.102:0/2236893570 >> v1:172.21.15.154:6789/0 conn(0x7fa4900c4fa0 legacy=0x7fa490162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.026+0000 7fa48ea00640 1 -- 172.21.15.102:0/2236893570 >> v1:172.21.15.102:6789/0 conn(0x7fa49015c3a0 legacy=0x7fa4900c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:45.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.026+0000 7fa48ea00640 1 -- 172.21.15.102:0/2236893570 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa490158db0 con 0x7fa490141650 2024-03-18T16:31:45.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.029+0000 7fa48ea00640 1 --2- 172.21.15.102:0/2236893570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa490141650 0x7fa490158420 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa4800097a0 tx=0x7fa48002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83bb811c0746f8c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.033+0000 7fa48d600640 1 -- 172.21.15.102:0/2236893570 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480035020 con 0x7fa490141650 2024-03-18T16:31:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.033+0000 7fa48d600640 1 -- 172.21.15.102:0/2236893570 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa480031830 con 0x7fa490141650 2024-03-18T16:31:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.033+0000 7fa48d600640 1 -- 172.21.15.102:0/2236893570 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480031c00 con 0x7fa490141650 2024-03-18T16:31:45.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.034+0000 7fa48fe00640 1 -- 172.21.15.102:0/2236893570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa490141650 msgr2=0x7fa490158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:45.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.034+0000 7fa48fe00640 1 --2- 172.21.15.102:0/2236893570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa490141650 0x7fa490158420 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa4800097a0 tx=0x7fa48002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:31:45.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.034+0000 7fa48fe00640 1 -- 172.21.15.102:0/2236893570 shutdown_connections 2024-03-18T16:31:45.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.034+0000 7fa48fe00640 1 --2- 172.21.15.102:0/2236893570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa490141650 0x7fa490158420 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:45.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.034+0000 7fa48fe00640 1 -- 172.21.15.102:0/2236893570 >> 172.21.15.102:0/2236893570 conn(0x7fa4900cb890 msgr2=0x7fa4900cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:45.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.034+0000 7fa48fe00640 1 -- 172.21.15.102:0/2236893570 shutdown_connections 2024-03-18T16:31:45.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.034+0000 7fa48fe00640 1 -- 172.21.15.102:0/2236893570 wait complete. 2024-03-18T16:31:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.036+0000 7fa48fe00640 1 Processor -- start 2024-03-18T16:31:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.036+0000 7fa48fe00640 1 -- start start 2024-03-18T16:31:45.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.037+0000 7fa48fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900c4fa0 0x7fa490155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.037+0000 7fa48fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa490141650 0x7fa49013df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.037+0000 7fa48fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa49015c3a0 0x7fa49013e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.037+0000 7fa48fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900c66a0 con 0x7fa49015c3a0 2024-03-18T16:31:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.037+0000 7fa48fe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900c63c0 con 0x7fa4900c4fa0 2024-03-18T16:31:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.037+0000 7fa48fe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa49015c7c0 con 0x7fa490141650 2024-03-18T16:31:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.038+0000 7fa48ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900c4fa0 0x7fa490155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.038+0000 7fa48f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa49015c3a0 0x7fa49013e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.038+0000 7fa48e000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa490141650 0x7fa49013df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.038+0000 7fa48ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900c4fa0 0x7fa490155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48638/0 (socket says 172.21.15.102:48638) 2024-03-18T16:31:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.038+0000 7fa48ea00640 1 -- 172.21.15.102:0/209108217 learned_addr learned my addr 172.21.15.102:0/209108217 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:45.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.042+0000 7fa48ea00640 1 -- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa490141650 msgr2=0x7fa49013df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:45.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.042+0000 7fa48ea00640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa490141650 0x7fa49013df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:45.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.042+0000 7fa48ea00640 1 -- 172.21.15.102:0/209108217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa49015c3a0 msgr2=0x7fa49013e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:45.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.042+0000 7fa48ea00640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa49015c3a0 0x7fa49013e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:45.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.042+0000 7fa48ea00640 1 -- 172.21.15.102:0/209108217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4901ee8e0 con 0x7fa4900c4fa0 2024-03-18T16:31:45.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.045+0000 7fa48ea00640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900c4fa0 0x7fa490155830 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa4800098c0 tx=0x7fa4800046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:45.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.048+0000 7fa487400640 1 -- 172.21.15.102:0/209108217 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480041070 con 0x7fa4900c4fa0 2024-03-18T16:31:45.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.049+0000 7fa487400640 1 -- 172.21.15.102:0/209108217 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48003ab20 con 0x7fa4900c4fa0 2024-03-18T16:31:45.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.049+0000 7fa487400640 1 -- 172.21.15.102:0/209108217 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480035040 con 0x7fa4900c4fa0 2024-03-18T16:31:45.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.049+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4901eebc0 con 0x7fa4900c4fa0 2024-03-18T16:31:45.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.049+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4901ef180 con 0x7fa4900c4fa0 2024-03-18T16:31:45.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.050+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4900c7af0 con 0x7fa4900c4fa0 2024-03-18T16:31:45.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.069+0000 7fa487400640 1 -- 172.21.15.102:0/209108217 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa480038020 con 0x7fa4900c4fa0 2024-03-18T16:31:45.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.070+0000 7fa487400640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa47006bc00 0x7fa47006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:45.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.070+0000 7fa48e000640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa47006bc00 0x7fa47006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:45.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.071+0000 7fa48e000640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa47006bc00 0x7fa47006e0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa4900ca3a0 tx=0x7fa47c007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:45.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.072+0000 7fa487400640 1 -- 172.21.15.102:0/209108217 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa480078e60 con 0x7fa4900c4fa0 2024-03-18T16:31:45.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.322+0000 7fa487400640 1 -- 172.21.15.102:0/209108217 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa4900c7af0 con 0x7fa4900c4fa0 2024-03-18T16:31:45.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.690+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa490074fd0 con 0x7fa4900c4fa0 2024-03-18T16:31:45.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.704+0000 7fa487400640 1 -- 172.21.15.102:0/209108217 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa48007adc0 con 0x7fa4900c4fa0 2024-03-18T16:31:45.707 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:45.707 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:45.707 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa47006bc00 msgr2=0x7fa47006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa47006bc00 0x7fa47006e0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa4900ca3a0 tx=0x7fa47c007e80 comp rx=0 tx=0).stop 2024-03-18T16:31:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900c4fa0 msgr2=0x7fa490155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:45.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900c4fa0 0x7fa490155830 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa4800098c0 tx=0x7fa4800046c0 comp rx=0 tx=0).stop 2024-03-18T16:31:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 shutdown_connections 2024-03-18T16:31:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa47006bc00 0x7fa47006e0c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa49015c3a0 0x7fa49013e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa490141650 0x7fa49013df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 --2- 172.21.15.102:0/209108217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4900c4fa0 0x7fa490155830 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 >> 172.21.15.102:0/209108217 conn(0x7fa4900cb890 msgr2=0x7fa4900cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 shutdown_connections 2024-03-18T16:31:45.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:45.707+0000 7fa48fe00640 1 -- 172.21.15.102:0/209108217 wait complete. 2024-03-18T16:31:46.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:46.336 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:47.337 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:47.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.519+0000 7f80e4a00640 1 Processor -- start 2024-03-18T16:31:47.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.519+0000 7f80e4a00640 1 -- start start 2024-03-18T16:31:47.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.521+0000 7f80e4a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f80e00c8740 0x7f80e0072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:47.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.521+0000 7f80e4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:47.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.521+0000 7f80e4a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e0157ac0 con 0x7f80e00c8b20 2024-03-18T16:31:47.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.521+0000 7f80e4a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f80e015a860 con 0x7f80e00c8740 2024-03-18T16:31:47.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.521+0000 7f80e4a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f80e0072840 con 0x7f80e0158760 2024-03-18T16:31:47.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.524+0000 7f80de800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:47.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.524+0000 7f80de800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e015b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52256/0 (socket says 172.21.15.102:52256) 2024-03-18T16:31:47.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.524+0000 7f80de800640 1 -- 172.21.15.102:0/989933014 learned_addr learned my addr 172.21.15.102:0/989933014 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.527+0000 7f80de800640 1 -- 172.21.15.102:0/989933014 >> v2:172.21.15.154:3300/0 conn(0x7f80e00c8740 msgr2=0x7f80e0072300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.527+0000 7f80de800640 1 --2- 172.21.15.102:0/989933014 >> v2:172.21.15.154:3300/0 conn(0x7f80e00c8740 0x7f80e0072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:47.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.527+0000 7f80de800640 1 -- 172.21.15.102:0/989933014 >> v1:172.21.15.154:6789/0 conn(0x7f80e00c8b20 legacy=0x7f80e01572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:47.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.527+0000 7f80de800640 1 -- 172.21.15.102:0/989933014 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e0157da0 con 0x7f80e0158760 2024-03-18T16:31:47.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.596+0000 7f80de800640 1 --2- 172.21.15.102:0/989933014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e015b2b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009e50 tx=0x7f80d802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8118ded00b372817 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:47.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.609+0000 7f80dca00640 1 -- 172.21.15.102:0/989933014 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d802ca70 con 0x7f80e0158760 2024-03-18T16:31:47.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.609+0000 7f80dca00640 1 -- 172.21.15.102:0/989933014 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d802cc10 con 0x7f80e0158760 2024-03-18T16:31:47.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.609+0000 7f80dca00640 1 -- 172.21.15.102:0/989933014 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d80343f0 con 0x7f80e0158760 2024-03-18T16:31:47.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.610+0000 7f80e4a00640 1 -- 172.21.15.102:0/989933014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 msgr2=0x7f80e015b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:47.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.610+0000 7f80e4a00640 1 --2- 172.21.15.102:0/989933014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e015b2b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009e50 tx=0x7f80d802bed0 comp rx=0 tx=0).stop 2024-03-18T16:31:47.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.610+0000 7f80e4a00640 1 -- 172.21.15.102:0/989933014 shutdown_connections 2024-03-18T16:31:47.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.610+0000 7f80e4a00640 1 --2- 172.21.15.102:0/989933014 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e015b2b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:47.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.610+0000 7f80e4a00640 1 --2- 172.21.15.102:0/989933014 >> v2:172.21.15.154:3300/0 conn(0x7f80e00c8740 0x7f80e0072300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:47.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.610+0000 7f80e4a00640 1 -- 172.21.15.102:0/989933014 >> 172.21.15.102:0/989933014 conn(0x7f80e00700d0 msgr2=0x7f80e006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:47.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.610+0000 7f80e4a00640 1 -- 172.21.15.102:0/989933014 shutdown_connections 2024-03-18T16:31:47.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.611+0000 7f80e4a00640 1 -- 172.21.15.102:0/989933014 wait complete. 2024-03-18T16:31:47.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.613+0000 7f80e4a00640 1 Processor -- start 2024-03-18T16:31:47.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.613+0000 7f80e4a00640 1 -- start start 2024-03-18T16:31:47.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.615+0000 7f80e4a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e00c8740 0x7f80e01f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:47.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.615+0000 7f80e4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80e00c8b20 0x7f80e0145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:47.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.615+0000 7f80e4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e00cc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:47.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.615+0000 7f80e4a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f80e015aab0 con 0x7f80e00c8b20 2024-03-18T16:31:47.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.615+0000 7f80e4a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f80e015b7f0 con 0x7f80e00c8740 2024-03-18T16:31:47.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.615+0000 7f80e4a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f80e015ad90 con 0x7f80e0158760 2024-03-18T16:31:47.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.616+0000 7f80de800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e00cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:47.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.616+0000 7f80dde00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e00c8740 0x7f80e01f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:47.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.616+0000 7f80de800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e00cc280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:52258/0 (socket says 172.21.15.102:52258) 2024-03-18T16:31:47.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.616+0000 7f80de800640 1 -- 172.21.15.102:0/942567340 learned_addr learned my addr 172.21.15.102:0/942567340 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:47.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.620+0000 7f80dde00640 1 -- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 msgr2=0x7f80e00cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:47.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.620+0000 7f80dde00640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e00cc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:47.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.620+0000 7f80dde00640 1 -- 172.21.15.102:0/942567340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80e00c8b20 msgr2=0x7f80e0145310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:47.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.620+0000 7f80dde00640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80e00c8b20 0x7f80e0145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:47.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.620+0000 7f80dde00640 1 -- 172.21.15.102:0/942567340 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e00caf20 con 0x7f80e00c8740 2024-03-18T16:31:47.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.622+0000 7f80dde00640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e00c8740 0x7f80e01f43f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f80d000b410 tx=0x7f80d000b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:47.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.627+0000 7f80d6000640 1 -- 172.21.15.102:0/942567340 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0015020 con 0x7f80e00c8740 2024-03-18T16:31:47.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.627+0000 7f80d6000640 1 -- 172.21.15.102:0/942567340 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80d0011850 con 0x7f80e00c8740 2024-03-18T16:31:47.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.627+0000 7f80d6000640 1 -- 172.21.15.102:0/942567340 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0011c40 con 0x7f80e00c8740 2024-03-18T16:31:47.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.627+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e00cb1e0 con 0x7f80e00c8740 2024-03-18T16:31:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.627+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e00cb7a0 con 0x7f80e00c8740 2024-03-18T16:31:47.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.629+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e0074fd0 con 0x7f80e00c8740 2024-03-18T16:31:47.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.648+0000 7f80d6000640 1 -- 172.21.15.102:0/942567340 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f80d0018020 con 0x7f80e00c8740 2024-03-18T16:31:47.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.650+0000 7f80d6000640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f80c006bc20 0x7f80c006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:47.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.650+0000 7f80dd400640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f80c006bc20 0x7f80c006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:47.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.651+0000 7f80dd400640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f80c006bc20 0x7f80c006e0e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f80cc006fd0 tx=0x7f80cc006bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:47.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.652+0000 7f80d6000640 1 -- 172.21.15.102:0/942567340 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f80d0011de0 con 0x7f80e00c8740 2024-03-18T16:31:47.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:47.903+0000 7f80d6000640 1 -- 172.21.15.102:0/942567340 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f80d001f020 con 0x7f80e00c8740 2024-03-18T16:31:48.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.147+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80e0073280 con 0x7f80e00c8740 2024-03-18T16:31:48.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.159+0000 7f80d6000640 1 -- 172.21.15.102:0/942567340 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f80d0059a70 con 0x7f80e00c8740 2024-03-18T16:31:48.161 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:48.161 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:48.161 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:48.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.162+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f80c006bc20 msgr2=0x7f80c006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:48.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.162+0000 7f80e4a00640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f80c006bc20 0x7f80c006e0e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f80cc006fd0 tx=0x7f80cc006bc0 comp rx=0 tx=0).stop 2024-03-18T16:31:48.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.162+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e00c8740 msgr2=0x7f80e01f43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:48.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.162+0000 7f80e4a00640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e00c8740 0x7f80e01f43f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f80d000b410 tx=0x7f80d000b8d0 comp rx=0 tx=0).stop 2024-03-18T16:31:48.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.163+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 shutdown_connections 2024-03-18T16:31:48.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.163+0000 7f80e4a00640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f80c006bc20 0x7f80c006e0e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:48.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.163+0000 7f80e4a00640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f80e0158760 0x7f80e00cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:48.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.163+0000 7f80e4a00640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80e00c8b20 0x7f80e0145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:48.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.163+0000 7f80e4a00640 1 --2- 172.21.15.102:0/942567340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80e00c8740 0x7f80e01f43f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:48.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.163+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 >> 172.21.15.102:0/942567340 conn(0x7f80e00700d0 msgr2=0x7f80e00c3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:48.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.163+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 shutdown_connections 2024-03-18T16:31:48.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:48.163+0000 7f80e4a00640 1 -- 172.21.15.102:0/942567340 wait complete. 2024-03-18T16:31:48.785 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:48.785 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:49.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:49.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.960+0000 7fedff200640 1 Processor -- start 2024-03-18T16:31:49.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.960+0000 7fedff200640 1 -- start start 2024-03-18T16:31:49.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.961+0000 7fedff200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fee000cfe70 0x7fee0015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:49.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.961+0000 7fedff200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fee000cc2f0 0x7fee000cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:49.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.962+0000 7fedff200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee000ccc80 con 0x7fee000d0250 2024-03-18T16:31:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.962+0000 7fedff200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fee000cabd0 con 0x7fee000cc2f0 2024-03-18T16:31:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.962+0000 7fedff200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fee000caea0 con 0x7fee000cfe70 2024-03-18T16:31:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.962+0000 7fedfe800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fee000cc2f0 0x7fee000cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.962+0000 7fedfde00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fee000cfe70 0x7fee0015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.963+0000 7fedfe800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fee000cc2f0 0x7fee000cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50118/0 (socket says 172.21.15.102:50118) 2024-03-18T16:31:49.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.963+0000 7fedfe800640 1 -- 172.21.15.102:0/1643802269 learned_addr learned my addr 172.21.15.102:0/1643802269 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.965+0000 7fedfe800640 1 -- 172.21.15.102:0/1643802269 >> v2:172.21.15.154:3300/0 conn(0x7fee000cfe70 msgr2=0x7fee0015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.965+0000 7fedfe800640 1 --2- 172.21.15.102:0/1643802269 >> v2:172.21.15.154:3300/0 conn(0x7fee000cfe70 0x7fee0015ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.965+0000 7fedfe800640 1 -- 172.21.15.102:0/1643802269 >> v1:172.21.15.154:6789/0 conn(0x7fee000d0250 legacy=0x7fee000c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:49.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.965+0000 7fedfe800640 1 -- 172.21.15.102:0/1643802269 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee000cb180 con 0x7fee000cc2f0 2024-03-18T16:31:49.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.968+0000 7fedfe800640 1 --2- 172.21.15.102:0/1643802269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cc2f0 0x7fee000cc740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fee000cd800 tx=0x7fedf802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193a6e0bb1344f00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:49.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.971+0000 7fedfca00640 1 -- 172.21.15.102:0/1643802269 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf802cbb0 con 0x7fee000cc2f0 2024-03-18T16:31:49.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.973+0000 7fedfca00640 1 -- 172.21.15.102:0/1643802269 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedf802cd50 con 0x7fee000cc2f0 2024-03-18T16:31:49.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.973+0000 7fedfca00640 1 -- 172.21.15.102:0/1643802269 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf8034610 con 0x7fee000cc2f0 2024-03-18T16:31:49.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.974+0000 7fedff200640 1 -- 172.21.15.102:0/1643802269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cc2f0 msgr2=0x7fee000cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:49.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.974+0000 7fedff200640 1 --2- 172.21.15.102:0/1643802269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cc2f0 0x7fee000cc740 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fee000cd800 tx=0x7fedf802c010 comp rx=0 tx=0).stop 2024-03-18T16:31:49.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.974+0000 7fedff200640 1 -- 172.21.15.102:0/1643802269 shutdown_connections 2024-03-18T16:31:49.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.974+0000 7fedff200640 1 --2- 172.21.15.102:0/1643802269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cc2f0 0x7fee000cc740 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:49.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.974+0000 7fedff200640 1 --2- 172.21.15.102:0/1643802269 >> v2:172.21.15.154:3300/0 conn(0x7fee000cfe70 0x7fee0015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:49.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.974+0000 7fedff200640 1 -- 172.21.15.102:0/1643802269 >> 172.21.15.102:0/1643802269 conn(0x7fee0008c3e0 msgr2=0x7fee0006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:49.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.974+0000 7fedff200640 1 -- 172.21.15.102:0/1643802269 shutdown_connections 2024-03-18T16:31:49.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.974+0000 7fedff200640 1 -- 172.21.15.102:0/1643802269 wait complete. 2024-03-18T16:31:49.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.977+0000 7fedff200640 1 Processor -- start 2024-03-18T16:31:49.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.977+0000 7fedff200640 1 -- start start 2024-03-18T16:31:49.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.978+0000 7fedff200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee000cc2f0 0x7fee001f0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:49.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.978+0000 7fedff200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cfe70 0x7fee00158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:49.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.978+0000 7fedff200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee000d0250 0x7fee001e4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:49.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.978+0000 7fedff200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fee000cba20 con 0x7fee000cfe70 2024-03-18T16:31:49.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.978+0000 7fedff200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fee000cb740 con 0x7fee000cc2f0 2024-03-18T16:31:49.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.978+0000 7fedff200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fee000cbd00 con 0x7fee000d0250 2024-03-18T16:31:49.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.979+0000 7fedfe800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee000d0250 0x7fee001e4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:49.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.979+0000 7fedfd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cfe70 0x7fee00158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:49.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.979+0000 7fedfde00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee000cc2f0 0x7fee001f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:49.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.979+0000 7fedfd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cfe70 0x7fee00158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50126/0 (socket says 172.21.15.102:50126) 2024-03-18T16:31:49.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.979+0000 7fedfd400640 1 -- 172.21.15.102:0/3776547236 learned_addr learned my addr 172.21.15.102:0/3776547236 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:49.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.983+0000 7fedfd400640 1 -- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee000d0250 msgr2=0x7fee001e4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:49.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.983+0000 7fedfd400640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee000d0250 0x7fee001e4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:49.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.983+0000 7fedfd400640 1 -- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee000cc2f0 msgr2=0x7fee001f0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:49.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.983+0000 7fedfd400640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee000cc2f0 0x7fee001f0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:49.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.983+0000 7fedfd400640 1 -- 172.21.15.102:0/3776547236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee001f0880 con 0x7fee000cfe70 2024-03-18T16:31:49.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.986+0000 7fedfd400640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cfe70 0x7fee00158f80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fedec00c470 tx=0x7fedec00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:49.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.990+0000 7fedf6000640 1 -- 172.21.15.102:0/3776547236 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec0052d0 con 0x7fee000cfe70 2024-03-18T16:31:49.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.990+0000 7fedf6000640 1 -- 172.21.15.102:0/3776547236 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedec005490 con 0x7fee000cfe70 2024-03-18T16:31:49.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.990+0000 7fedf6000640 1 -- 172.21.15.102:0/3776547236 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec007260 con 0x7fee000cfe70 2024-03-18T16:31:49.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.991+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee001e05e0 con 0x7fee000cfe70 2024-03-18T16:31:49.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.992+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee001efb40 con 0x7fee000cfe70 2024-03-18T16:31:49.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:49.993+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee00074fd0 con 0x7fee000cfe70 2024-03-18T16:31:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.013+0000 7fedf6000640 1 -- 172.21.15.102:0/3776547236 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fedec007400 con 0x7fee000cfe70 2024-03-18T16:31:50.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.015+0000 7fedf6000640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fede006bc50 0x7fede006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:50.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.015+0000 7fedfde00640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fede006bc50 0x7fede006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:50.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.016+0000 7fedfde00640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fede006bc50 0x7fede006e110 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009b40 tx=0x7fedf0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:50.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.018+0000 7fedf6000640 1 -- 172.21.15.102:0/3776547236 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fedec056860 con 0x7fee000cfe70 2024-03-18T16:31:50.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.280+0000 7fedf6000640 1 -- 172.21.15.102:0/3776547236 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fedec0587c0 con 0x7fee000cfe70 2024-03-18T16:31:50.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.517+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee000cf1a0 con 0x7fee000cfe70 2024-03-18T16:31:50.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.541+0000 7fedf6000640 1 -- 172.21.15.102:0/3776547236 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fedec05c860 con 0x7fee000cfe70 2024-03-18T16:31:50.543 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:50.544 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:50.544 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:50.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fede006bc50 msgr2=0x7fede006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:50.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fede006bc50 0x7fede006e110 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009b40 tx=0x7fedf0009190 comp rx=0 tx=0).stop 2024-03-18T16:31:50.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cfe70 msgr2=0x7fee00158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:50.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cfe70 0x7fee00158f80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fedec00c470 tx=0x7fedec00c930 comp rx=0 tx=0).stop 2024-03-18T16:31:50.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 shutdown_connections 2024-03-18T16:31:50.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fede006bc50 0x7fede006e110 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:50.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fee000d0250 0x7fee001e4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:50.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee000cfe70 0x7fee00158f80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:50.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 --2- 172.21.15.102:0/3776547236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee000cc2f0 0x7fee001f0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:50.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 >> 172.21.15.102:0/3776547236 conn(0x7fee0008c3e0 msgr2=0x7fee000ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:50.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 shutdown_connections 2024-03-18T16:31:50.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:50.544+0000 7fedff200640 1 -- 172.21.15.102:0/3776547236 wait complete. 2024-03-18T16:31:51.174 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:51.174 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:52.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:52.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.348+0000 7f1643200640 1 Processor -- start 2024-03-18T16:31:52.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.348+0000 7f1643200640 1 -- start start 2024-03-18T16:31:52.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.349+0000 7f1643200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f164415c3a0 0x7f1644163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:52.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.349+0000 7f1643200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16440c72d0 con 0x7f1644141650 2024-03-18T16:31:52.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.349+0000 7f1643200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1644158840 con 0x7f1644158420 2024-03-18T16:31:52.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.349+0000 7f1643200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1644159370 con 0x7f164415c3a0 2024-03-18T16:31:52.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.350+0000 7f1641e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1644141650 0x7f16440c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47448/0 (socket says 172.21.15.102:47448) 2024-03-18T16:31:52.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.350+0000 7f1641e00640 1 -- 172.21.15.102:0/2285704656 learned_addr learned my addr 172.21.15.102:0/2285704656 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:31:52.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.350+0000 7f1642800640 1 --2- 172.21.15.102:0/2285704656 >> v2:172.21.15.154:3300/0 conn(0x7f164415c3a0 0x7f1644163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:52.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.354+0000 7f1640a00640 1 -- 172.21.15.102:0/2285704656 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335307551 0 0) 0x7f16440c72d0 con 0x7f1644141650 2024-03-18T16:31:52.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.354+0000 7f1640a00640 1 -- 172.21.15.102:0/2285704656 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1628003520 con 0x7f1644141650 2024-03-18T16:31:52.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.355+0000 7f1642800640 1 -- 172.21.15.102:0/2285704656 >> v1:172.21.15.154:6789/0 conn(0x7f1644158420 legacy=0x7f16440c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:52.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.355+0000 7f1642800640 1 -- 172.21.15.102:0/2285704656 >> v1:172.21.15.102:6789/0 conn(0x7f1644141650 legacy=0x7f16440c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:52.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.355+0000 7f1642800640 1 -- 172.21.15.102:0/2285704656 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1644158ad0 con 0x7f164415c3a0 2024-03-18T16:31:52.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.360+0000 7f1642800640 1 --2- 172.21.15.102:0/2285704656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f164415c3a0 0x7f1644163cd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f163c009d90 tx=0x7f163c02bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47f3ea3d1d603d09 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:52.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.365+0000 7f1640a00640 1 -- 172.21.15.102:0/2285704656 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c039070 con 0x7f164415c3a0 2024-03-18T16:31:52.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.365+0000 7f1640a00640 1 -- 172.21.15.102:0/2285704656 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f163c02ccf0 con 0x7f164415c3a0 2024-03-18T16:31:52.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.365+0000 7f1640a00640 1 -- 172.21.15.102:0/2285704656 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c034660 con 0x7f164415c3a0 2024-03-18T16:31:52.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.366+0000 7f1643200640 1 -- 172.21.15.102:0/2285704656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f164415c3a0 msgr2=0x7f1644163cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:52.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.366+0000 7f1643200640 1 --2- 172.21.15.102:0/2285704656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f164415c3a0 0x7f1644163cd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f163c009d90 tx=0x7f163c02bf40 comp rx=0 tx=0).stop 2024-03-18T16:31:52.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.366+0000 7f1643200640 1 -- 172.21.15.102:0/2285704656 shutdown_connections 2024-03-18T16:31:52.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.366+0000 7f1643200640 1 --2- 172.21.15.102:0/2285704656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f164415c3a0 0x7f1644163cd0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:52.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.366+0000 7f1643200640 1 -- 172.21.15.102:0/2285704656 >> 172.21.15.102:0/2285704656 conn(0x7f16440cb890 msgr2=0x7f16440cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:52.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.366+0000 7f1643200640 1 -- 172.21.15.102:0/2285704656 shutdown_connections 2024-03-18T16:31:52.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.366+0000 7f1643200640 1 -- 172.21.15.102:0/2285704656 wait complete. 2024-03-18T16:31:52.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.369+0000 7f1643200640 1 Processor -- start 2024-03-18T16:31:52.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.369+0000 7f1643200640 1 -- start start 2024-03-18T16:31:52.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.370+0000 7f1643200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1644141650 0x7f16441f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:52.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.370+0000 7f1643200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644158420 0x7f16441f0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:52.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.370+0000 7f1643200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f164415c3a0 0x7f16441e3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:52.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.371+0000 7f1643200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f16440c63c0 con 0x7f1644158420 2024-03-18T16:31:52.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.371+0000 7f1643200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1644159090 con 0x7f1644141650 2024-03-18T16:31:52.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.371+0000 7f1643200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f16440c66a0 con 0x7f164415c3a0 2024-03-18T16:31:52.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.371+0000 7f1641e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1644141650 0x7f16441f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:52.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.371+0000 7f1641400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644158420 0x7f16441f0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:52.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.371+0000 7f1642800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f164415c3a0 0x7f16441e3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:52.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.372+0000 7f1641e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1644141650 0x7f16441f4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38014/0 (socket says 172.21.15.102:38014) 2024-03-18T16:31:52.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.372+0000 7f1641e00640 1 -- 172.21.15.102:0/551501648 learned_addr learned my addr 172.21.15.102:0/551501648 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:52.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.375+0000 7f1641e00640 1 -- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f164415c3a0 msgr2=0x7f16441e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:52.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.375+0000 7f1641e00640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f164415c3a0 0x7f16441e3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:52.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.375+0000 7f1641e00640 1 -- 172.21.15.102:0/551501648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644158420 msgr2=0x7f16441f0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:52.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.375+0000 7f1641e00640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644158420 0x7f16441f0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:52.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.375+0000 7f1641e00640 1 -- 172.21.15.102:0/551501648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16441e3ca0 con 0x7f1644141650 2024-03-18T16:31:52.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.377+0000 7f1641e00640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1644141650 0x7f16441f4110 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f163400c500 tx=0x7f163400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:52.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.381+0000 7f163a000640 1 -- 172.21.15.102:0/551501648 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1634017070 con 0x7f1644141650 2024-03-18T16:31:52.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.381+0000 7f163a000640 1 -- 172.21.15.102:0/551501648 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1634005ce0 con 0x7f1644141650 2024-03-18T16:31:52.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.381+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16441e3f80 con 0x7f1644141650 2024-03-18T16:31:52.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.381+0000 7f163a000640 1 -- 172.21.15.102:0/551501648 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1634007330 con 0x7f1644141650 2024-03-18T16:31:52.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.382+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16441f3910 con 0x7f1644141650 2024-03-18T16:31:52.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.383+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1644074fd0 con 0x7f1644141650 2024-03-18T16:31:52.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.648+0000 7f163a000640 1 -- 172.21.15.102:0/551501648 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f163401a070 con 0x7f1644141650 2024-03-18T16:31:52.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.649+0000 7f163a000640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f162406bc20 0x7f162406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:52.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.649+0000 7f1641400640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f162406bc20 0x7f162406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:52.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.650+0000 7f163a000640 1 -- 172.21.15.102:0/551501648 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1634056910 con 0x7f1644141650 2024-03-18T16:31:52.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.650+0000 7f1641400640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f162406bc20 0x7f162406e0e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f16440cc430 tx=0x7f1630009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:52.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.661+0000 7f163a000640 1 -- 172.21.15.102:0/551501648 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1634021020 con 0x7f1644141650 2024-03-18T16:31:52.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.921+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1644157870 con 0x7f1644141650 2024-03-18T16:31:52.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.934+0000 7f163a000640 1 -- 172.21.15.102:0/551501648 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1634058870 con 0x7f1644141650 2024-03-18T16:31:52.936 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:52.936 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:52.936 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:52.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f162406bc20 msgr2=0x7f162406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:52.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f162406bc20 0x7f162406e0e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f16440cc430 tx=0x7f1630009040 comp rx=0 tx=0).stop 2024-03-18T16:31:52.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1644141650 msgr2=0x7f16441f4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:52.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1644141650 0x7f16441f4110 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f163400c500 tx=0x7f163400c9c0 comp rx=0 tx=0).stop 2024-03-18T16:31:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 shutdown_connections 2024-03-18T16:31:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f162406bc20 0x7f162406e0e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f164415c3a0 0x7f16441e3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1644158420 0x7f16441f0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 --2- 172.21.15.102:0/551501648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1644141650 0x7f16441f4110 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 >> 172.21.15.102:0/551501648 conn(0x7f16440cb890 msgr2=0x7f1644156290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 shutdown_connections 2024-03-18T16:31:52.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:52.936+0000 7f1643200640 1 -- 172.21.15.102:0/551501648 wait complete. 2024-03-18T16:31:53.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:31:53.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:31:53.559 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:53.559 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:54.560 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:54.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.738+0000 7ff87a800640 1 Processor -- start 2024-03-18T16:31:54.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.738+0000 7ff87a800640 1 -- start start 2024-03-18T16:31:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.739+0000 7ff87a800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff87406b470 0x7ff87415ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.739+0000 7ff87a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff874154150 0x7ff874154580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.739+0000 7ff87a800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff87406bf60 con 0x7ff87406de50 2024-03-18T16:31:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.739+0000 7ff87a800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff8740ca0a0 con 0x7ff874154150 2024-03-18T16:31:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.739+0000 7ff87a800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff8740c94b0 con 0x7ff87406b470 2024-03-18T16:31:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.740+0000 7ff879400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff87406b470 0x7ff87415ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.740+0000 7ff879e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff874154150 0x7ff874154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.740+0000 7ff879400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff87406b470 0x7ff87415ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38022/0 (socket says 172.21.15.102:38022) 2024-03-18T16:31:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.740+0000 7ff879400640 1 -- 172.21.15.102:0/3479988605 learned_addr learned my addr 172.21.15.102:0/3479988605 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.742+0000 7ff879400640 1 -- 172.21.15.102:0/3479988605 >> v2:172.21.15.102:3300/0 conn(0x7ff874154150 msgr2=0x7ff874154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.743+0000 7ff879400640 1 --2- 172.21.15.102:0/3479988605 >> v2:172.21.15.102:3300/0 conn(0x7ff874154150 0x7ff874154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.743+0000 7ff879400640 1 -- 172.21.15.102:0/3479988605 >> v1:172.21.15.102:6789/0 conn(0x7ff87406de50 legacy=0x7ff87406b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.743+0000 7ff879400640 1 -- 172.21.15.102:0/3479988605 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8740c9770 con 0x7ff87406b470 2024-03-18T16:31:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.745+0000 7ff879400640 1 --2- 172.21.15.102:0/3479988605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff87406b470 0x7ff87415ad60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff87406f6d0 tx=0x7ff86802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a4a44237a572f90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.749+0000 7ff873e00640 1 -- 172.21.15.102:0/3479988605 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868034070 con 0x7ff87406b470 2024-03-18T16:31:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.749+0000 7ff873e00640 1 -- 172.21.15.102:0/3479988605 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff868030860 con 0x7ff87406b470 2024-03-18T16:31:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.749+0000 7ff873e00640 1 -- 172.21.15.102:0/3479988605 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868030c30 con 0x7ff87406b470 2024-03-18T16:31:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.750+0000 7ff87a800640 1 -- 172.21.15.102:0/3479988605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff87406b470 msgr2=0x7ff87415ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.750+0000 7ff87a800640 1 --2- 172.21.15.102:0/3479988605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff87406b470 0x7ff87415ad60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff87406f6d0 tx=0x7ff86802b9a0 comp rx=0 tx=0).stop 2024-03-18T16:31:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.750+0000 7ff87a800640 1 -- 172.21.15.102:0/3479988605 shutdown_connections 2024-03-18T16:31:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.750+0000 7ff87a800640 1 --2- 172.21.15.102:0/3479988605 >> v2:172.21.15.102:3300/0 conn(0x7ff874154150 0x7ff874154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.750+0000 7ff87a800640 1 --2- 172.21.15.102:0/3479988605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff87406b470 0x7ff87415ad60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.750+0000 7ff87a800640 1 -- 172.21.15.102:0/3479988605 >> 172.21.15.102:0/3479988605 conn(0x7ff8740727e0 msgr2=0x7ff874072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.750+0000 7ff87a800640 1 -- 172.21.15.102:0/3479988605 shutdown_connections 2024-03-18T16:31:54.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.751+0000 7ff87a800640 1 -- 172.21.15.102:0/3479988605 wait complete. 2024-03-18T16:31:54.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.753+0000 7ff87a800640 1 Processor -- start 2024-03-18T16:31:54.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.753+0000 7ff87a800640 1 -- start start 2024-03-18T16:31:54.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.755+0000 7ff87a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff87406b470 0x7ff8741580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:54.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.755+0000 7ff87a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff87406de50 0x7ff87414e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:54.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.755+0000 7ff87a800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff874154150 0x7ff8741efce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:54.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.755+0000 7ff87a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff8740c5190 con 0x7ff87406b470 2024-03-18T16:31:54.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.755+0000 7ff87a800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8740c9d30 con 0x7ff874154150 2024-03-18T16:31:54.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.755+0000 7ff87a800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff8740c5310 con 0x7ff87406de50 2024-03-18T16:31:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.755+0000 7ff879400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff87406b470 0x7ff8741580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.756+0000 7ff879e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff874154150 0x7ff8741efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.756+0000 7ff878a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff87406de50 0x7ff87414e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.756+0000 7ff879e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff874154150 0x7ff8741efce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38036/0 (socket says 172.21.15.102:38036) 2024-03-18T16:31:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.756+0000 7ff879400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff87406b470 0x7ff8741580e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:39080/0 (socket says 172.21.15.102:39080) 2024-03-18T16:31:54.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.756+0000 7ff879e00640 1 -- 172.21.15.102:0/1202625197 learned_addr learned my addr 172.21.15.102:0/1202625197 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.760+0000 7ff879400640 1 -- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff87406de50 msgr2=0x7ff87414e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.760+0000 7ff879400640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff87406de50 0x7ff87414e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.760+0000 7ff879400640 1 -- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff874154150 msgr2=0x7ff8741efce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.760+0000 7ff879400640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff874154150 0x7ff8741efce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.760+0000 7ff879400640 1 -- 172.21.15.102:0/1202625197 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff874158770 con 0x7ff87406b470 2024-03-18T16:31:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.763+0000 7ff879400640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff87406b470 0x7ff8741580e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff868033000 tx=0x7ff868004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.767+0000 7ff871600640 1 -- 172.21.15.102:0/1202625197 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868048070 con 0x7ff87406b470 2024-03-18T16:31:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.767+0000 7ff871600640 1 -- 172.21.15.102:0/1202625197 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff868030660 con 0x7ff87406b470 2024-03-18T16:31:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.767+0000 7ff871600640 1 -- 172.21.15.102:0/1202625197 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868034040 con 0x7ff87406b470 2024-03-18T16:31:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.767+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87415dbd0 con 0x7ff87406b470 2024-03-18T16:31:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.768+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff87415e190 con 0x7ff87406b470 2024-03-18T16:31:54.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:54.769+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff874074fd0 con 0x7ff87406b470 2024-03-18T16:31:55.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.066+0000 7ff871600640 1 -- 172.21.15.102:0/1202625197 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff86804b070 con 0x7ff87406b470 2024-03-18T16:31:55.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.067+0000 7ff871600640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff85406bc50 0x7ff85406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:55.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.068+0000 7ff878a00640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff85406bc50 0x7ff85406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:55.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.068+0000 7ff878a00640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff85406bc50 0x7ff85406e110 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff8740c71b0 tx=0x7ff860009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:55.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.084+0000 7ff871600640 1 -- 172.21.15.102:0/1202625197 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff8680791b0 con 0x7ff87406b470 2024-03-18T16:31:55.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.085+0000 7ff871600640 1 -- 172.21.15.102:0/1202625197 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff868052080 con 0x7ff87406b470 2024-03-18T16:31:55.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.507+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8740c5f90 con 0x7ff87406b470 2024-03-18T16:31:55.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.529+0000 7ff871600640 1 -- 172.21.15.102:0/1202625197 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff86807b110 con 0x7ff87406b470 2024-03-18T16:31:55.532 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:55.532 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:55.532 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:55.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff85406bc50 msgr2=0x7ff85406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:55.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff85406bc50 0x7ff85406e110 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff8740c71b0 tx=0x7ff860009040 comp rx=0 tx=0).stop 2024-03-18T16:31:55.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff87406b470 msgr2=0x7ff8741580e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:55.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff87406b470 0x7ff8741580e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff868033000 tx=0x7ff868004730 comp rx=0 tx=0).stop 2024-03-18T16:31:55.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 shutdown_connections 2024-03-18T16:31:55.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff85406bc50 0x7ff85406e110 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:55.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff874154150 0x7ff8741efce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:55.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff87406de50 0x7ff87414e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:55.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 --2- 172.21.15.102:0/1202625197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff87406b470 0x7ff8741580e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:55.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 >> 172.21.15.102:0/1202625197 conn(0x7ff8740727e0 msgr2=0x7ff8740ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:55.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 shutdown_connections 2024-03-18T16:31:55.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:55.532+0000 7ff87a800640 1 -- 172.21.15.102:0/1202625197 wait complete. 2024-03-18T16:31:56.163 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:56.164 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:57.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:57.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.358+0000 7f69dd600640 1 Processor -- start 2024-03-18T16:31:57.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.358+0000 7f69dd600640 1 -- start start 2024-03-18T16:31:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.359+0000 7f69dd600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f69d80cfe70 0x7f69d815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.359+0000 7f69dd600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f69d80d0250 0x7f69d80c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.359+0000 7f69dd600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d80cabd0 con 0x7f69d80cc2f0 2024-03-18T16:31:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.359+0000 7f69dd600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f69d80caea0 con 0x7f69d80d0250 2024-03-18T16:31:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.359+0000 7f69dd600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f69d80cb180 con 0x7f69d80cfe70 2024-03-18T16:31:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.360+0000 7f69d7e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f69d80cfe70 0x7f69d815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:57.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.360+0000 7f69d7e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f69d80cfe70 0x7f69d815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:38050/0 (socket says 172.21.15.102:38050) 2024-03-18T16:31:57.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.360+0000 7f69d7e00640 1 -- 172.21.15.102:0/50931486 learned_addr learned my addr 172.21.15.102:0/50931486 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:57.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.364+0000 7f69d7e00640 1 -- 172.21.15.102:0/50931486 >> v2:172.21.15.102:3300/0 conn(0x7f69d80d0250 msgr2=0x7f69d80c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:57.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.364+0000 7f69d7e00640 1 --2- 172.21.15.102:0/50931486 >> v2:172.21.15.102:3300/0 conn(0x7f69d80d0250 0x7f69d80c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:57.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.364+0000 7f69d7e00640 1 -- 172.21.15.102:0/50931486 >> v1:172.21.15.154:6789/0 conn(0x7f69d80cc2f0 legacy=0x7f69d80cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:57.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.364+0000 7f69d7e00640 1 -- 172.21.15.102:0/50931486 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d80cb460 con 0x7f69d80cfe70 2024-03-18T16:31:57.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.368+0000 7f69d7e00640 1 --2- 172.21.15.102:0/50931486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d80cfe70 0x7f69d815ae40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f69d80cd800 tx=0x7f69c802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e94d74de1347dc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:57.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.372+0000 7f69d6a00640 1 -- 172.21.15.102:0/50931486 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c802f850 con 0x7f69d80cfe70 2024-03-18T16:31:57.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.372+0000 7f69d6a00640 1 -- 172.21.15.102:0/50931486 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c802f9f0 con 0x7f69d80cfe70 2024-03-18T16:31:57.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.372+0000 7f69d6a00640 1 -- 172.21.15.102:0/50931486 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c802fdc0 con 0x7f69d80cfe70 2024-03-18T16:31:57.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.373+0000 7f69dd600640 1 -- 172.21.15.102:0/50931486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d80cfe70 msgr2=0x7f69d815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:57.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.373+0000 7f69dd600640 1 --2- 172.21.15.102:0/50931486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d80cfe70 0x7f69d815ae40 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f69d80cd800 tx=0x7f69c802ba30 comp rx=0 tx=0).stop 2024-03-18T16:31:57.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.374+0000 7f69dd600640 1 -- 172.21.15.102:0/50931486 shutdown_connections 2024-03-18T16:31:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.374+0000 7f69dd600640 1 --2- 172.21.15.102:0/50931486 >> v2:172.21.15.102:3300/0 conn(0x7f69d80d0250 0x7f69d80c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.374+0000 7f69dd600640 1 --2- 172.21.15.102:0/50931486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d80cfe70 0x7f69d815ae40 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.374+0000 7f69dd600640 1 -- 172.21.15.102:0/50931486 >> 172.21.15.102:0/50931486 conn(0x7f69d808c3e0 msgr2=0x7f69d806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.374+0000 7f69dd600640 1 -- 172.21.15.102:0/50931486 shutdown_connections 2024-03-18T16:31:57.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.374+0000 7f69dd600640 1 -- 172.21.15.102:0/50931486 wait complete. 2024-03-18T16:31:57.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.376+0000 7f69dd600640 1 Processor -- start 2024-03-18T16:31:57.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.377+0000 7f69dd600640 1 -- start start 2024-03-18T16:31:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.378+0000 7f69dd600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d80cc2f0 0x7f69d80c5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.378+0000 7f69dd600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69d80cfe70 0x7f69d8155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.378+0000 7f69dd600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69d80d0250 0x7f69d8155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.378+0000 7f69dd600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f69d80cbd00 con 0x7f69d80cfe70 2024-03-18T16:31:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.378+0000 7f69dd600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69d80cba20 con 0x7f69d80cc2f0 2024-03-18T16:31:57.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.378+0000 7f69dd600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f69d80c6b20 con 0x7f69d80d0250 2024-03-18T16:31:57.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.379+0000 7f69dcc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69d80d0250 0x7f69d8155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:57.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.379+0000 7f69d7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d80cc2f0 0x7f69d80c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:57.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.379+0000 7f69dcc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69d80d0250 0x7f69d8155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:36078/0 (socket says 172.21.15.102:36078) 2024-03-18T16:31:57.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.379+0000 7f69dcc00640 1 -- 172.21.15.102:0/2958569286 learned_addr learned my addr 172.21.15.102:0/2958569286 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:57.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.383+0000 7f69dcc00640 1 -- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d80cc2f0 msgr2=0x7f69d80c5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:57.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.383+0000 7f69dcc00640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d80cc2f0 0x7f69d80c5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:57.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.383+0000 7f69dcc00640 1 -- 172.21.15.102:0/2958569286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69d80cfe70 msgr2=0x7f69d8155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:57.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.383+0000 7f69dcc00640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69d80cfe70 0x7f69d8155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:57.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.383+0000 7f69dcc00640 1 -- 172.21.15.102:0/2958569286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d8155150 con 0x7f69d80d0250 2024-03-18T16:31:57.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.386+0000 7f69dcc00640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69d80d0250 0x7f69d8155e50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f69d000c6b0 tx=0x7f69d000cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:57.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.458+0000 7f69cfe00640 1 -- 172.21.15.102:0/2958569286 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d0003b70 con 0x7f69d80d0250 2024-03-18T16:31:57.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.458+0000 7f69cfe00640 1 -- 172.21.15.102:0/2958569286 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d0003d30 con 0x7f69d80d0250 2024-03-18T16:31:57.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.458+0000 7f69cfe00640 1 -- 172.21.15.102:0/2958569286 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d0005310 con 0x7f69d80d0250 2024-03-18T16:31:57.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.459+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d81f3830 con 0x7f69d80d0250 2024-03-18T16:31:57.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.459+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d81f3df0 con 0x7f69d80d0250 2024-03-18T16:31:57.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.460+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d80c8eb0 con 0x7f69d80d0250 2024-03-18T16:31:57.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.736+0000 7f69cfe00640 1 -- 172.21.15.102:0/2958569286 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f69d00054b0 con 0x7f69d80d0250 2024-03-18T16:31:57.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.738+0000 7f69cfe00640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f69b806bc00 0x7f69b806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:57.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.738+0000 7f69d7e00640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f69b806bc00 0x7f69b806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:57.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.739+0000 7f69d7e00640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f69b806bc00 0x7f69b806e0c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:57.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.740+0000 7f69cfe00640 1 -- 172.21.15.102:0/2958569286 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f69d00057a0 con 0x7f69d80d0250 2024-03-18T16:31:57.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:57.758+0000 7f69cfe00640 1 -- 172.21.15.102:0/2958569286 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f69d0059e30 con 0x7f69d80d0250 2024-03-18T16:31:58.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.001+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69d80cf1a0 con 0x7f69d80d0250 2024-03-18T16:31:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.015+0000 7f69cfe00640 1 -- 172.21.15.102:0/2958569286 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f69d0059730 con 0x7f69d80d0250 2024-03-18T16:31:58.018 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:31:58.018 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:31:58.018 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.018+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f69b806bc00 msgr2=0x7f69b806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.018+0000 7f69dd600640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f69b806bc00 0x7f69b806e0c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f69c8036000 tx=0x7f69c8008030 comp rx=0 tx=0).stop 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69d80d0250 msgr2=0x7f69d8155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69d80d0250 0x7f69d8155e50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f69d000c6b0 tx=0x7f69d000cb70 comp rx=0 tx=0).stop 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 shutdown_connections 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f69b806bc00 0x7f69b806e0c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f69d80d0250 0x7f69d8155e50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69d80cfe70 0x7f69d8155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 --2- 172.21.15.102:0/2958569286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d80cc2f0 0x7f69d80c5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 >> 172.21.15.102:0/2958569286 conn(0x7f69d808c3e0 msgr2=0x7f69d80ce4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 shutdown_connections 2024-03-18T16:31:58.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:58.019+0000 7f69dd600640 1 -- 172.21.15.102:0/2958569286 wait complete. 2024-03-18T16:31:58.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:31:58.652 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:31:59.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:31:59.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.842+0000 7fbae4400640 1 Processor -- start 2024-03-18T16:31:59.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.842+0000 7fbae4400640 1 -- start start 2024-03-18T16:31:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.844+0000 7fbae4400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc15af20 0x7fbadc0c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.844+0000 7fbae4400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbadc070f70 0x7fbadc0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.844+0000 7fbae4400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbadc0c78a0 con 0x7fbadc154920 2024-03-18T16:31:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.844+0000 7fbae4400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbadc0c7b80 con 0x7fbadc070f70 2024-03-18T16:31:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.844+0000 7fbae4400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbadc0c7e60 con 0x7fbadc15af20 2024-03-18T16:31:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.864+0000 7fbae1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc15af20 0x7fbadc0c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.865+0000 7fbae1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc15af20 0x7fbadc0c97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:36094/0 (socket says 172.21.15.102:36094) 2024-03-18T16:31:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.865+0000 7fbae1000640 1 -- 172.21.15.102:0/3555747723 learned_addr learned my addr 172.21.15.102:0/3555747723 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:59.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.867+0000 7fbae1000640 1 -- 172.21.15.102:0/3555747723 >> v2:172.21.15.102:3300/0 conn(0x7fbadc070f70 msgr2=0x7fbadc0c9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:59.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.867+0000 7fbae1000640 1 --2- 172.21.15.102:0/3555747723 >> v2:172.21.15.102:3300/0 conn(0x7fbadc070f70 0x7fbadc0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:59.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.867+0000 7fbae1000640 1 -- 172.21.15.102:0/3555747723 >> v1:172.21.15.102:6789/0 conn(0x7fbadc154920 legacy=0x7fbadc0cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:31:59.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.867+0000 7fbae1000640 1 -- 172.21.15.102:0/3555747723 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc0c8140 con 0x7fbadc15af20 2024-03-18T16:31:59.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.869+0000 7fbae1000640 1 --2- 172.21.15.102:0/3555747723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc15af20 0x7fbadc0c97c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009870 tx=0x7fbac802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19717db7ee23daf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.873+0000 7fbadbe00640 1 -- 172.21.15.102:0/3555747723 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8035020 con 0x7fbadc15af20 2024-03-18T16:31:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.873+0000 7fbadbe00640 1 -- 172.21.15.102:0/3555747723 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac8031640 con 0x7fbadc15af20 2024-03-18T16:31:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.873+0000 7fbadbe00640 1 -- 172.21.15.102:0/3555747723 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac802d040 con 0x7fbadc15af20 2024-03-18T16:31:59.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.874+0000 7fbae4400640 1 -- 172.21.15.102:0/3555747723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc15af20 msgr2=0x7fbadc0c97c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:59.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.874+0000 7fbae4400640 1 --2- 172.21.15.102:0/3555747723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc15af20 0x7fbadc0c97c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009870 tx=0x7fbac802b8f0 comp rx=0 tx=0).stop 2024-03-18T16:31:59.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.874+0000 7fbae4400640 1 -- 172.21.15.102:0/3555747723 shutdown_connections 2024-03-18T16:31:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.874+0000 7fbae4400640 1 --2- 172.21.15.102:0/3555747723 >> v2:172.21.15.102:3300/0 conn(0x7fbadc070f70 0x7fbadc0c9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.874+0000 7fbae4400640 1 --2- 172.21.15.102:0/3555747723 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc15af20 0x7fbadc0c97c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.874+0000 7fbae4400640 1 -- 172.21.15.102:0/3555747723 >> 172.21.15.102:0/3555747723 conn(0x7fbadc072e30 msgr2=0x7fbadc073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:31:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.874+0000 7fbae4400640 1 -- 172.21.15.102:0/3555747723 shutdown_connections 2024-03-18T16:31:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.874+0000 7fbae4400640 1 -- 172.21.15.102:0/3555747723 wait complete. 2024-03-18T16:31:59.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.877+0000 7fbae4400640 1 Processor -- start 2024-03-18T16:31:59.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.877+0000 7fbae4400640 1 -- start start 2024-03-18T16:31:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.879+0000 7fbae4400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc070f70 0x7fbadc149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:59.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.879+0000 7fbae4400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbadc154920 0x7fbadc152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:59.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.879+0000 7fbae4400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbadc15af20 0x7fbadc153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:59.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.879+0000 7fbae4400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc0cd990 con 0x7fbadc154920 2024-03-18T16:31:59.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.879+0000 7fbae4400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc15b3c0 con 0x7fbadc15af20 2024-03-18T16:31:59.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.879+0000 7fbae4400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbadc0cdb10 con 0x7fbadc070f70 2024-03-18T16:31:59.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.879+0000 7fbae1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc070f70 0x7fbadc149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:59.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.880+0000 7fbae1a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc070f70 0x7fbadc149b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:36096/0 (socket says 172.21.15.102:36096) 2024-03-18T16:31:59.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.880+0000 7fbae1a00640 1 -- 172.21.15.102:0/3371575561 learned_addr learned my addr 172.21.15.102:0/3371575561 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:31:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.884+0000 7fbae1000640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbadc154920 0x7fbadc152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:59.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.887+0000 7fbae1a00640 1 -- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbadc15af20 msgr2=0x7fbadc153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:59.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.887+0000 7fbae1a00640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbadc15af20 0x7fbadc153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:59.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.887+0000 7fbae1a00640 1 -- 172.21.15.102:0/3371575561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbadc154920 msgr2=0x7fbadc152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:31:59.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.887+0000 7fbae1a00640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbadc154920 0x7fbadc152b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:31:59.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.887+0000 7fbae1a00640 1 -- 172.21.15.102:0/3371575561 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc1f0210 con 0x7fbadc070f70 2024-03-18T16:31:59.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.893+0000 7fbae1a00640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc070f70 0x7fbadc149b10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbad000c470 tx=0x7fbad000c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:31:59.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.899+0000 7fbad9600640 1 -- 172.21.15.102:0/3371575561 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0017070 con 0x7fbadc070f70 2024-03-18T16:31:59.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.899+0000 7fbad9600640 1 -- 172.21.15.102:0/3371575561 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad00035f0 con 0x7fbadc070f70 2024-03-18T16:31:59.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.899+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc1f04f0 con 0x7fbadc070f70 2024-03-18T16:31:59.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.899+0000 7fbad9600640 1 -- 172.21.15.102:0/3371575561 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad00039e0 con 0x7fbadc070f70 2024-03-18T16:31:59.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.899+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc1f0ab0 con 0x7fbadc070f70 2024-03-18T16:31:59.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.900+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc074fd0 con 0x7fbadc070f70 2024-03-18T16:31:59.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.938+0000 7fbad9600640 1 -- 172.21.15.102:0/3371575561 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbad001a070 con 0x7fbadc070f70 2024-03-18T16:31:59.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.939+0000 7fbad9600640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbabc06bc50 0x7fbabc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:31:59.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.939+0000 7fbae1000640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbabc06bc50 0x7fbabc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:31:59.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:31:59.940+0000 7fbae1000640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbabc06bc50 0x7fbabc06e110 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbac8036000 tx=0x7fbac803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:00.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.221+0000 7fbad9600640 1 -- 172.21.15.102:0/3371575561 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fbad0056810 con 0x7fbadc070f70 2024-03-18T16:32:00.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.231+0000 7fbad9600640 1 -- 172.21.15.102:0/3371575561 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbad0021020 con 0x7fbadc070f70 2024-03-18T16:32:00.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.469+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbadc070ce0 con 0x7fbadc070f70 2024-03-18T16:32:00.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.482+0000 7fbad9600640 1 -- 172.21.15.102:0/3371575561 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fbad00010c0 con 0x7fbadc070f70 2024-03-18T16:32:00.485 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:00.485 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:00.485 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.484+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbabc06bc50 msgr2=0x7fbabc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.484+0000 7fbae4400640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbabc06bc50 0x7fbabc06e110 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbac8036000 tx=0x7fbac803c040 comp rx=0 tx=0).stop 2024-03-18T16:32:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.484+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc070f70 msgr2=0x7fbadc149b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.484+0000 7fbae4400640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc070f70 0x7fbadc149b10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbad000c470 tx=0x7fbad000c930 comp rx=0 tx=0).stop 2024-03-18T16:32:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.484+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 shutdown_connections 2024-03-18T16:32:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.485+0000 7fbae4400640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbabc06bc50 0x7fbabc06e110 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.485+0000 7fbae4400640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbadc15af20 0x7fbadc153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.485+0000 7fbae4400640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbadc154920 0x7fbadc152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.485+0000 7fbae4400640 1 --2- 172.21.15.102:0/3371575561 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbadc070f70 0x7fbadc149b10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:00.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.485+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 >> 172.21.15.102:0/3371575561 conn(0x7fbadc072e30 msgr2=0x7fbadc073210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:00.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.485+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 shutdown_connections 2024-03-18T16:32:00.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:00.485+0000 7fbae4400640 1 -- 172.21.15.102:0/3371575561 wait complete. 2024-03-18T16:32:01.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:01.107 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:02.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.307+0000 7fe182000640 1 Processor -- start 2024-03-18T16:32:02.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.308+0000 7fe182000640 1 -- start start 2024-03-18T16:32:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.309+0000 7fe182000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe17c154520 0x7fe17c0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.309+0000 7fe182000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3cd0 0x7fe17c0c9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.309+0000 7fe182000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c15a170 con 0x7fe17c0c3890 2024-03-18T16:32:02.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.309+0000 7fe182000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe17c154900 con 0x7fe17c154520 2024-03-18T16:32:02.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.309+0000 7fe182000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c0c4160 con 0x7fe17c0c3cd0 2024-03-18T16:32:02.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.309+0000 7fe17be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe17c0c3890 0x7fe17c0c8920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49716/0 (socket says 172.21.15.102:49716) 2024-03-18T16:32:02.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.309+0000 7fe17be00640 1 -- 172.21.15.102:0/623949610 learned_addr learned my addr 172.21.15.102:0/623949610 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:02.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.310+0000 7fe181600640 1 --2- 172.21.15.102:0/623949610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3cd0 0x7fe17c0c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:02.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.310+0000 7fe180c00640 1 --2- 172.21.15.102:0/623949610 >> v2:172.21.15.154:3300/0 conn(0x7fe17c154520 0x7fe17c0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.312+0000 7fe181600640 1 -- 172.21.15.102:0/623949610 >> v2:172.21.15.154:3300/0 conn(0x7fe17c154520 msgr2=0x7fe17c0c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.312+0000 7fe181600640 1 --2- 172.21.15.102:0/623949610 >> v2:172.21.15.154:3300/0 conn(0x7fe17c154520 0x7fe17c0c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.312+0000 7fe181600640 1 -- 172.21.15.102:0/623949610 >> v1:172.21.15.102:6789/0 conn(0x7fe17c0c3890 legacy=0x7fe17c0c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:02.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.312+0000 7fe181600640 1 -- 172.21.15.102:0/623949610 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c0c4440 con 0x7fe17c0c3cd0 2024-03-18T16:32:02.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.315+0000 7fe181600640 1 --2- 172.21.15.102:0/623949610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3cd0 0x7fe17c0c9370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe174009d90 tx=0x7fe17402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fea9b25a7376881 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:02.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.319+0000 7fe17b400640 1 -- 172.21.15.102:0/623949610 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe174039070 con 0x7fe17c0c3cd0 2024-03-18T16:32:02.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.319+0000 7fe17b400640 1 -- 172.21.15.102:0/623949610 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe17402cc60 con 0x7fe17c0c3cd0 2024-03-18T16:32:02.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.319+0000 7fe17b400640 1 -- 172.21.15.102:0/623949610 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe174034660 con 0x7fe17c0c3cd0 2024-03-18T16:32:02.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.320+0000 7fe182000640 1 -- 172.21.15.102:0/623949610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3cd0 msgr2=0x7fe17c0c9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:02.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.320+0000 7fe182000640 1 --2- 172.21.15.102:0/623949610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3cd0 0x7fe17c0c9370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe174009d90 tx=0x7fe17402beb0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.320+0000 7fe182000640 1 -- 172.21.15.102:0/623949610 shutdown_connections 2024-03-18T16:32:02.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.320+0000 7fe182000640 1 --2- 172.21.15.102:0/623949610 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3cd0 0x7fe17c0c9370 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.320+0000 7fe182000640 1 --2- 172.21.15.102:0/623949610 >> v2:172.21.15.154:3300/0 conn(0x7fe17c154520 0x7fe17c0c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.320+0000 7fe182000640 1 -- 172.21.15.102:0/623949610 >> 172.21.15.102:0/623949610 conn(0x7fe17c066f80 msgr2=0x7fe17c067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:02.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.320+0000 7fe182000640 1 -- 172.21.15.102:0/623949610 shutdown_connections 2024-03-18T16:32:02.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.320+0000 7fe182000640 1 -- 172.21.15.102:0/623949610 wait complete. 2024-03-18T16:32:02.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.323+0000 7fe182000640 1 Processor -- start 2024-03-18T16:32:02.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.323+0000 7fe182000640 1 -- start start 2024-03-18T16:32:02.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.324+0000 7fe182000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3890 0x7fe17c1e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:02.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.325+0000 7fe182000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c154520 0x7fe17c1f0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:02.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.325+0000 7fe182000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17c152ff0 0x7fe17c1f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:02.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.325+0000 7fe182000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c15a9f0 con 0x7fe17c152ff0 2024-03-18T16:32:02.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.325+0000 7fe182000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c15a710 con 0x7fe17c154520 2024-03-18T16:32:02.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.325+0000 7fe182000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c0ca3f0 con 0x7fe17c0c3890 2024-03-18T16:32:02.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.325+0000 7fe180c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3890 0x7fe17c1e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:02.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.325+0000 7fe181600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17c152ff0 0x7fe17c1f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:02.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.326+0000 7fe180c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3890 0x7fe17c1e6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:42708/0 (socket says 172.21.15.102:42708) 2024-03-18T16:32:02.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.326+0000 7fe180c00640 1 -- 172.21.15.102:0/2785431103 learned_addr learned my addr 172.21.15.102:0/2785431103 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:02.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.327+0000 7fe17be00640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c154520 0x7fe17c1f0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:02.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.329+0000 7fe180c00640 1 -- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c154520 msgr2=0x7fe17c1f0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:02.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.329+0000 7fe180c00640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c154520 0x7fe17c1f0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.329+0000 7fe180c00640 1 -- 172.21.15.102:0/2785431103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17c152ff0 msgr2=0x7fe17c1f08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:02.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.329+0000 7fe180c00640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17c152ff0 0x7fe17c1f08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.329+0000 7fe180c00640 1 -- 172.21.15.102:0/2785431103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c0c55b0 con 0x7fe17c0c3890 2024-03-18T16:32:02.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.331+0000 7fe180c00640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3890 0x7fe17c1e6ac0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe17000e8d0 tx=0x7fe17000ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.335+0000 7fe178c00640 1 -- 172.21.15.102:0/2785431103 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170017070 con 0x7fe17c0c3890 2024-03-18T16:32:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.335+0000 7fe178c00640 1 -- 172.21.15.102:0/2785431103 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe17000c8e0 con 0x7fe17c0c3890 2024-03-18T16:32:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.335+0000 7fe178c00640 1 -- 172.21.15.102:0/2785431103 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17000ccd0 con 0x7fe17c0c3890 2024-03-18T16:32:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.335+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c1f43e0 con 0x7fe17c0c3890 2024-03-18T16:32:02.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.335+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17c1e7000 con 0x7fe17c0c3890 2024-03-18T16:32:02.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.337+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe17c074fd0 con 0x7fe17c0c3890 2024-03-18T16:32:02.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.358+0000 7fe178c00640 1 -- 172.21.15.102:0/2785431103 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe17001a070 con 0x7fe17c0c3890 2024-03-18T16:32:02.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.360+0000 7fe178c00640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15c06bc50 0x7fe15c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:02.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.360+0000 7fe17be00640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15c06bc50 0x7fe15c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:02.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.361+0000 7fe17be00640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15c06bc50 0x7fe15c06e110 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe17c1ede40 tx=0x7fe168007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:02.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.599+0000 7fe178c00640 1 -- 172.21.15.102:0/2785431103 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe170057680 con 0x7fe17c0c3890 2024-03-18T16:32:02.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.610+0000 7fe178c00640 1 -- 172.21.15.102:0/2785431103 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe1700595e0 con 0x7fe17c0c3890 2024-03-18T16:32:02.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.859+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe17c0c9150 con 0x7fe17c0c3890 2024-03-18T16:32:02.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.872+0000 7fe178c00640 1 -- 172.21.15.102:0/2785431103 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe17005d680 con 0x7fe17c0c3890 2024-03-18T16:32:02.873 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:02.874 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:02.874 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:02.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15c06bc50 msgr2=0x7fe15c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:02.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15c06bc50 0x7fe15c06e110 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe17c1ede40 tx=0x7fe168007e80 comp rx=0 tx=0).stop 2024-03-18T16:32:02.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3890 msgr2=0x7fe17c1e6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:02.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3890 0x7fe17c1e6ac0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe17000e8d0 tx=0x7fe17000ed90 comp rx=0 tx=0).stop 2024-03-18T16:32:02.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 shutdown_connections 2024-03-18T16:32:02.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe15c06bc50 0x7fe15c06e110 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe17c152ff0 0x7fe17c1f08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c154520 0x7fe17c1f0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 --2- 172.21.15.102:0/2785431103 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe17c0c3890 0x7fe17c1e6ac0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:02.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 >> 172.21.15.102:0/2785431103 conn(0x7fe17c066f80 msgr2=0x7fe17c0c7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:02.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 shutdown_connections 2024-03-18T16:32:02.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:02.874+0000 7fe182000640 1 -- 172.21.15.102:0/2785431103 wait complete. 2024-03-18T16:32:03.506 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:03.506 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:04.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:04.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.690+0000 7ff6e1000640 1 Processor -- start 2024-03-18T16:32:04.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.690+0000 7ff6e1000640 1 -- start start 2024-03-18T16:32:04.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.692+0000 7ff6e1000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff6dc0c7f30 0x7ff6dc157d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:04.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.692+0000 7ff6e1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c2d90 0x7ff6dc15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:04.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.692+0000 7ff6e1000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6dc073530 con 0x7ff6dc072810 2024-03-18T16:32:04.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.692+0000 7ff6e1000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff6dc0c5d80 con 0x7ff6dc0c7f30 2024-03-18T16:32:04.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.692+0000 7ff6e1000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc157900 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.693+0000 7ff6daa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c2d90 0x7ff6dc15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.693+0000 7ff6daa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c2d90 0x7ff6dc15b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:42712/0 (socket says 172.21.15.102:42712) 2024-03-18T16:32:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.693+0000 7ff6daa00640 1 -- 172.21.15.102:0/1745093001 learned_addr learned my addr 172.21.15.102:0/1745093001 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:04.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.693+0000 7ff6db400640 1 --2- 172.21.15.102:0/1745093001 >> v2:172.21.15.102:3300/0 conn(0x7ff6dc0c7f30 0x7ff6dc157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.695+0000 7ff6daa00640 1 -- 172.21.15.102:0/1745093001 >> v2:172.21.15.102:3300/0 conn(0x7ff6dc0c7f30 msgr2=0x7ff6dc157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.695+0000 7ff6daa00640 1 --2- 172.21.15.102:0/1745093001 >> v2:172.21.15.102:3300/0 conn(0x7ff6dc0c7f30 0x7ff6dc157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.695+0000 7ff6daa00640 1 -- 172.21.15.102:0/1745093001 >> v1:172.21.15.102:6789/0 conn(0x7ff6dc072810 legacy=0x7ff6dc06f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:04.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.695+0000 7ff6daa00640 1 -- 172.21.15.102:0/1745093001 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc0c57e0 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.698+0000 7ff6daa00640 1 --2- 172.21.15.102:0/1745093001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c2d90 0x7ff6dc15b0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009870 tx=0x7ff6c802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66ba2a8df598a8aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.701+0000 7ff6da000640 1 -- 172.21.15.102:0/1745093001 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c802f830 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.701+0000 7ff6da000640 1 -- 172.21.15.102:0/1745093001 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c802f9d0 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.701+0000 7ff6da000640 1 -- 172.21.15.102:0/1745093001 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c802fda0 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.702+0000 7ff6e1000640 1 -- 172.21.15.102:0/1745093001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c2d90 msgr2=0x7ff6dc15b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.702+0000 7ff6e1000640 1 --2- 172.21.15.102:0/1745093001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c2d90 0x7ff6dc15b0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009870 tx=0x7ff6c802ba10 comp rx=0 tx=0).stop 2024-03-18T16:32:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.702+0000 7ff6e1000640 1 -- 172.21.15.102:0/1745093001 shutdown_connections 2024-03-18T16:32:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.702+0000 7ff6e1000640 1 --2- 172.21.15.102:0/1745093001 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c2d90 0x7ff6dc15b0c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.702+0000 7ff6e1000640 1 --2- 172.21.15.102:0/1745093001 >> v2:172.21.15.102:3300/0 conn(0x7ff6dc0c7f30 0x7ff6dc157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.702+0000 7ff6e1000640 1 -- 172.21.15.102:0/1745093001 >> 172.21.15.102:0/1745093001 conn(0x7ff6dc0cb890 msgr2=0x7ff6dc0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.703+0000 7ff6e1000640 1 -- 172.21.15.102:0/1745093001 shutdown_connections 2024-03-18T16:32:04.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.703+0000 7ff6e1000640 1 -- 172.21.15.102:0/1745093001 wait complete. 2024-03-18T16:32:04.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.705+0000 7ff6e1000640 1 Processor -- start 2024-03-18T16:32:04.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.705+0000 7ff6e1000640 1 -- start start 2024-03-18T16:32:04.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.707+0000 7ff6e1000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc072810 0x7ff6dc1491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.707+0000 7ff6e1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6dc0c2d90 0x7ff6dc154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.707+0000 7ff6e1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c7f30 0x7ff6dc1373f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.707+0000 7ff6e1000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc0c5aa0 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.707+0000 7ff6e1000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc15b600 con 0x7ff6dc072810 2024-03-18T16:32:04.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.707+0000 7ff6e1000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff6dc1572b0 con 0x7ff6dc0c7f30 2024-03-18T16:32:04.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.708+0000 7ff6dbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c7f30 0x7ff6dc1373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:04.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.708+0000 7ff6daa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6dc0c2d90 0x7ff6dc154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:04.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.708+0000 7ff6db400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc072810 0x7ff6dc1491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:04.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.708+0000 7ff6dbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c7f30 0x7ff6dc1373f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:42728/0 (socket says 172.21.15.102:42728) 2024-03-18T16:32:04.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.708+0000 7ff6dbe00640 1 -- 172.21.15.102:0/3704085503 learned_addr learned my addr 172.21.15.102:0/3704085503 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:04.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.712+0000 7ff6daa00640 1 -- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c7f30 msgr2=0x7ff6dc1373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:04.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.712+0000 7ff6daa00640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c7f30 0x7ff6dc1373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:04.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.712+0000 7ff6daa00640 1 -- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc072810 msgr2=0x7ff6dc1491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:04.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.712+0000 7ff6daa00640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc072810 0x7ff6dc1491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:04.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.712+0000 7ff6daa00640 1 -- 172.21.15.102:0/3704085503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc1553b0 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.715+0000 7ff6daa00640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6dc0c2d90 0x7ff6dc154e70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0c5f00 tx=0x7ff6c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:04.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.719+0000 7ff6d3400640 1 -- 172.21.15.102:0/3704085503 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8047070 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.720+0000 7ff6d3400640 1 -- 172.21.15.102:0/3704085503 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c8037e10 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.720+0000 7ff6d3400640 1 -- 172.21.15.102:0/3704085503 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c80426c0 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.720+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc149ba0 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.720+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc1381c0 con 0x7ff6dc0c2d90 2024-03-18T16:32:04.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:04.721+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6dc147b20 con 0x7ff6dc0c2d90 2024-03-18T16:32:05.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.026+0000 7ff6d3400640 1 -- 172.21.15.102:0/3704085503 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff6c804a070 con 0x7ff6dc0c2d90 2024-03-18T16:32:05.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.027+0000 7ff6d3400640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6bc06bc50 0x7ff6bc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:05.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.028+0000 7ff6db400640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6bc06bc50 0x7ff6bc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:05.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.029+0000 7ff6db400640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6bc06bc50 0x7ff6bc06e110 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc006fd0 tx=0x7ff6cc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.052+0000 7ff6d3400640 1 -- 172.21.15.102:0/3704085503 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff6c80799d0 con 0x7ff6dc0c2d90 2024-03-18T16:32:05.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.053+0000 7ff6d3400640 1 -- 172.21.15.102:0/3704085503 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff6c8033030 con 0x7ff6dc0c2d90 2024-03-18T16:32:05.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.365+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6dc157580 con 0x7ff6dc0c2d90 2024-03-18T16:32:05.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.417+0000 7ff6d3400640 1 -- 172.21.15.102:0/3704085503 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff6c807b930 con 0x7ff6dc0c2d90 2024-03-18T16:32:05.418 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:05.419 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:05.419 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6bc06bc50 msgr2=0x7ff6bc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6bc06bc50 0x7ff6bc06e110 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc006fd0 tx=0x7ff6cc008040 comp rx=0 tx=0).stop 2024-03-18T16:32:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6dc0c2d90 msgr2=0x7ff6dc154e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6dc0c2d90 0x7ff6dc154e70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0c5f00 tx=0x7ff6c8004730 comp rx=0 tx=0).stop 2024-03-18T16:32:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 shutdown_connections 2024-03-18T16:32:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6bc06bc50 0x7ff6bc06e110 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff6dc0c7f30 0x7ff6dc1373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:05.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6dc0c2d90 0x7ff6dc154e70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 --2- 172.21.15.102:0/3704085503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff6dc072810 0x7ff6dc1491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 >> 172.21.15.102:0/3704085503 conn(0x7ff6dc0cb890 msgr2=0x7ff6dc0c8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 shutdown_connections 2024-03-18T16:32:05.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:05.420+0000 7ff6e1000640 1 -- 172.21.15.102:0/3704085503 wait complete. 2024-03-18T16:32:06.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:06.050 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:07.051 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:07.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.226+0000 7f88adc00640 1 Processor -- start 2024-03-18T16:32:07.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.226+0000 7f88adc00640 1 -- start start 2024-03-18T16:32:07.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.228+0000 7f88adc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a806b470 0x7f88a815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:07.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.228+0000 7f88adc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88a8154150 0x7f88a81545c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:07.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.228+0000 7f88adc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a806bf60 con 0x7f88a806de50 2024-03-18T16:32:07.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.228+0000 7f88adc00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f88a80ca0a0 con 0x7f88a8154150 2024-03-18T16:32:07.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.228+0000 7f88adc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f88a80c94b0 con 0x7f88a806b470 2024-03-18T16:32:07.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.229+0000 7f88a7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a806b470 0x7f88a815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:07.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.229+0000 7f88a7400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f88a806de50 0x7f88a806b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:49724/0 (socket says 172.21.15.102:49724) 2024-03-18T16:32:07.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.229+0000 7f88a7400640 1 -- 172.21.15.102:0/2026424146 learned_addr learned my addr 172.21.15.102:0/2026424146 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:07.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.229+0000 7f88ad200640 1 --2- 172.21.15.102:0/2026424146 >> v2:172.21.15.154:3300/0 conn(0x7f88a8154150 0x7f88a81545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:07.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.231+0000 7f88a7e00640 1 -- 172.21.15.102:0/2026424146 >> v2:172.21.15.154:3300/0 conn(0x7f88a8154150 msgr2=0x7f88a81545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:07.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.231+0000 7f88a7e00640 1 --2- 172.21.15.102:0/2026424146 >> v2:172.21.15.154:3300/0 conn(0x7f88a8154150 0x7f88a81545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.231+0000 7f88a7e00640 1 -- 172.21.15.102:0/2026424146 >> v1:172.21.15.102:6789/0 conn(0x7f88a806de50 legacy=0x7f88a806b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:07.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.231+0000 7f88a7e00640 1 -- 172.21.15.102:0/2026424146 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a80c9770 con 0x7f88a806b470 2024-03-18T16:32:07.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.233+0000 7f88a7e00640 1 --2- 172.21.15.102:0/2026424146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a806b470 0x7f88a815ad60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f88980097a0 tx=0x7f889802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3eeefd567cb9425b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:07.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.238+0000 7f88a6a00640 1 -- 172.21.15.102:0/2026424146 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8898035020 con 0x7f88a806b470 2024-03-18T16:32:07.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.238+0000 7f88a6a00640 1 -- 172.21.15.102:0/2026424146 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8898031810 con 0x7f88a806b470 2024-03-18T16:32:07.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.238+0000 7f88a6a00640 1 -- 172.21.15.102:0/2026424146 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8898031be0 con 0x7f88a806b470 2024-03-18T16:32:07.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.238+0000 7f88adc00640 1 -- 172.21.15.102:0/2026424146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a806b470 msgr2=0x7f88a815ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:07.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.238+0000 7f88adc00640 1 --2- 172.21.15.102:0/2026424146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a806b470 0x7f88a815ad60 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f88980097a0 tx=0x7f889802b8b0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.239+0000 7f88adc00640 1 -- 172.21.15.102:0/2026424146 shutdown_connections 2024-03-18T16:32:07.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.239+0000 7f88adc00640 1 --2- 172.21.15.102:0/2026424146 >> v2:172.21.15.154:3300/0 conn(0x7f88a8154150 0x7f88a81545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.239+0000 7f88adc00640 1 --2- 172.21.15.102:0/2026424146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a806b470 0x7f88a815ad60 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.239+0000 7f88adc00640 1 -- 172.21.15.102:0/2026424146 >> 172.21.15.102:0/2026424146 conn(0x7f88a80727e0 msgr2=0x7f88a8072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:07.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.239+0000 7f88adc00640 1 -- 172.21.15.102:0/2026424146 shutdown_connections 2024-03-18T16:32:07.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.239+0000 7f88adc00640 1 -- 172.21.15.102:0/2026424146 wait complete. 2024-03-18T16:32:07.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.240+0000 7f88adc00640 1 Processor -- start 2024-03-18T16:32:07.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.241+0000 7f88adc00640 1 -- start start 2024-03-18T16:32:07.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.241+0000 7f88adc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88a806b470 0x7f88a814c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:07.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.241+0000 7f88adc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88a806de50 0x7f88a8156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:07.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.241+0000 7f88adc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a8154150 0x7f88a8156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:07.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.241+0000 7f88adc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f88a80c5190 con 0x7f88a806b470 2024-03-18T16:32:07.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.241+0000 7f88adc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f88a80c9d30 con 0x7f88a806de50 2024-03-18T16:32:07.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.241+0000 7f88adc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f88a80c5310 con 0x7f88a8154150 2024-03-18T16:32:07.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.242+0000 7f88ad200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a8154150 0x7f88a8156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:07.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.242+0000 7f88a7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88a806de50 0x7f88a8156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:07.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.242+0000 7f88a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88a806b470 0x7f88a814c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:07.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.242+0000 7f88ad200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a8154150 0x7f88a8156570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:42754/0 (socket says 172.21.15.102:42754) 2024-03-18T16:32:07.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.242+0000 7f88ad200640 1 -- 172.21.15.102:0/1355047021 learned_addr learned my addr 172.21.15.102:0/1355047021 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:07.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.246+0000 7f88ad200640 1 -- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88a806de50 msgr2=0x7f88a8156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:07.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.246+0000 7f88ad200640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88a806de50 0x7f88a8156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.246+0000 7f88ad200640 1 -- 172.21.15.102:0/1355047021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88a806b470 msgr2=0x7f88a814c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:07.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.246+0000 7f88ad200640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88a806b470 0x7f88a814c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.246+0000 7f88ad200640 1 -- 172.21.15.102:0/1355047021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a8154e70 con 0x7f88a8154150 2024-03-18T16:32:07.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.249+0000 7f88ad200640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a8154150 0x7f88a8156570 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009d30 tx=0x7f88a000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:07.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.251+0000 7f889fe00640 1 -- 172.21.15.102:0/1355047021 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0018020 con 0x7f88a8154150 2024-03-18T16:32:07.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.251+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a81f0b60 con 0x7f88a8154150 2024-03-18T16:32:07.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.252+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a81553f0 con 0x7f88a8154150 2024-03-18T16:32:07.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.253+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a8074fd0 con 0x7f88a8154150 2024-03-18T16:32:07.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.253+0000 7f889fe00640 1 -- 172.21.15.102:0/1355047021 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a0014480 con 0x7f88a8154150 2024-03-18T16:32:07.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.253+0000 7f889fe00640 1 -- 172.21.15.102:0/1355047021 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0014830 con 0x7f88a8154150 2024-03-18T16:32:07.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.527+0000 7f889fe00640 1 -- 172.21.15.102:0/1355047021 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f88a0005d10 con 0x7f88a8154150 2024-03-18T16:32:07.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.530+0000 7f889fe00640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f888806bbd0 0x7f888806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:07.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.530+0000 7f889fe00640 1 -- 172.21.15.102:0/1355047021 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f88a008c2a0 con 0x7f88a8154150 2024-03-18T16:32:07.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.530+0000 7f88a7e00640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f888806bbd0 0x7f888806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:07.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.531+0000 7f88a7e00640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f888806bbd0 0x7f888806e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8898036000 tx=0x7f8898008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:07.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.542+0000 7f889fe00640 1 -- 172.21.15.102:0/1355047021 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f88a0022030 con 0x7f88a8154150 2024-03-18T16:32:07.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.789+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88a8068db0 con 0x7f88a8154150 2024-03-18T16:32:07.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.878+0000 7f889fe00640 1 -- 172.21.15.102:0/1355047021 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f88a0010040 con 0x7f88a8154150 2024-03-18T16:32:07.880 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:07.880 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:07.880 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f888806bbd0 msgr2=0x7f888806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f888806bbd0 0x7f888806e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8898036000 tx=0x7f8898008030 comp rx=0 tx=0).stop 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a8154150 msgr2=0x7f88a8156570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a8154150 0x7f88a8156570 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f88a0009d30 tx=0x7f88a000c490 comp rx=0 tx=0).stop 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 shutdown_connections 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f888806bbd0 0x7f888806e090 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f88a8154150 0x7f88a8156570 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88a806de50 0x7f88a8156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 --2- 172.21.15.102:0/1355047021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88a806b470 0x7f88a814c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 >> 172.21.15.102:0/1355047021 conn(0x7f88a80727e0 msgr2=0x7f88a80c65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 shutdown_connections 2024-03-18T16:32:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:07.881+0000 7f88adc00640 1 -- 172.21.15.102:0/1355047021 wait complete. 2024-03-18T16:32:08.502 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:08.502 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:09.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:09.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.674+0000 7f3d5ae00640 1 Processor -- start 2024-03-18T16:32:09.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.674+0000 7f3d5ae00640 1 -- start start 2024-03-18T16:32:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.675+0000 7f3d5ae00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d5415c3a0 0x7f3d54163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.675+0000 7f3d5ae00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d540c72d0 con 0x7f3d54141650 2024-03-18T16:32:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.675+0000 7f3d5ae00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d54158840 con 0x7f3d54158420 2024-03-18T16:32:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.675+0000 7f3d5ae00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3d54159370 con 0x7f3d5415c3a0 2024-03-18T16:32:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.676+0000 7f3d59000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3d54158420 0x7f3d540c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:38044/0 (socket says 172.21.15.102:38044) 2024-03-18T16:32:09.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.676+0000 7f3d59000640 1 -- 172.21.15.102:0/2619821786 learned_addr learned my addr 172.21.15.102:0/2619821786 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:09.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.678+0000 7f3d5a400640 1 --2- 172.21.15.102:0/2619821786 >> v2:172.21.15.154:3300/0 conn(0x7f3d5415c3a0 0x7f3d54163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:09.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.682+0000 7f3d5a400640 1 -- 172.21.15.102:0/2619821786 >> v1:172.21.15.154:6789/0 conn(0x7f3d54158420 legacy=0x7f3d540c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:09.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.682+0000 7f3d5a400640 1 -- 172.21.15.102:0/2619821786 >> v1:172.21.15.102:6789/0 conn(0x7f3d54141650 legacy=0x7f3d540c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:09.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.682+0000 7f3d5a400640 1 -- 172.21.15.102:0/2619821786 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d54158ad0 con 0x7f3d5415c3a0 2024-03-18T16:32:09.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.688+0000 7f3d5a400640 1 --2- 172.21.15.102:0/2619821786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d5415c3a0 0x7f3d54163cd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009d90 tx=0x7f3d4c02bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39e8cd235e453576 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:09.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.695+0000 7f3d53e00640 1 -- 172.21.15.102:0/2619821786 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c039070 con 0x7f3d5415c3a0 2024-03-18T16:32:09.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.695+0000 7f3d53e00640 1 -- 172.21.15.102:0/2619821786 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d4c02ccf0 con 0x7f3d5415c3a0 2024-03-18T16:32:09.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.695+0000 7f3d53e00640 1 -- 172.21.15.102:0/2619821786 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c034660 con 0x7f3d5415c3a0 2024-03-18T16:32:09.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.696+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2619821786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d5415c3a0 msgr2=0x7f3d54163cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:09.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.696+0000 7f3d5ae00640 1 --2- 172.21.15.102:0/2619821786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d5415c3a0 0x7f3d54163cd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009d90 tx=0x7f3d4c02bf40 comp rx=0 tx=0).stop 2024-03-18T16:32:09.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.696+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2619821786 shutdown_connections 2024-03-18T16:32:09.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.696+0000 7f3d5ae00640 1 --2- 172.21.15.102:0/2619821786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d5415c3a0 0x7f3d54163cd0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:09.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.696+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2619821786 >> 172.21.15.102:0/2619821786 conn(0x7f3d540cb890 msgr2=0x7f3d540cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:09.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.696+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2619821786 shutdown_connections 2024-03-18T16:32:09.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.696+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2619821786 wait complete. 2024-03-18T16:32:09.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.698+0000 7f3d5ae00640 1 Processor -- start 2024-03-18T16:32:09.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.698+0000 7f3d5ae00640 1 -- start start 2024-03-18T16:32:09.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.699+0000 7f3d5ae00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d54141650 0x7f3d54153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:09.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.699+0000 7f3d5ae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d54158420 0x7f3d540ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:09.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.699+0000 7f3d5ae00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d5415c3a0 0x7f3d540cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:09.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.699+0000 7f3d5ae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3d540c63c0 con 0x7f3d54158420 2024-03-18T16:32:09.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.699+0000 7f3d5ae00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d54159090 con 0x7f3d54141650 2024-03-18T16:32:09.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.699+0000 7f3d5ae00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3d540c66a0 con 0x7f3d5415c3a0 2024-03-18T16:32:09.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.700+0000 7f3d5a400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d5415c3a0 0x7f3d540cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:09.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.700+0000 7f3d59a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d54141650 0x7f3d54153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:09.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.700+0000 7f3d59000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d54158420 0x7f3d540ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:09.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.700+0000 7f3d59a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d54141650 0x7f3d54153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56660/0 (socket says 172.21.15.102:56660) 2024-03-18T16:32:09.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.700+0000 7f3d59a00640 1 -- 172.21.15.102:0/2561704792 learned_addr learned my addr 172.21.15.102:0/2561704792 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:09.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.704+0000 7f3d59a00640 1 -- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d5415c3a0 msgr2=0x7f3d540cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:09.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.704+0000 7f3d59a00640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d5415c3a0 0x7f3d540cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:09.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.704+0000 7f3d59a00640 1 -- 172.21.15.102:0/2561704792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d54158420 msgr2=0x7f3d540ca6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:09.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.704+0000 7f3d59a00640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d54158420 0x7f3d540ca6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:09.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.704+0000 7f3d59a00640 1 -- 172.21.15.102:0/2561704792 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d54133980 con 0x7f3d54141650 2024-03-18T16:32:09.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.707+0000 7f3d59a00640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d54141650 0x7f3d54153140 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800c430 tx=0x7f3d4800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:09.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.711+0000 7f3d51600640 1 -- 172.21.15.102:0/2561704792 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d480052d0 con 0x7f3d54141650 2024-03-18T16:32:09.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.711+0000 7f3d51600640 1 -- 172.21.15.102:0/2561704792 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d48005490 con 0x7f3d54141650 2024-03-18T16:32:09.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.711+0000 7f3d51600640 1 -- 172.21.15.102:0/2561704792 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48007260 con 0x7f3d54141650 2024-03-18T16:32:09.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.711+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d54151f50 con 0x7f3d54141650 2024-03-18T16:32:09.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.711+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d54131310 con 0x7f3d54141650 2024-03-18T16:32:09.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.713+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d54141200 con 0x7f3d54141650 2024-03-18T16:32:09.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.732+0000 7f3d51600640 1 -- 172.21.15.102:0/2561704792 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3d48007400 con 0x7f3d54141650 2024-03-18T16:32:09.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.733+0000 7f3d51600640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d3406bc20 0x7f3d3406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:09.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.733+0000 7f3d59000640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d3406bc20 0x7f3d3406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:09.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.734+0000 7f3d59000640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d3406bc20 0x7f3d3406e0e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3d54140260 tx=0x7f3d40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:09.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.736+0000 7f3d51600640 1 -- 172.21.15.102:0/2561704792 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3d48056920 con 0x7f3d54141650 2024-03-18T16:32:09.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:09.993+0000 7f3d51600640 1 -- 172.21.15.102:0/2561704792 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3d54141200 con 0x7f3d54141650 2024-03-18T16:32:10.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.334+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d540c74a0 con 0x7f3d54141650 2024-03-18T16:32:10.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.348+0000 7f3d51600640 1 -- 172.21.15.102:0/2561704792 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3d540c74a0 con 0x7f3d54141650 2024-03-18T16:32:10.350 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:10.350 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:10.350 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:10.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d3406bc20 msgr2=0x7f3d3406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:10.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d3406bc20 0x7f3d3406e0e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3d54140260 tx=0x7f3d40009040 comp rx=0 tx=0).stop 2024-03-18T16:32:10.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d54141650 msgr2=0x7f3d54153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d54141650 0x7f3d54153140 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800c430 tx=0x7f3d4800c8f0 comp rx=0 tx=0).stop 2024-03-18T16:32:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 shutdown_connections 2024-03-18T16:32:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3d3406bc20 0x7f3d3406e0e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d5415c3a0 0x7f3d540cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d54158420 0x7f3d540ca6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 --2- 172.21.15.102:0/2561704792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d54141650 0x7f3d54153140 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.350+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 >> 172.21.15.102:0/2561704792 conn(0x7f3d540cb890 msgr2=0x7f3d540cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.351+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 shutdown_connections 2024-03-18T16:32:10.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:10.351+0000 7f3d5ae00640 1 -- 172.21.15.102:0/2561704792 wait complete. 2024-03-18T16:32:10.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:10.980 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:11.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:12.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.159+0000 7f7af6600640 1 Processor -- start 2024-03-18T16:32:12.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.160+0000 7f7af6600640 1 -- start start 2024-03-18T16:32:12.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.161+0000 7f7af6600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7af006f6b0 0x7f7af00c8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.162+0000 7f7af6600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7af00c8970 0x7f7af006fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.162+0000 7f7af6600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 0x7f7af015d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.162+0000 7f7af6600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7af00c73a0 con 0x7f7af006f6b0 2024-03-18T16:32:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.162+0000 7f7af6600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7af00c7ca0 con 0x7f7af00c8970 2024-03-18T16:32:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.162+0000 7f7af6600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7af00c49c0 con 0x7f7af006ffd0 2024-03-18T16:32:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.162+0000 7f7aef400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7af006f6b0 0x7f7af00c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.163+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 0x7f7af015d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:12.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.163+0000 7f7aeea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7af00c8970 0x7f7af006fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:12.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.163+0000 7f7aef400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7af006f6b0 0x7f7af00c8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34474/0 (socket says 172.21.15.102:34474) 2024-03-18T16:32:12.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.163+0000 7f7aef400640 1 -- 172.21.15.102:0/2480439494 learned_addr learned my addr 172.21.15.102:0/2480439494 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:12.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.165+0000 7f7aeea00640 1 -- 172.21.15.102:0/2480439494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 msgr2=0x7f7af015d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:12.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.165+0000 7f7aeea00640 1 --2- 172.21.15.102:0/2480439494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 0x7f7af015d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.165+0000 7f7aeea00640 1 -- 172.21.15.102:0/2480439494 >> v2:172.21.15.102:3300/0 conn(0x7f7af006f6b0 msgr2=0x7f7af00c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:12.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.165+0000 7f7aeea00640 1 --2- 172.21.15.102:0/2480439494 >> v2:172.21.15.102:3300/0 conn(0x7f7af006f6b0 0x7f7af00c8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.165+0000 7f7aeea00640 1 -- 172.21.15.102:0/2480439494 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af0070400 con 0x7f7af00c8970 2024-03-18T16:32:12.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.168+0000 7f7aeea00640 1 --2- 172.21.15.102:0/2480439494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 0x7f7af006fa90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7af00ca030 tx=0x7f7adc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ecbfff9aaf708a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:12.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.172+0000 7f7aee000640 1 -- 172.21.15.102:0/2480439494 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0315a0 con 0x7f7af00c8970 2024-03-18T16:32:12.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.172+0000 7f7aee000640 1 -- 172.21.15.102:0/2480439494 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7adc037070 con 0x7f7af00c8970 2024-03-18T16:32:12.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.172+0000 7f7aee000640 1 -- 172.21.15.102:0/2480439494 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc031960 con 0x7f7af00c8970 2024-03-18T16:32:12.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.173+0000 7f7af6600640 1 -- 172.21.15.102:0/2480439494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 msgr2=0x7f7af006fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:12.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.173+0000 7f7af6600640 1 --2- 172.21.15.102:0/2480439494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 0x7f7af006fa90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7af00ca030 tx=0x7f7adc02b840 comp rx=0 tx=0).stop 2024-03-18T16:32:12.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.173+0000 7f7af6600640 1 -- 172.21.15.102:0/2480439494 shutdown_connections 2024-03-18T16:32:12.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.173+0000 7f7af6600640 1 --2- 172.21.15.102:0/2480439494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 0x7f7af015d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.173+0000 7f7af6600640 1 --2- 172.21.15.102:0/2480439494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 0x7f7af006fa90 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.173+0000 7f7af6600640 1 --2- 172.21.15.102:0/2480439494 >> v2:172.21.15.102:3300/0 conn(0x7f7af006f6b0 0x7f7af00c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.173+0000 7f7af6600640 1 -- 172.21.15.102:0/2480439494 >> 172.21.15.102:0/2480439494 conn(0x7f7af00c8f70 msgr2=0x7f7af00c9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:12.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.173+0000 7f7af6600640 1 -- 172.21.15.102:0/2480439494 shutdown_connections 2024-03-18T16:32:12.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.173+0000 7f7af6600640 1 -- 172.21.15.102:0/2480439494 wait complete. 2024-03-18T16:32:12.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.176+0000 7f7af6600640 1 Processor -- start 2024-03-18T16:32:12.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.176+0000 7f7af6600640 1 -- start start 2024-03-18T16:32:12.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.177+0000 7f7af6600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af006f6b0 0x7f7af01f0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:12.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.177+0000 7f7af6600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 0x7f7af014dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:12.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.177+0000 7f7af6600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 0x7f7af014aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:12.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.177+0000 7f7af6600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7af00c7f10 con 0x7f7af006f6b0 2024-03-18T16:32:12.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.177+0000 7f7af6600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7af00c75f0 con 0x7f7af00c8970 2024-03-18T16:32:12.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.177+0000 7f7af6600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7af0154810 con 0x7f7af006ffd0 2024-03-18T16:32:12.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.178+0000 7f7aef400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af006f6b0 0x7f7af01f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:12.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.178+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 0x7f7af014aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:12.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.178+0000 7f7aeea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 0x7f7af014dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:12.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.178+0000 7f7aef400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af006f6b0 0x7f7af01f0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34484/0 (socket says 172.21.15.102:34484) 2024-03-18T16:32:12.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.178+0000 7f7aef400640 1 -- 172.21.15.102:0/1155227111 learned_addr learned my addr 172.21.15.102:0/1155227111 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:12.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.182+0000 7f7aefe00640 1 -- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 msgr2=0x7f7af014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:12.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.182+0000 7f7aefe00640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 0x7f7af014dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.182+0000 7f7aefe00640 1 -- 172.21.15.102:0/1155227111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af006f6b0 msgr2=0x7f7af01f0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:12.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.182+0000 7f7aefe00640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af006f6b0 0x7f7af01f0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.182+0000 7f7aefe00640 1 -- 172.21.15.102:0/1155227111 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af01e2670 con 0x7f7af00c8970 2024-03-18T16:32:12.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.182+0000 7f7aef400640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af006f6b0 0x7f7af01f0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:32:12.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.184+0000 7f7aefe00640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 0x7f7af014aa60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8009d90 tx=0x7f7ae800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:12.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.187+0000 7f7ae7400640 1 -- 172.21.15.102:0/1155227111 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8018020 con 0x7f7af00c8970 2024-03-18T16:32:12.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.187+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af01e2950 con 0x7f7af00c8970 2024-03-18T16:32:12.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.187+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af01f4170 con 0x7f7af00c8970 2024-03-18T16:32:12.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.188+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af00c83a0 con 0x7f7af00c8970 2024-03-18T16:32:12.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.190+0000 7f7ae7400640 1 -- 172.21.15.102:0/1155227111 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae8014480 con 0x7f7af00c8970 2024-03-18T16:32:12.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.190+0000 7f7ae7400640 1 -- 172.21.15.102:0/1155227111 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8014830 con 0x7f7af00c8970 2024-03-18T16:32:12.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.465+0000 7f7ae7400640 1 -- 172.21.15.102:0/1155227111 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7ae8005d10 con 0x7f7af00c8970 2024-03-18T16:32:12.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.467+0000 7f7ae7400640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bbd0 0x7f7ad006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:12.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.468+0000 7f7aef400640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bbd0 0x7f7ad006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:12.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.468+0000 7f7ae7400640 1 -- 172.21.15.102:0/1155227111 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7ae80581c0 con 0x7f7af00c8970 2024-03-18T16:32:12.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.468+0000 7f7aef400640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bbd0 0x7f7ad006e090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0011310 tx=0x7f7ae0009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:12.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.479+0000 7f7ae7400640 1 -- 172.21.15.102:0/1155227111 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7af00c83a0 con 0x7f7af00c8970 2024-03-18T16:32:12.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.748+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7af00c6b60 con 0x7f7af00c8970 2024-03-18T16:32:12.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.761+0000 7f7ae7400640 1 -- 172.21.15.102:0/1155227111 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7af00c6b60 con 0x7f7af00c8970 2024-03-18T16:32:12.763 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:12.763 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:12.763 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bbd0 msgr2=0x7f7ad006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bbd0 0x7f7ad006e090 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0011310 tx=0x7f7ae0009240 comp rx=0 tx=0).stop 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 msgr2=0x7f7af014aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 0x7f7af014aa60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8009d90 tx=0x7f7ae800c490 comp rx=0 tx=0).stop 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 shutdown_connections 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7ad006bbd0 0x7f7ad006e090 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7af00c8970 0x7f7af014aa60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7af006ffd0 0x7f7af014dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 --2- 172.21.15.102:0/1155227111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7af006f6b0 0x7f7af01f0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 >> 172.21.15.102:0/1155227111 conn(0x7f7af00c8f70 msgr2=0x7f7af0153270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 shutdown_connections 2024-03-18T16:32:12.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:12.764+0000 7f7af6600640 1 -- 172.21.15.102:0/1155227111 wait complete. 2024-03-18T16:32:13.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:13.385 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:14.386 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:14.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.555+0000 7fbe2be00640 1 Processor -- start 2024-03-18T16:32:14.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.556+0000 7fbe2be00640 1 -- start start 2024-03-18T16:32:14.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.557+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:14.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.557+0000 7fbe2be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe2c0c72d0 con 0x7fbe2c0c4fa0 2024-03-18T16:32:14.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.557+0000 7fbe2be00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe2c158840 con 0x7fbe2c141650 2024-03-18T16:32:14.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.557+0000 7fbe2be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2c15c8e0 con 0x7fbe2c158420 2024-03-18T16:32:14.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.557+0000 7fbe2a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:14.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.558+0000 7fbe2aa00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fbe2c141650 0x7fbe2c0c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:35766/0 (socket says 172.21.15.102:35766) 2024-03-18T16:32:14.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.558+0000 7fbe2aa00640 1 -- 172.21.15.102:0/953396469 learned_addr learned my addr 172.21.15.102:0/953396469 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:14.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.560+0000 7fbe2a000640 1 -- 172.21.15.102:0/953396469 >> v1:172.21.15.154:6789/0 conn(0x7fbe2c141650 legacy=0x7fbe2c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:14.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.560+0000 7fbe2a000640 1 -- 172.21.15.102:0/953396469 >> v1:172.21.15.102:6789/0 conn(0x7fbe2c0c4fa0 legacy=0x7fbe2c162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:14.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.560+0000 7fbe2a000640 1 -- 172.21.15.102:0/953396469 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c158ad0 con 0x7fbe2c158420 2024-03-18T16:32:14.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.563+0000 7fbe2a000640 1 --2- 172.21.15.102:0/953396469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c15c3a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbe180097d0 tx=0x7fbe1802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fa1706c8133d006 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:14.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.566+0000 7fbe29600640 1 -- 172.21.15.102:0/953396469 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe18034070 con 0x7fbe2c158420 2024-03-18T16:32:14.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.566+0000 7fbe29600640 1 -- 172.21.15.102:0/953396469 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe18030800 con 0x7fbe2c158420 2024-03-18T16:32:14.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.566+0000 7fbe29600640 1 -- 172.21.15.102:0/953396469 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe18030bd0 con 0x7fbe2c158420 2024-03-18T16:32:14.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.567+0000 7fbe2be00640 1 -- 172.21.15.102:0/953396469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 msgr2=0x7fbe2c15c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:14.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.567+0000 7fbe2be00640 1 --2- 172.21.15.102:0/953396469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c15c3a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fbe180097d0 tx=0x7fbe1802b940 comp rx=0 tx=0).stop 2024-03-18T16:32:14.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.567+0000 7fbe2be00640 1 -- 172.21.15.102:0/953396469 shutdown_connections 2024-03-18T16:32:14.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.567+0000 7fbe2be00640 1 --2- 172.21.15.102:0/953396469 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c15c3a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:14.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.567+0000 7fbe2be00640 1 -- 172.21.15.102:0/953396469 >> 172.21.15.102:0/953396469 conn(0x7fbe2c0cb890 msgr2=0x7fbe2c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:14.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.567+0000 7fbe2be00640 1 -- 172.21.15.102:0/953396469 shutdown_connections 2024-03-18T16:32:14.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.567+0000 7fbe2be00640 1 -- 172.21.15.102:0/953396469 wait complete. 2024-03-18T16:32:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.570+0000 7fbe2be00640 1 Processor -- start 2024-03-18T16:32:14.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.570+0000 7fbe2be00640 1 -- start start 2024-03-18T16:32:14.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.571+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe2c0c4fa0 0x7fbe2c1f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:14.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.571+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe2c141650 0x7fbe2c0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:14.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.571+0000 7fbe2be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c13ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:14.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.571+0000 7fbe2be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c159090 con 0x7fbe2c141650 2024-03-18T16:32:14.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.571+0000 7fbe2be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2c159370 con 0x7fbe2c0c4fa0 2024-03-18T16:32:14.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.571+0000 7fbe2be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2c0c63c0 con 0x7fbe2c158420 2024-03-18T16:32:14.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.572+0000 7fbe2b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c13ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:14.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.572+0000 7fbe2aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe2c0c4fa0 0x7fbe2c1f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:14.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.572+0000 7fbe2b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c13ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58296/0 (socket says 172.21.15.102:58296) 2024-03-18T16:32:14.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.572+0000 7fbe2b400640 1 -- 172.21.15.102:0/1911823069 learned_addr learned my addr 172.21.15.102:0/1911823069 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:14.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.572+0000 7fbe2a000640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe2c141650 0x7fbe2c0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:14.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.576+0000 7fbe2b400640 1 -- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe2c0c4fa0 msgr2=0x7fbe2c1f03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:14.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.576+0000 7fbe2b400640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe2c0c4fa0 0x7fbe2c1f03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:14.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.576+0000 7fbe2b400640 1 -- 172.21.15.102:0/1911823069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe2c141650 msgr2=0x7fbe2c0caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:14.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.576+0000 7fbe2b400640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe2c141650 0x7fbe2c0caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:14.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.576+0000 7fbe2b400640 1 -- 172.21.15.102:0/1911823069 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2c1f0930 con 0x7fbe2c158420 2024-03-18T16:32:14.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.578+0000 7fbe2b400640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c13ddb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c7a0 tx=0x7fbe2400cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:14.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.582+0000 7fbe22a00640 1 -- 172.21.15.102:0/1911823069 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24003b70 con 0x7fbe2c158420 2024-03-18T16:32:14.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.582+0000 7fbe22a00640 1 -- 172.21.15.102:0/1911823069 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe24003d30 con 0x7fbe2c158420 2024-03-18T16:32:14.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.582+0000 7fbe22a00640 1 -- 172.21.15.102:0/1911823069 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2400aa90 con 0x7fbe2c158420 2024-03-18T16:32:14.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.582+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2c1ec7f0 con 0x7fbe2c158420 2024-03-18T16:32:14.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.582+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe2c1ecdb0 con 0x7fbe2c158420 2024-03-18T16:32:14.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.583+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe2c0c74a0 con 0x7fbe2c158420 2024-03-18T16:32:14.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.604+0000 7fbe22a00640 1 -- 172.21.15.102:0/1911823069 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbe2400ac30 con 0x7fbe2c158420 2024-03-18T16:32:14.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.605+0000 7fbe22a00640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbe0c06bc00 0x7fbe0c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:14.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.606+0000 7fbe2aa00640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbe0c06bc00 0x7fbe0c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:14.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.606+0000 7fbe2aa00640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbe0c06bc00 0x7fbe0c06e0c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c005880 tx=0x7fbe1c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:14.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.773+0000 7fbe22a00640 1 -- 172.21.15.102:0/1911823069 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fbe24057840 con 0x7fbe2c158420 2024-03-18T16:32:14.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:14.860+0000 7fbe22a00640 1 -- 172.21.15.102:0/1911823069 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbe2c0c74a0 con 0x7fbe2c158420 2024-03-18T16:32:15.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.109+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe2c074fd0 con 0x7fbe2c158420 2024-03-18T16:32:15.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.177+0000 7fbe22a00640 1 -- 172.21.15.102:0/1911823069 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fbe240597a0 con 0x7fbe2c158420 2024-03-18T16:32:15.179 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:15.179 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:15.179 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbe0c06bc00 msgr2=0x7fbe0c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbe0c06bc00 0x7fbe0c06e0c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c005880 tx=0x7fbe1c009040 comp rx=0 tx=0).stop 2024-03-18T16:32:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 msgr2=0x7fbe2c13ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c13ddb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbe2400c7a0 tx=0x7fbe2400cc60 comp rx=0 tx=0).stop 2024-03-18T16:32:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 shutdown_connections 2024-03-18T16:32:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbe0c06bc00 0x7fbe0c06e0c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbe2c158420 0x7fbe2c13ddb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:15.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe2c141650 0x7fbe2c0caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:15.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 --2- 172.21.15.102:0/1911823069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe2c0c4fa0 0x7fbe2c1f03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:15.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 >> 172.21.15.102:0/1911823069 conn(0x7fbe2c0cb890 msgr2=0x7fbe2c071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:15.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 shutdown_connections 2024-03-18T16:32:15.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:15.179+0000 7fbe2be00640 1 -- 172.21.15.102:0/1911823069 wait complete. 2024-03-18T16:32:15.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:15.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:16.802 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:16.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.972+0000 7fa0a9000640 1 Processor -- start 2024-03-18T16:32:16.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.973+0000 7fa0a9000640 1 -- start start 2024-03-18T16:32:16.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.974+0000 7fa0a9000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406b470 0x7fa0a415ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:16.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.974+0000 7fa0a9000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0a4154150 0x7fa0a41545c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:16.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.974+0000 7fa0a9000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a406bf60 con 0x7fa0a406de50 2024-03-18T16:32:16.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.974+0000 7fa0a9000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa0a40ca0a0 con 0x7fa0a4154150 2024-03-18T16:32:16.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.974+0000 7fa0a9000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a40c94b0 con 0x7fa0a406b470 2024-03-18T16:32:16.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.975+0000 7fa0a3400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406b470 0x7fa0a415ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:16.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.975+0000 7fa0a3400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406b470 0x7fa0a415ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58302/0 (socket says 172.21.15.102:58302) 2024-03-18T16:32:16.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.976+0000 7fa0a3400640 1 -- 172.21.15.102:0/302889890 learned_addr learned my addr 172.21.15.102:0/302889890 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:16.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.976+0000 7fa0a3e00640 1 --2- 172.21.15.102:0/302889890 >> v2:172.21.15.154:3300/0 conn(0x7fa0a4154150 0x7fa0a41545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:16.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.978+0000 7fa0a3400640 1 -- 172.21.15.102:0/302889890 >> v2:172.21.15.154:3300/0 conn(0x7fa0a4154150 msgr2=0x7fa0a41545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.978+0000 7fa0a3400640 1 --2- 172.21.15.102:0/302889890 >> v2:172.21.15.154:3300/0 conn(0x7fa0a4154150 0x7fa0a41545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.978+0000 7fa0a3400640 1 -- 172.21.15.102:0/302889890 >> v1:172.21.15.102:6789/0 conn(0x7fa0a406de50 legacy=0x7fa0a406b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:16.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.978+0000 7fa0a3400640 1 -- 172.21.15.102:0/302889890 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a40c9770 con 0x7fa0a406b470 2024-03-18T16:32:16.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.981+0000 7fa0a3400640 1 --2- 172.21.15.102:0/302889890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406b470 0x7fa0a415ad60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa094009870 tx=0x7fa09402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be61e48ca043083c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:16.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.983+0000 7fa0a2000640 1 -- 172.21.15.102:0/302889890 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa094035020 con 0x7fa0a406b470 2024-03-18T16:32:16.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.986+0000 7fa0a2000640 1 -- 172.21.15.102:0/302889890 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa094031640 con 0x7fa0a406b470 2024-03-18T16:32:16.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.986+0000 7fa0a2000640 1 -- 172.21.15.102:0/302889890 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09402d040 con 0x7fa0a406b470 2024-03-18T16:32:16.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.987+0000 7fa0a9000640 1 -- 172.21.15.102:0/302889890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406b470 msgr2=0x7fa0a415ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:16.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.987+0000 7fa0a9000640 1 --2- 172.21.15.102:0/302889890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406b470 0x7fa0a415ad60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa094009870 tx=0x7fa09402b8f0 comp rx=0 tx=0).stop 2024-03-18T16:32:16.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.987+0000 7fa0a9000640 1 -- 172.21.15.102:0/302889890 shutdown_connections 2024-03-18T16:32:16.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.987+0000 7fa0a9000640 1 --2- 172.21.15.102:0/302889890 >> v2:172.21.15.154:3300/0 conn(0x7fa0a4154150 0x7fa0a41545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:16.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.987+0000 7fa0a9000640 1 --2- 172.21.15.102:0/302889890 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406b470 0x7fa0a415ad60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:16.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.987+0000 7fa0a9000640 1 -- 172.21.15.102:0/302889890 >> 172.21.15.102:0/302889890 conn(0x7fa0a40727e0 msgr2=0x7fa0a4072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:16.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.987+0000 7fa0a9000640 1 -- 172.21.15.102:0/302889890 shutdown_connections 2024-03-18T16:32:16.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.987+0000 7fa0a9000640 1 -- 172.21.15.102:0/302889890 wait complete. 2024-03-18T16:32:16.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.989+0000 7fa0a9000640 1 Processor -- start 2024-03-18T16:32:16.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.990+0000 7fa0a9000640 1 -- start start 2024-03-18T16:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.991+0000 7fa0a9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a406b470 0x7fa0a41e3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.991+0000 7fa0a9000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406de50 0x7fa0a414f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.991+0000 7fa0a9000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0a4154150 0x7fa0a414f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.991+0000 7fa0a9000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a40c5190 con 0x7fa0a406b470 2024-03-18T16:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.991+0000 7fa0a9000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a40c9d30 con 0x7fa0a4154150 2024-03-18T16:32:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.991+0000 7fa0a9000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a40c5310 con 0x7fa0a406de50 2024-03-18T16:32:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.992+0000 7fa0a2a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406de50 0x7fa0a414f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.992+0000 7fa0a3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0a4154150 0x7fa0a414f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.992+0000 7fa0a2a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406de50 0x7fa0a414f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:58318/0 (socket says 172.21.15.102:58318) 2024-03-18T16:32:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.992+0000 7fa0a2a00640 1 -- 172.21.15.102:0/3508625192 learned_addr learned my addr 172.21.15.102:0/3508625192 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.996+0000 7fa0a2a00640 1 -- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0a4154150 msgr2=0x7fa0a414f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.996+0000 7fa0a2a00640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0a4154150 0x7fa0a414f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.996+0000 7fa0a2a00640 1 -- 172.21.15.102:0/3508625192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a406b470 msgr2=0x7fa0a41e3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.996+0000 7fa0a2a00640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a406b470 0x7fa0a41e3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.996+0000 7fa0a2a00640 1 -- 172.21.15.102:0/3508625192 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a41e3e10 con 0x7fa0a406de50 2024-03-18T16:32:17.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:16.998+0000 7fa0a2a00640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406de50 0x7fa0a414f0f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c440 tx=0x7fa09000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.003+0000 7fa09b400640 1 -- 172.21.15.102:0/3508625192 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa090017070 con 0x7fa0a406de50 2024-03-18T16:32:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.003+0000 7fa09b400640 1 -- 172.21.15.102:0/3508625192 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0900035f0 con 0x7fa0a406de50 2024-03-18T16:32:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.003+0000 7fa09b400640 1 -- 172.21.15.102:0/3508625192 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0900039e0 con 0x7fa0a406de50 2024-03-18T16:32:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.003+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a41f3ff0 con 0x7fa0a406de50 2024-03-18T16:32:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.003+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0a41e66e0 con 0x7fa0a406de50 2024-03-18T16:32:17.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.005+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0a4074fd0 con 0x7fa0a406de50 2024-03-18T16:32:17.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.029+0000 7fa09b400640 1 -- 172.21.15.102:0/3508625192 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa090003b80 con 0x7fa0a406de50 2024-03-18T16:32:17.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.031+0000 7fa09b400640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa08406bc00 0x7fa08406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:17.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.031+0000 7fa0a3400640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa08406bc00 0x7fa08406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:17.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.032+0000 7fa0a3400640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa08406bc00 0x7fa08406e0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa094036000 tx=0x7fa094030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:17.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.274+0000 7fa09b400640 1 -- 172.21.15.102:0/3508625192 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa090056290 con 0x7fa0a406de50 2024-03-18T16:32:17.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.287+0000 7fa09b400640 1 -- 172.21.15.102:0/3508625192 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa0900010c0 con 0x7fa0a406de50 2024-03-18T16:32:17.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.585+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0a40cc450 con 0x7fa0a406de50 2024-03-18T16:32:17.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.611+0000 7fa09b400640 1 -- 172.21.15.102:0/3508625192 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa09005e030 con 0x7fa0a406de50 2024-03-18T16:32:17.613 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:17.613 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:17.613 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa08406bc00 msgr2=0x7fa08406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa08406bc00 0x7fa08406e0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa094036000 tx=0x7fa094030000 comp rx=0 tx=0).stop 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406de50 msgr2=0x7fa0a414f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406de50 0x7fa0a414f0f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c440 tx=0x7fa09000c900 comp rx=0 tx=0).stop 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 shutdown_connections 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa08406bc00 0x7fa08406e0c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0a4154150 0x7fa0a414f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0a406de50 0x7fa0a414f0f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 --2- 172.21.15.102:0/3508625192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a406b470 0x7fa0a41e3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:17.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.615+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 >> 172.21.15.102:0/3508625192 conn(0x7fa0a40727e0 msgr2=0x7fa0a40c6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:17.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.616+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 shutdown_connections 2024-03-18T16:32:17.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:17.616+0000 7fa0a9000640 1 -- 172.21.15.102:0/3508625192 wait complete. 2024-03-18T16:32:18.237 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:18.237 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:19.239 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.429+0000 7f854ea00640 1 Processor -- start 2024-03-18T16:32:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.429+0000 7f854ea00640 1 -- start start 2024-03-18T16:32:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.430+0000 7f854ea00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f854815a310 0x7f854806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.430+0000 7f854ea00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85480ca500 con 0x7f854815baa0 2024-03-18T16:32:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.430+0000 7f854ea00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85480ca7a0 con 0x7f854815a6f0 2024-03-18T16:32:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.430+0000 7f854ea00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f85480cd410 con 0x7f854815a310 2024-03-18T16:32:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.431+0000 7f854d600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f854815a310 0x7f854806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.431+0000 7f854e000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f854815baa0 0x7f854815bf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58046/0 (socket says 172.21.15.102:58046) 2024-03-18T16:32:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.431+0000 7f854e000640 1 -- 172.21.15.102:0/3784056892 learned_addr learned my addr 172.21.15.102:0/3784056892 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.435+0000 7f8547e00640 1 -- 172.21.15.102:0/3784056892 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 8091733 0 0) 0x7f85480ca500 con 0x7f854815baa0 2024-03-18T16:32:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.435+0000 7f8547e00640 1 -- 172.21.15.102:0/3784056892 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8530003520 con 0x7f854815baa0 2024-03-18T16:32:19.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.437+0000 7f854d600640 1 -- 172.21.15.102:0/3784056892 >> v1:172.21.15.154:6789/0 conn(0x7f854815a6f0 legacy=0x7f854815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:19.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.437+0000 7f854d600640 1 -- 172.21.15.102:0/3784056892 >> v1:172.21.15.102:6789/0 conn(0x7f854815baa0 legacy=0x7f854815bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:19.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.437+0000 7f854d600640 1 -- 172.21.15.102:0/3784056892 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85480caa80 con 0x7f854815a310 2024-03-18T16:32:19.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.441+0000 7f854d600640 1 --2- 172.21.15.102:0/3784056892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815a310 0x7f854806d3e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f853c0097a0 tx=0x7f853c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=439b45997f80f6d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.447+0000 7f8547e00640 1 -- 172.21.15.102:0/3784056892 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c034070 con 0x7f854815a310 2024-03-18T16:32:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.447+0000 7f8547e00640 1 -- 172.21.15.102:0/3784056892 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853c030610 con 0x7f854815a310 2024-03-18T16:32:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.447+0000 7f8547e00640 1 -- 172.21.15.102:0/3784056892 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c0309e0 con 0x7f854815a310 2024-03-18T16:32:19.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.448+0000 7f854ea00640 1 -- 172.21.15.102:0/3784056892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815a310 msgr2=0x7f854806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:19.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.448+0000 7f854ea00640 1 --2- 172.21.15.102:0/3784056892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815a310 0x7f854806d3e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f853c0097a0 tx=0x7f853c02b960 comp rx=0 tx=0).stop 2024-03-18T16:32:19.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.448+0000 7f854ea00640 1 -- 172.21.15.102:0/3784056892 shutdown_connections 2024-03-18T16:32:19.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.448+0000 7f854ea00640 1 --2- 172.21.15.102:0/3784056892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815a310 0x7f854806d3e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:19.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.448+0000 7f854ea00640 1 -- 172.21.15.102:0/3784056892 >> 172.21.15.102:0/3784056892 conn(0x7f854808c1a0 msgr2=0x7f8548066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:19.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.448+0000 7f854ea00640 1 -- 172.21.15.102:0/3784056892 shutdown_connections 2024-03-18T16:32:19.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.448+0000 7f854ea00640 1 -- 172.21.15.102:0/3784056892 wait complete. 2024-03-18T16:32:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.450+0000 7f854ea00640 1 Processor -- start 2024-03-18T16:32:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.450+0000 7f854ea00640 1 -- start start 2024-03-18T16:32:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.450+0000 7f854ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f854815a310 0x7f854814c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.451+0000 7f854ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f854815a6f0 0x7f85480c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.451+0000 7f854ea00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815baa0 0x7f85480c4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.451+0000 7f854ea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f854806d920 con 0x7f854815a310 2024-03-18T16:32:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.451+0000 7f854ea00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f85480cd130 con 0x7f854815baa0 2024-03-18T16:32:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.451+0000 7f854ea00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f85481548c0 con 0x7f854815a6f0 2024-03-18T16:32:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.451+0000 7f854e000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815baa0 0x7f85480c4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.451+0000 7f854d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f854815a310 0x7f854814c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.452+0000 7f854cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f854815a6f0 0x7f85480c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.452+0000 7f854d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f854815a310 0x7f854814c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34520/0 (socket says 172.21.15.102:34520) 2024-03-18T16:32:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.452+0000 7f854d600640 1 -- 172.21.15.102:0/4282750049 learned_addr learned my addr 172.21.15.102:0/4282750049 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:19.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.455+0000 7f854e000640 1 -- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f854815a6f0 msgr2=0x7f85480c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:19.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.455+0000 7f854e000640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f854815a6f0 0x7f85480c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:19.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.455+0000 7f854e000640 1 -- 172.21.15.102:0/4282750049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f854815a310 msgr2=0x7f854814c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:19.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.455+0000 7f854e000640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f854815a310 0x7f854814c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:19.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.455+0000 7f854e000640 1 -- 172.21.15.102:0/4282750049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854814c7c0 con 0x7f854815baa0 2024-03-18T16:32:19.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.458+0000 7f854e000640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815baa0 0x7f85480c4c40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f854000caa0 tx=0x7f854000cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:19.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.461+0000 7f8545600640 1 -- 172.21.15.102:0/4282750049 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540017070 con 0x7f854815baa0 2024-03-18T16:32:19.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.461+0000 7f8545600640 1 -- 172.21.15.102:0/4282750049 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f854000ab40 con 0x7f854815baa0 2024-03-18T16:32:19.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.461+0000 7f8545600640 1 -- 172.21.15.102:0/4282750049 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540012400 con 0x7f854815baa0 2024-03-18T16:32:19.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.461+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85480c5c30 con 0x7f854815baa0 2024-03-18T16:32:19.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.462+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85480c61f0 con 0x7f854815baa0 2024-03-18T16:32:19.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.463+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8548074fd0 con 0x7f854815baa0 2024-03-18T16:32:19.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.724+0000 7f8545600640 1 -- 172.21.15.102:0/4282750049 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f854001a070 con 0x7f854815baa0 2024-03-18T16:32:19.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.726+0000 7f8545600640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f852806bc20 0x7f852806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:19.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.726+0000 7f854d600640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f852806bc20 0x7f852806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:19.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.727+0000 7f8545600640 1 -- 172.21.15.102:0/4282750049 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f85400570a0 con 0x7f854815baa0 2024-03-18T16:32:19.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.727+0000 7f854d600640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f852806bc20 0x7f852806e0e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f853c009560 tx=0x7f853c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:19.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.738+0000 7f8545600640 1 -- 172.21.15.102:0/4282750049 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f854814c7c0 con 0x7f854815baa0 2024-03-18T16:32:19.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:19.987+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8548151ec0 con 0x7f854815baa0 2024-03-18T16:32:20.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.000+0000 7f8545600640 1 -- 172.21.15.102:0/4282750049 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8540059000 con 0x7f854815baa0 2024-03-18T16:32:20.002 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:20.002 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:20.003 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:20.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f852806bc20 msgr2=0x7f852806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:20.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f852806bc20 0x7f852806e0e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f853c009560 tx=0x7f853c03b040 comp rx=0 tx=0).stop 2024-03-18T16:32:20.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815baa0 msgr2=0x7f85480c4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:20.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815baa0 0x7f85480c4c40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f854000caa0 tx=0x7f854000cea0 comp rx=0 tx=0).stop 2024-03-18T16:32:20.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 shutdown_connections 2024-03-18T16:32:20.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f852806bc20 0x7f852806e0e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:20.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f854815baa0 0x7f85480c4c40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:20.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f854815a6f0 0x7f85480c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:20.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 --2- 172.21.15.102:0/4282750049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f854815a310 0x7f854814c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:20.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 >> 172.21.15.102:0/4282750049 conn(0x7f854808c1a0 msgr2=0x7f85481520e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:20.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 shutdown_connections 2024-03-18T16:32:20.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:20.003+0000 7f854ea00640 1 -- 172.21.15.102:0/4282750049 wait complete. 2024-03-18T16:32:20.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:20.626 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:21.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:21.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.802+0000 7fdcfb200640 1 Processor -- start 2024-03-18T16:32:21.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.802+0000 7fdcfb200640 1 -- start start 2024-03-18T16:32:21.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.803+0000 7fdcfb200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdcf4141650 0x7fdcf4158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:21.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.803+0000 7fdcfb200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdcf40c4fa0 0x7fdcf40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:21.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.803+0000 7fdcfb200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf40c72d0 con 0x7fdcf415c3a0 2024-03-18T16:32:21.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.803+0000 7fdcfb200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fdcf4159370 con 0x7fdcf4141650 2024-03-18T16:32:21.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.803+0000 7fdcfb200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdcf4158ad0 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.804+0000 7fdcfa800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdcf40c4fa0 0x7fdcf40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:21.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.804+0000 7fdcf9400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdcf415c3a0 0x7fdcf40c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51536/0 (socket says 172.21.15.102:51536) 2024-03-18T16:32:21.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.804+0000 7fdcf9400640 1 -- 172.21.15.102:0/1194059906 learned_addr learned my addr 172.21.15.102:0/1194059906 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:21.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.805+0000 7fdcf9e00640 1 --2- 172.21.15.102:0/1194059906 >> v2:172.21.15.102:3300/0 conn(0x7fdcf4141650 0x7fdcf4158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:21.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.808+0000 7fdcfa800640 1 -- 172.21.15.102:0/1194059906 >> v2:172.21.15.102:3300/0 conn(0x7fdcf4141650 msgr2=0x7fdcf4158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:21.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.808+0000 7fdcfa800640 1 --2- 172.21.15.102:0/1194059906 >> v2:172.21.15.102:3300/0 conn(0x7fdcf4141650 0x7fdcf4158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:21.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.808+0000 7fdcfa800640 1 -- 172.21.15.102:0/1194059906 >> v1:172.21.15.102:6789/0 conn(0x7fdcf415c3a0 legacy=0x7fdcf40c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:21.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.808+0000 7fdcfa800640 1 -- 172.21.15.102:0/1194059906 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf4158db0 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.811+0000 7fdcfa800640 1 --2- 172.21.15.102:0/1194059906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcf40c4fa0 0x7fdcf40c5410 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40c9720 tx=0x7fdcf002bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f27cb98deb086d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:21.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.816+0000 7fdcf8a00640 1 -- 172.21.15.102:0/1194059906 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0039070 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.816+0000 7fdcf8a00640 1 -- 172.21.15.102:0/1194059906 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf002cc70 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.816+0000 7fdcf8a00640 1 -- 172.21.15.102:0/1194059906 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0034660 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.817+0000 7fdcfb200640 1 -- 172.21.15.102:0/1194059906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcf40c4fa0 msgr2=0x7fdcf40c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:21.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.817+0000 7fdcfb200640 1 --2- 172.21.15.102:0/1194059906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcf40c4fa0 0x7fdcf40c5410 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdcf40c9720 tx=0x7fdcf002bec0 comp rx=0 tx=0).stop 2024-03-18T16:32:21.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.818+0000 7fdcfb200640 1 -- 172.21.15.102:0/1194059906 shutdown_connections 2024-03-18T16:32:21.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.818+0000 7fdcfb200640 1 --2- 172.21.15.102:0/1194059906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcf40c4fa0 0x7fdcf40c5410 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:21.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.818+0000 7fdcfb200640 1 --2- 172.21.15.102:0/1194059906 >> v2:172.21.15.102:3300/0 conn(0x7fdcf4141650 0x7fdcf4158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:21.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.818+0000 7fdcfb200640 1 -- 172.21.15.102:0/1194059906 >> 172.21.15.102:0/1194059906 conn(0x7fdcf40cb890 msgr2=0x7fdcf40cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:21.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.818+0000 7fdcfb200640 1 -- 172.21.15.102:0/1194059906 shutdown_connections 2024-03-18T16:32:21.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.818+0000 7fdcfb200640 1 -- 172.21.15.102:0/1194059906 wait complete. 2024-03-18T16:32:21.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.819+0000 7fdcfb200640 1 Processor -- start 2024-03-18T16:32:21.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.820+0000 7fdcfb200640 1 -- start start 2024-03-18T16:32:21.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.820+0000 7fdcfb200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcf40c4fa0 0x7fdcf413bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:21.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.820+0000 7fdcfb200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcf4141650 0x7fdcf41f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:21.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.820+0000 7fdcfb200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcf415c3a0 0x7fdcf41f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:21.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.820+0000 7fdcfb200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf40c66a0 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.820+0000 7fdcfb200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf40c63c0 con 0x7fdcf415c3a0 2024-03-18T16:32:21.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.820+0000 7fdcfb200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf415c7c0 con 0x7fdcf4141650 2024-03-18T16:32:21.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.821+0000 7fdcf9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcf40c4fa0 0x7fdcf413bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:21.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.821+0000 7fdcfa800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcf415c3a0 0x7fdcf41f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:21.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.821+0000 7fdcf9400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcf4141650 0x7fdcf41f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:21.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.821+0000 7fdcf9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcf40c4fa0 0x7fdcf413bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42572/0 (socket says 172.21.15.102:42572) 2024-03-18T16:32:21.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.822+0000 7fdcf9e00640 1 -- 172.21.15.102:0/510447785 learned_addr learned my addr 172.21.15.102:0/510447785 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:21.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.825+0000 7fdcf9e00640 1 -- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcf4141650 msgr2=0x7fdcf41f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:21.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.825+0000 7fdcf9e00640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcf4141650 0x7fdcf41f14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:21.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.825+0000 7fdcf9e00640 1 -- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcf415c3a0 msgr2=0x7fdcf41f19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:21.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.825+0000 7fdcf9e00640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcf415c3a0 0x7fdcf41f19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:21.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.825+0000 7fdcf9e00640 1 -- 172.21.15.102:0/510447785 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf415bfa0 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.825+0000 7fdcf9400640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcf4141650 0x7fdcf41f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:32:21.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.828+0000 7fdcf9e00640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcf40c4fa0 0x7fdcf413bec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdce800e9e0 tx=0x7fdce800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:21.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.833+0000 7fdcee000640 1 -- 172.21.15.102:0/510447785 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8017070 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.833+0000 7fdcee000640 1 -- 172.21.15.102:0/510447785 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce800c9d0 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.833+0000 7fdcee000640 1 -- 172.21.15.102:0/510447785 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce800cdc0 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.833+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf415bcc0 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.833+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf413ad80 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.834+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcf4074fd0 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.861+0000 7fdcee000640 1 -- 172.21.15.102:0/510447785 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdce801a070 con 0x7fdcf40c4fa0 2024-03-18T16:32:21.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.863+0000 7fdcee000640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcd806bc20 0x7fdcd806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:21.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.863+0000 7fdcf9400640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcd806bc20 0x7fdcd806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:21.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.864+0000 7fdcf9400640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcd806bc20 0x7fdcd806e0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4155cc0 tx=0x7fdce4007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:21.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:21.866+0000 7fdcee000640 1 -- 172.21.15.102:0/510447785 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fdce80576a0 con 0x7fdcf40c4fa0 2024-03-18T16:32:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.166+0000 7fdcee000640 1 -- 172.21.15.102:0/510447785 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdce8059600 con 0x7fdcf40c4fa0 2024-03-18T16:32:22.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.409+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcf40b7f00 con 0x7fdcf40c4fa0 2024-03-18T16:32:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.443+0000 7fdcee000640 1 -- 172.21.15.102:0/510447785 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fdce805d6a0 con 0x7fdcf40c4fa0 2024-03-18T16:32:22.445 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:22.445 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.446+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcd806bc20 msgr2=0x7fdcd806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.446+0000 7fdcfb200640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcd806bc20 0x7fdcd806e0e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4155cc0 tx=0x7fdce4007e80 comp rx=0 tx=0).stop 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.446+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcf40c4fa0 msgr2=0x7fdcf413bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.446+0000 7fdcfb200640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcf40c4fa0 0x7fdcf413bec0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdce800e9e0 tx=0x7fdce800eea0 comp rx=0 tx=0).stop 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.447+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 shutdown_connections 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.447+0000 7fdcfb200640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcd806bc20 0x7fdcd806e0e0 secure :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4155cc0 tx=0x7fdce4007e80 comp rx=0 tx=0).stop 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.447+0000 7fdcfb200640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdcf415c3a0 0x7fdcf41f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.447+0000 7fdcfb200640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdcf4141650 0x7fdcf41f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.447+0000 7fdcfb200640 1 --2- 172.21.15.102:0/510447785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdcf40c4fa0 0x7fdcf413bec0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.447+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 >> 172.21.15.102:0/510447785 conn(0x7fdcf40cb890 msgr2=0x7fdcf40cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.447+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 shutdown_connections 2024-03-18T16:32:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:22.447+0000 7fdcfb200640 1 -- 172.21.15.102:0/510447785 wait complete. 2024-03-18T16:32:23.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:23.069 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:23.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:32:23.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:32:24.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.244+0000 7f1c8ca00640 1 Processor -- start 2024-03-18T16:32:24.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.244+0000 7f1c8ca00640 1 -- start start 2024-03-18T16:32:24.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.245+0000 7f1c8ca00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1c880cfe70 0x7f1c8815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:24.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.245+0000 7f1c8ca00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cc2f0 0x7f1c880cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:24.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.245+0000 7f1c8ca00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c880cabd0 con 0x7f1c880d0250 2024-03-18T16:32:24.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.245+0000 7f1c8ca00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1c880caea0 con 0x7f1c880cfe70 2024-03-18T16:32:24.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.245+0000 7f1c8ca00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1c880cb180 con 0x7f1c880cc2f0 2024-03-18T16:32:24.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.246+0000 7f1c86800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1c880cfe70 0x7f1c8815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:24.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.246+0000 7f1c87200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cc2f0 0x7f1c880cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:24.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.246+0000 7f1c87200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cc2f0 0x7f1c880cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57606/0 (socket says 172.21.15.102:57606) 2024-03-18T16:32:24.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.246+0000 7f1c87200640 1 -- 172.21.15.102:0/588854895 learned_addr learned my addr 172.21.15.102:0/588854895 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:24.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.249+0000 7f1c87200640 1 -- 172.21.15.102:0/588854895 >> v2:172.21.15.154:3300/0 conn(0x7f1c880cfe70 msgr2=0x7f1c8815ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:24.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.249+0000 7f1c87200640 1 --2- 172.21.15.102:0/588854895 >> v2:172.21.15.154:3300/0 conn(0x7f1c880cfe70 0x7f1c8815ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.249+0000 7f1c87200640 1 -- 172.21.15.102:0/588854895 >> v1:172.21.15.154:6789/0 conn(0x7f1c880d0250 legacy=0x7f1c880c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:24.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.249+0000 7f1c87200640 1 -- 172.21.15.102:0/588854895 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c880cb460 con 0x7f1c880cc2f0 2024-03-18T16:32:24.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.251+0000 7f1c87200640 1 --2- 172.21.15.102:0/588854895 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cc2f0 0x7f1c880cc710 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009e50 tx=0x7f1c8002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f81f203da69cc31 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:24.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.255+0000 7f1c85400640 1 -- 172.21.15.102:0/588854895 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8002cb90 con 0x7f1c880cc2f0 2024-03-18T16:32:24.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.255+0000 7f1c85400640 1 -- 172.21.15.102:0/588854895 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8002cd30 con 0x7f1c880cc2f0 2024-03-18T16:32:24.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.255+0000 7f1c85400640 1 -- 172.21.15.102:0/588854895 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80034610 con 0x7f1c880cc2f0 2024-03-18T16:32:24.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.256+0000 7f1c8ca00640 1 -- 172.21.15.102:0/588854895 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cc2f0 msgr2=0x7f1c880cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.256+0000 7f1c8ca00640 1 --2- 172.21.15.102:0/588854895 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cc2f0 0x7f1c880cc710 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009e50 tx=0x7f1c8002bff0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.256+0000 7f1c8ca00640 1 -- 172.21.15.102:0/588854895 shutdown_connections 2024-03-18T16:32:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.256+0000 7f1c8ca00640 1 --2- 172.21.15.102:0/588854895 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cc2f0 0x7f1c880cc710 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.256+0000 7f1c8ca00640 1 --2- 172.21.15.102:0/588854895 >> v2:172.21.15.154:3300/0 conn(0x7f1c880cfe70 0x7f1c8815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.256+0000 7f1c8ca00640 1 -- 172.21.15.102:0/588854895 >> 172.21.15.102:0/588854895 conn(0x7f1c8808c3e0 msgr2=0x7f1c8806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.256+0000 7f1c8ca00640 1 -- 172.21.15.102:0/588854895 shutdown_connections 2024-03-18T16:32:24.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.256+0000 7f1c8ca00640 1 -- 172.21.15.102:0/588854895 wait complete. 2024-03-18T16:32:24.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.257+0000 7f1c8ca00640 1 Processor -- start 2024-03-18T16:32:24.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.258+0000 7f1c8ca00640 1 -- start start 2024-03-18T16:32:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.258+0000 7f1c8ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c880cc2f0 0x7f1c881e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.258+0000 7f1c8ca00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cfe70 0x7f1c88150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.258+0000 7f1c8ca00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c880d0250 0x7f1c88159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.258+0000 7f1c8ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1c880cbd00 con 0x7f1c880cc2f0 2024-03-18T16:32:24.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.258+0000 7f1c8ca00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1c880cba20 con 0x7f1c880d0250 2024-03-18T16:32:24.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.258+0000 7f1c8ca00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1c880ccc50 con 0x7f1c880cfe70 2024-03-18T16:32:24.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.259+0000 7f1c85e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cfe70 0x7f1c88150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:24.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.259+0000 7f1c87200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c880d0250 0x7f1c88159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:24.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.259+0000 7f1c86800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c880cc2f0 0x7f1c881e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:24.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.260+0000 7f1c87200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c880d0250 0x7f1c88159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48036/0 (socket says 172.21.15.102:48036) 2024-03-18T16:32:24.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.260+0000 7f1c87200640 1 -- 172.21.15.102:0/1803302743 learned_addr learned my addr 172.21.15.102:0/1803302743 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:24.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.263+0000 7f1c87200640 1 -- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cfe70 msgr2=0x7f1c88150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.263+0000 7f1c87200640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cfe70 0x7f1c88150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.263+0000 7f1c87200640 1 -- 172.21.15.102:0/1803302743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c880cc2f0 msgr2=0x7f1c881e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.263+0000 7f1c87200640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c880cc2f0 0x7f1c881e2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.263+0000 7f1c87200640 1 -- 172.21.15.102:0/1803302743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c881f41d0 con 0x7f1c880d0250 2024-03-18T16:32:24.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.265+0000 7f1c87200640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c880d0250 0x7f1c88159250 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009e50 tx=0x7f1c80004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.269+0000 7f1c7ea00640 1 -- 172.21.15.102:0/1803302743 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80043020 con 0x7f1c880d0250 2024-03-18T16:32:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.269+0000 7f1c7ea00640 1 -- 172.21.15.102:0/1803302743 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8003e450 con 0x7f1c880d0250 2024-03-18T16:32:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.269+0000 7f1c7ea00640 1 -- 172.21.15.102:0/1803302743 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80009510 con 0x7f1c880d0250 2024-03-18T16:32:24.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.270+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c881f3370 con 0x7f1c880d0250 2024-03-18T16:32:24.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.270+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c881f3930 con 0x7f1c880d0250 2024-03-18T16:32:24.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.271+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c880c94a0 con 0x7f1c880d0250 2024-03-18T16:32:24.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.291+0000 7f1c7ea00640 1 -- 172.21.15.102:0/1803302743 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1c80046020 con 0x7f1c880d0250 2024-03-18T16:32:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.292+0000 7f1c7ea00640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1c6806bc20 0x7f1c6806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:24.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.292+0000 7f1c86800640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1c6806bc20 0x7f1c6806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:24.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.293+0000 7f1c86800640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1c6806bc20 0x7f1c6806e0e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009b40 tx=0x7f1c78009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:24.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.294+0000 7f1c7ea00640 1 -- 172.21.15.102:0/1803302743 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1c80079970 con 0x7f1c880d0250 2024-03-18T16:32:24.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.543+0000 7f1c7ea00640 1 -- 172.21.15.102:0/1803302743 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1c880c94a0 con 0x7f1c880d0250 2024-03-18T16:32:24.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.796+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c880c8cd0 con 0x7f1c880d0250 2024-03-18T16:32:24.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.810+0000 7f1c7ea00640 1 -- 172.21.15.102:0/1803302743 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1c880c8cd0 con 0x7f1c880d0250 2024-03-18T16:32:24.812 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:24.812 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:24.812 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:24.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1c6806bc20 msgr2=0x7f1c6806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:24.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1c6806bc20 0x7f1c6806e0e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009b40 tx=0x7f1c78009190 comp rx=0 tx=0).stop 2024-03-18T16:32:24.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c880d0250 msgr2=0x7f1c88159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:24.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c880d0250 0x7f1c88159250 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009e50 tx=0x7f1c80004a40 comp rx=0 tx=0).stop 2024-03-18T16:32:24.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 shutdown_connections 2024-03-18T16:32:24.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1c6806bc20 0x7f1c6806e0e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c880d0250 0x7f1c88159250 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1c880cfe70 0x7f1c88150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 --2- 172.21.15.102:0/1803302743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c880cc2f0 0x7f1c881e2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:24.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 >> 172.21.15.102:0/1803302743 conn(0x7f1c8808c3e0 msgr2=0x7f1c880c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:24.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.812+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 shutdown_connections 2024-03-18T16:32:24.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:24.813+0000 7f1c8ca00640 1 -- 172.21.15.102:0/1803302743 wait complete. 2024-03-18T16:32:25.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:25.434 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:26.435 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:26.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.605+0000 7f1a0a000640 1 Processor -- start 2024-03-18T16:32:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.605+0000 7f1a0a000640 1 -- start start 2024-03-18T16:32:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.606+0000 7f1a0a000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1a040cfe70 0x7f1a0415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:26.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.606+0000 7f1a0a000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a040d0250 0x7f1a040c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:26.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.606+0000 7f1a0a000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a040cabd0 con 0x7f1a040cc2f0 2024-03-18T16:32:26.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.606+0000 7f1a0a000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1a040caea0 con 0x7f1a040cfe70 2024-03-18T16:32:26.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.606+0000 7f1a0a000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1a040cb180 con 0x7f1a040d0250 2024-03-18T16:32:26.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.607+0000 7f1a03e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a040d0250 0x7f1a040c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:26.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.607+0000 7f1a03e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1a040d0250 0x7f1a040c65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48052/0 (socket says 172.21.15.102:48052) 2024-03-18T16:32:26.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.607+0000 7f1a03e00640 1 -- 172.21.15.102:0/4134219055 learned_addr learned my addr 172.21.15.102:0/4134219055 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:26.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.608+0000 7f1a08c00640 1 --2- 172.21.15.102:0/4134219055 >> v2:172.21.15.102:3300/0 conn(0x7f1a040cfe70 0x7f1a0415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:26.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.611+0000 7f1a03e00640 1 -- 172.21.15.102:0/4134219055 >> v2:172.21.15.102:3300/0 conn(0x7f1a040cfe70 msgr2=0x7f1a0415ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:26.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.611+0000 7f1a03e00640 1 --2- 172.21.15.102:0/4134219055 >> v2:172.21.15.102:3300/0 conn(0x7f1a040cfe70 0x7f1a0415ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:26.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.611+0000 7f1a03e00640 1 -- 172.21.15.102:0/4134219055 >> v1:172.21.15.102:6789/0 conn(0x7f1a040cc2f0 legacy=0x7f1a040cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:26.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.611+0000 7f1a03e00640 1 -- 172.21.15.102:0/4134219055 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a040cb460 con 0x7f1a040d0250 2024-03-18T16:32:26.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.614+0000 7f1a03e00640 1 --2- 172.21.15.102:0/4134219055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040d0250 0x7f1a040c65e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1a040c6370 tx=0x7f19f002ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74e4c85cdcab4208 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:26.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.618+0000 7f1a03400640 1 -- 172.21.15.102:0/4134219055 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f002f850 con 0x7f1a040d0250 2024-03-18T16:32:26.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.618+0000 7f1a03400640 1 -- 172.21.15.102:0/4134219055 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f002f9f0 con 0x7f1a040d0250 2024-03-18T16:32:26.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.618+0000 7f1a03400640 1 -- 172.21.15.102:0/4134219055 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f002fdc0 con 0x7f1a040d0250 2024-03-18T16:32:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.619+0000 7f1a0a000640 1 -- 172.21.15.102:0/4134219055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040d0250 msgr2=0x7f1a040c65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.619+0000 7f1a0a000640 1 --2- 172.21.15.102:0/4134219055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040d0250 0x7f1a040c65e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1a040c6370 tx=0x7f19f002ba30 comp rx=0 tx=0).stop 2024-03-18T16:32:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.619+0000 7f1a0a000640 1 -- 172.21.15.102:0/4134219055 shutdown_connections 2024-03-18T16:32:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.619+0000 7f1a0a000640 1 --2- 172.21.15.102:0/4134219055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040d0250 0x7f1a040c65e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.619+0000 7f1a0a000640 1 --2- 172.21.15.102:0/4134219055 >> v2:172.21.15.102:3300/0 conn(0x7f1a040cfe70 0x7f1a0415ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.619+0000 7f1a0a000640 1 -- 172.21.15.102:0/4134219055 >> 172.21.15.102:0/4134219055 conn(0x7f1a0408c3e0 msgr2=0x7f1a0406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.619+0000 7f1a0a000640 1 -- 172.21.15.102:0/4134219055 shutdown_connections 2024-03-18T16:32:26.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.619+0000 7f1a0a000640 1 -- 172.21.15.102:0/4134219055 wait complete. 2024-03-18T16:32:26.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.621+0000 7f1a0a000640 1 Processor -- start 2024-03-18T16:32:26.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.621+0000 7f1a0a000640 1 -- start start 2024-03-18T16:32:26.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.622+0000 7f1a0a000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040cc2f0 0x7f1a041f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:26.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.622+0000 7f1a0a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040cfe70 0x7f1a04152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:26.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.622+0000 7f1a0a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a040d0250 0x7f1a041e2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:26.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.622+0000 7f1a0a000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1a040cbd00 con 0x7f1a040cfe70 2024-03-18T16:32:26.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.622+0000 7f1a0a000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1a040cba20 con 0x7f1a040cc2f0 2024-03-18T16:32:26.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.622+0000 7f1a0a000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1a040c6b20 con 0x7f1a040d0250 2024-03-18T16:32:26.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.623+0000 7f1a08c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040cc2f0 0x7f1a041f3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:26.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.623+0000 7f1a03e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040cfe70 0x7f1a04152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:26.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.623+0000 7f1a09600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a040d0250 0x7f1a041e2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:26.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.623+0000 7f1a08c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040cc2f0 0x7f1a041f3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48056/0 (socket says 172.21.15.102:48056) 2024-03-18T16:32:26.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.623+0000 7f1a08c00640 1 -- 172.21.15.102:0/2218948075 learned_addr learned my addr 172.21.15.102:0/2218948075 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:26.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.627+0000 7f1a03e00640 1 -- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a040d0250 msgr2=0x7f1a041e2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:26.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.627+0000 7f1a03e00640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a040d0250 0x7f1a041e2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:26.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.627+0000 7f1a03e00640 1 -- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040cc2f0 msgr2=0x7f1a041f3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:26.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.627+0000 7f1a03e00640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040cc2f0 0x7f1a041f3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:26.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.627+0000 7f1a03e00640 1 -- 172.21.15.102:0/2218948075 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a041e2a70 con 0x7f1a040cfe70 2024-03-18T16:32:26.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.630+0000 7f1a03e00640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040cfe70 0x7f1a04152ae0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f19f0033000 tx=0x7f19f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:26.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.632+0000 7f1a00c00640 1 -- 172.21.15.102:0/2218948075 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f0037d10 con 0x7f1a040cfe70 2024-03-18T16:32:26.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.632+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a041f0a50 con 0x7f1a040cfe70 2024-03-18T16:32:26.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.632+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a041efc90 con 0x7f1a040cfe70 2024-03-18T16:32:26.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.634+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a040c3840 con 0x7f1a040cfe70 2024-03-18T16:32:26.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.635+0000 7f1a00c00640 1 -- 172.21.15.102:0/2218948075 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19f0046070 con 0x7f1a040cfe70 2024-03-18T16:32:26.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.635+0000 7f1a00c00640 1 -- 172.21.15.102:0/2218948075 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f004b460 con 0x7f1a040cfe70 2024-03-18T16:32:26.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.928+0000 7f1a00c00640 1 -- 172.21.15.102:0/2218948075 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f19f0040070 con 0x7f1a040cfe70 2024-03-18T16:32:26.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.929+0000 7f1a00c00640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19e406bbd0 0x7f19e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:26.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.930+0000 7f1a08c00640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19e406bbd0 0x7f19e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:26.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.930+0000 7f1a08c00640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19e406bbd0 0x7f19e406e090 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006fd0 tx=0x7f19f8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:26.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.952+0000 7f1a00c00640 1 -- 172.21.15.102:0/2218948075 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f19f007a0c0 con 0x7f1a040cfe70 2024-03-18T16:32:26.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:26.953+0000 7f1a00c00640 1 -- 172.21.15.102:0/2218948075 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f19f0033030 con 0x7f1a040cfe70 2024-03-18T16:32:27.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.288+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a040ce9e0 con 0x7f1a040cfe70 2024-03-18T16:32:27.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.321+0000 7f1a00c00640 1 -- 172.21.15.102:0/2218948075 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f19f007c020 con 0x7f1a040cfe70 2024-03-18T16:32:27.322 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:27.323 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:27.323 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:27.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19e406bbd0 msgr2=0x7f19e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:27.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19e406bbd0 0x7f19e406e090 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f19f8006fd0 tx=0x7f19f8008040 comp rx=0 tx=0).stop 2024-03-18T16:32:27.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040cfe70 msgr2=0x7f1a04152ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:27.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040cfe70 0x7f1a04152ae0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f19f0033000 tx=0x7f19f0004730 comp rx=0 tx=0).stop 2024-03-18T16:32:27.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 shutdown_connections 2024-03-18T16:32:27.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19e406bbd0 0x7f19e406e090 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:27.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1a040d0250 0x7f1a041e2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:27.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a040cfe70 0x7f1a04152ae0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:27.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 --2- 172.21.15.102:0/2218948075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a040cc2f0 0x7f1a041f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:27.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.324+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 >> 172.21.15.102:0/2218948075 conn(0x7f1a0408c3e0 msgr2=0x7f1a040cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:27.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.325+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 shutdown_connections 2024-03-18T16:32:27.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:27.325+0000 7f1a0a000640 1 -- 172.21.15.102:0/2218948075 wait complete. 2024-03-18T16:32:27.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:27.954 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:28.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.134+0000 7f4de0a00640 1 Processor -- start 2024-03-18T16:32:29.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.134+0000 7f4de0a00640 1 -- start start 2024-03-18T16:32:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.135+0000 7f4de0a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4ddc15a310 0x7f4ddc06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.136+0000 7f4de0a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4ddc15a6f0 0x7f4ddc15ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.136+0000 7f4de0a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15baa0 0x7f4ddc15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.136+0000 7f4de0a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4ddc0cd410 con 0x7f4ddc15a310 2024-03-18T16:32:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.136+0000 7f4de0a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4ddc15c470 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.136+0000 7f4de0a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4ddc0ca500 con 0x7f4ddc15baa0 2024-03-18T16:32:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.136+0000 7f4ddb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4ddc15a310 0x7f4ddc06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.136+0000 7f4ddaa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4ddc15a6f0 0x7f4ddc15ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.137+0000 7f4ddbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15baa0 0x7f4ddc15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:29.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.137+0000 7f4ddb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4ddc15a310 0x7f4ddc06d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42604/0 (socket says 172.21.15.102:42604) 2024-03-18T16:32:29.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.137+0000 7f4ddb400640 1 -- 172.21.15.102:0/1122234481 learned_addr learned my addr 172.21.15.102:0/1122234481 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:29.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.139+0000 7f4ddbe00640 1 -- 172.21.15.102:0/1122234481 >> v2:172.21.15.154:3300/0 conn(0x7f4ddc15a6f0 msgr2=0x7f4ddc15ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:29.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.139+0000 7f4ddbe00640 1 --2- 172.21.15.102:0/1122234481 >> v2:172.21.15.154:3300/0 conn(0x7f4ddc15a6f0 0x7f4ddc15ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.139+0000 7f4ddbe00640 1 -- 172.21.15.102:0/1122234481 >> v2:172.21.15.102:3300/0 conn(0x7f4ddc15a310 msgr2=0x7f4ddc06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:29.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.139+0000 7f4ddbe00640 1 --2- 172.21.15.102:0/1122234481 >> v2:172.21.15.102:3300/0 conn(0x7f4ddc15a310 0x7f4ddc06d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.139+0000 7f4ddbe00640 1 -- 172.21.15.102:0/1122234481 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ddc15b080 con 0x7f4ddc15baa0 2024-03-18T16:32:29.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.142+0000 7f4ddbe00640 1 --2- 172.21.15.102:0/1122234481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15baa0 0x7f4ddc15bf30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0d0c80 tx=0x7f4dd402be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23644d588e934caf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.145+0000 7f4dda000640 1 -- 172.21.15.102:0/1122234481 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4039070 con 0x7f4ddc15baa0 2024-03-18T16:32:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.145+0000 7f4dda000640 1 -- 172.21.15.102:0/1122234481 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dd402cbd0 con 0x7f4ddc15baa0 2024-03-18T16:32:29.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.145+0000 7f4dda000640 1 -- 172.21.15.102:0/1122234481 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4034440 con 0x7f4ddc15baa0 2024-03-18T16:32:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.146+0000 7f4de0a00640 1 -- 172.21.15.102:0/1122234481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15baa0 msgr2=0x7f4ddc15bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.146+0000 7f4de0a00640 1 --2- 172.21.15.102:0/1122234481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15baa0 0x7f4ddc15bf30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0d0c80 tx=0x7f4dd402be20 comp rx=0 tx=0).stop 2024-03-18T16:32:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.146+0000 7f4de0a00640 1 -- 172.21.15.102:0/1122234481 shutdown_connections 2024-03-18T16:32:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.146+0000 7f4de0a00640 1 --2- 172.21.15.102:0/1122234481 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15baa0 0x7f4ddc15bf30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.146+0000 7f4de0a00640 1 --2- 172.21.15.102:0/1122234481 >> v2:172.21.15.154:3300/0 conn(0x7f4ddc15a6f0 0x7f4ddc15ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.146+0000 7f4de0a00640 1 --2- 172.21.15.102:0/1122234481 >> v2:172.21.15.102:3300/0 conn(0x7f4ddc15a310 0x7f4ddc06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.146+0000 7f4de0a00640 1 -- 172.21.15.102:0/1122234481 >> 172.21.15.102:0/1122234481 conn(0x7f4ddc08c1a0 msgr2=0x7f4ddc066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:29.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.146+0000 7f4de0a00640 1 -- 172.21.15.102:0/1122234481 shutdown_connections 2024-03-18T16:32:29.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.146+0000 7f4de0a00640 1 -- 172.21.15.102:0/1122234481 wait complete. 2024-03-18T16:32:29.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.149+0000 7f4de0a00640 1 Processor -- start 2024-03-18T16:32:29.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.149+0000 7f4de0a00640 1 -- start start 2024-03-18T16:32:29.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.151+0000 7f4de0a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ddc15a310 0x7f4ddc155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.151+0000 7f4de0a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15a6f0 0x7f4ddc153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.151+0000 7f4de0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ddc15baa0 0x7f4ddc1f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.151+0000 7f4de0a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4ddc0cce50 con 0x7f4ddc15baa0 2024-03-18T16:32:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.151+0000 7f4de0a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4ddc0caa80 con 0x7f4ddc15a310 2024-03-18T16:32:29.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.151+0000 7f4de0a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4ddc0cd130 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.151+0000 7f4ddaa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15a6f0 0x7f4ddc153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:29.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.151+0000 7f4ddb400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ddc15a310 0x7f4ddc155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:29.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.152+0000 7f4ddbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ddc15baa0 0x7f4ddc1f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:29.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.152+0000 7f4ddb400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ddc15a310 0x7f4ddc155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48066/0 (socket says 172.21.15.102:48066) 2024-03-18T16:32:29.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.152+0000 7f4ddb400640 1 -- 172.21.15.102:0/4240090692 learned_addr learned my addr 172.21.15.102:0/4240090692 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:29.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.155+0000 7f4ddaa00640 1 -- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ddc15a310 msgr2=0x7f4ddc155fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:29.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.155+0000 7f4ddaa00640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ddc15a310 0x7f4ddc155fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.155+0000 7f4ddaa00640 1 -- 172.21.15.102:0/4240090692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ddc15baa0 msgr2=0x7f4ddc1f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:29.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.155+0000 7f4ddaa00640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ddc15baa0 0x7f4ddc1f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.155+0000 7f4ddaa00640 1 -- 172.21.15.102:0/4240090692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ddc0c9970 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.158+0000 7f4ddaa00640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15a6f0 0x7f4ddc153280 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800e880 tx=0x7f4dc800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:29.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.160+0000 7f4dd3400640 1 -- 172.21.15.102:0/4240090692 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8017070 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.160+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ddc0c9c50 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.160+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ddc154ea0 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.162+0000 7f4dd3400640 1 -- 172.21.15.102:0/4240090692 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc800c940 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.162+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ddc074fd0 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.162+0000 7f4dd3400640 1 -- 172.21.15.102:0/4240090692 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8021060 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.425+0000 7f4dd3400640 1 -- 172.21.15.102:0/4240090692 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4dc801a070 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.426+0000 7f4dd3400640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4dbc06bbd0 0x7f4dbc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:29.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.426+0000 7f4dd3400640 1 -- 172.21.15.102:0/4240090692 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f4dc808b450 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.427+0000 7f4ddb400640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4dbc06bbd0 0x7f4dbc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:29.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.427+0000 7f4ddb400640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4dbc06bbd0 0x7f4dbc06e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009bb0 tx=0x7f4dcc009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:29.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.437+0000 7f4dd3400640 1 -- 172.21.15.102:0/4240090692 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4dc8059600 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.759+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ddc0caed0 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.783+0000 7f4dd3400640 1 -- 172.21.15.102:0/4240090692 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f4dc805d6a0 con 0x7f4ddc15a6f0 2024-03-18T16:32:29.785 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:29.785 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4dbc06bbd0 msgr2=0x7f4dbc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4dbc06bbd0 0x7f4dbc06e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009bb0 tx=0x7f4dcc009240 comp rx=0 tx=0).stop 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15a6f0 msgr2=0x7f4ddc153280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15a6f0 0x7f4ddc153280 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800e880 tx=0x7f4dc800ed40 comp rx=0 tx=0).stop 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 shutdown_connections 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4dbc06bbd0 0x7f4dbc06e090 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ddc15baa0 0x7f4ddc1f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4ddc15a6f0 0x7f4ddc153280 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 --2- 172.21.15.102:0/4240090692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4ddc15a310 0x7f4ddc155fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 >> 172.21.15.102:0/4240090692 conn(0x7f4ddc08c1a0 msgr2=0x7f4ddc15d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 shutdown_connections 2024-03-18T16:32:29.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:29.787+0000 7f4de0a00640 1 -- 172.21.15.102:0/4240090692 wait complete. 2024-03-18T16:32:30.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:30.416 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:31.417 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:31.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.591+0000 7fdd21e00640 1 Processor -- start 2024-03-18T16:32:31.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.591+0000 7fdd21e00640 1 -- start start 2024-03-18T16:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.592+0000 7fdd21e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15a6f0 0x7fdd1c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.592+0000 7fdd21e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdd1c15baa0 0x7fdd1c15bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.592+0000 7fdd21e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd1c15c470 con 0x7fdd1c15a310 2024-03-18T16:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.592+0000 7fdd21e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fdd1c0cd410 con 0x7fdd1c15baa0 2024-03-18T16:32:31.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.592+0000 7fdd21e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1c0ca500 con 0x7fdd1c15a6f0 2024-03-18T16:32:31.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.593+0000 7fdd1be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15a6f0 0x7fdd1c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:31.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.593+0000 7fdd20a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fdd1c15a310 0x7fdd1c06d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:42854/0 (socket says 172.21.15.102:42854) 2024-03-18T16:32:31.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.593+0000 7fdd20a00640 1 -- 172.21.15.102:0/378990503 learned_addr learned my addr 172.21.15.102:0/378990503 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:31.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.593+0000 7fdd21400640 1 --2- 172.21.15.102:0/378990503 >> v2:172.21.15.102:3300/0 conn(0x7fdd1c15baa0 0x7fdd1c15bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:31.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.596+0000 7fdd1be00640 1 -- 172.21.15.102:0/378990503 >> v2:172.21.15.102:3300/0 conn(0x7fdd1c15baa0 msgr2=0x7fdd1c15bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:31.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.596+0000 7fdd1be00640 1 --2- 172.21.15.102:0/378990503 >> v2:172.21.15.102:3300/0 conn(0x7fdd1c15baa0 0x7fdd1c15bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:31.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.596+0000 7fdd1b400640 1 -- 172.21.15.102:0/378990503 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1414377821 0 0) 0x7fdd1c15c470 con 0x7fdd1c15a310 2024-03-18T16:32:31.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.596+0000 7fdd1be00640 1 -- 172.21.15.102:0/378990503 >> v1:172.21.15.154:6789/0 conn(0x7fdd1c15a310 legacy=0x7fdd1c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:31.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.596+0000 7fdd1be00640 1 -- 172.21.15.102:0/378990503 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c15b050 con 0x7fdd1c15a6f0 2024-03-18T16:32:31.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.598+0000 7fdd1be00640 1 --2- 172.21.15.102:0/378990503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15a6f0 0x7fdd1c15ab10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdd080097a0 tx=0x7fdd0802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7b9f96446634e33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.602+0000 7fdd1b400640 1 -- 172.21.15.102:0/378990503 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08035020 con 0x7fdd1c15a6f0 2024-03-18T16:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.602+0000 7fdd1b400640 1 -- 172.21.15.102:0/378990503 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd08031810 con 0x7fdd1c15a6f0 2024-03-18T16:32:31.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.602+0000 7fdd1b400640 1 -- 172.21.15.102:0/378990503 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08031be0 con 0x7fdd1c15a6f0 2024-03-18T16:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.603+0000 7fdd21e00640 1 -- 172.21.15.102:0/378990503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15a6f0 msgr2=0x7fdd1c15ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.603+0000 7fdd21e00640 1 --2- 172.21.15.102:0/378990503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15a6f0 0x7fdd1c15ab10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdd080097a0 tx=0x7fdd0802b8b0 comp rx=0 tx=0).stop 2024-03-18T16:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.603+0000 7fdd21e00640 1 -- 172.21.15.102:0/378990503 shutdown_connections 2024-03-18T16:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.603+0000 7fdd21e00640 1 --2- 172.21.15.102:0/378990503 >> v2:172.21.15.102:3300/0 conn(0x7fdd1c15baa0 0x7fdd1c15bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.603+0000 7fdd21e00640 1 --2- 172.21.15.102:0/378990503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15a6f0 0x7fdd1c15ab10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.603+0000 7fdd21e00640 1 -- 172.21.15.102:0/378990503 >> 172.21.15.102:0/378990503 conn(0x7fdd1c08c1a0 msgr2=0x7fdd1c066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.603+0000 7fdd21e00640 1 -- 172.21.15.102:0/378990503 shutdown_connections 2024-03-18T16:32:31.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.603+0000 7fdd21e00640 1 -- 172.21.15.102:0/378990503 wait complete. 2024-03-18T16:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.605+0000 7fdd21e00640 1 Processor -- start 2024-03-18T16:32:31.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.605+0000 7fdd21e00640 1 -- start start 2024-03-18T16:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.606+0000 7fdd21e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd1c15a310 0x7fdd1c1f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.606+0000 7fdd21e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd1c15a6f0 0x7fdd1c0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.606+0000 7fdd21e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15baa0 0x7fdd1c0c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.606+0000 7fdd21e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c0cce50 con 0x7fdd1c15a310 2024-03-18T16:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.606+0000 7fdd21e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c0caa80 con 0x7fdd1c15a6f0 2024-03-18T16:32:31.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.606+0000 7fdd21e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1c0cd130 con 0x7fdd1c15baa0 2024-03-18T16:32:31.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.607+0000 7fdd20a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd1c15a310 0x7fdd1c1f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:31.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.607+0000 7fdd21400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15baa0 0x7fdd1c0c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:31.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.607+0000 7fdd1be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd1c15a6f0 0x7fdd1c0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:31.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.607+0000 7fdd20a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd1c15a310 0x7fdd1c1f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40140/0 (socket says 172.21.15.102:40140) 2024-03-18T16:32:31.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.607+0000 7fdd20a00640 1 -- 172.21.15.102:0/3376100893 learned_addr learned my addr 172.21.15.102:0/3376100893 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:31.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.611+0000 7fdd21400640 1 -- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd1c15a6f0 msgr2=0x7fdd1c0c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:31.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.611+0000 7fdd21400640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd1c15a6f0 0x7fdd1c0c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:31.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.611+0000 7fdd21400640 1 -- 172.21.15.102:0/3376100893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd1c15a310 msgr2=0x7fdd1c1f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:31.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.611+0000 7fdd21400640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd1c15a310 0x7fdd1c1f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:31.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.611+0000 7fdd21400640 1 -- 172.21.15.102:0/3376100893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c0cfb90 con 0x7fdd1c15baa0 2024-03-18T16:32:31.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.611+0000 7fdd20a00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd1c15a310 0x7fdd1c1f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:32:31.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.614+0000 7fdd21400640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15baa0 0x7fdd1c0c7c60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009d90 tx=0x7fdd1400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:31.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.618+0000 7fdd18c00640 1 -- 172.21.15.102:0/3376100893 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14018020 con 0x7fdd1c15baa0 2024-03-18T16:32:31.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.618+0000 7fdd18c00640 1 -- 172.21.15.102:0/3376100893 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd14014480 con 0x7fdd1c15baa0 2024-03-18T16:32:31.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.618+0000 7fdd18c00640 1 -- 172.21.15.102:0/3376100893 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd14014870 con 0x7fdd1c15baa0 2024-03-18T16:32:31.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.619+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1c158e00 con 0x7fdd1c15baa0 2024-03-18T16:32:31.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.619+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd1c1590a0 con 0x7fdd1c15baa0 2024-03-18T16:32:31.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.620+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd1c15e6c0 con 0x7fdd1c15baa0 2024-03-18T16:32:31.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.878+0000 7fdd18c00640 1 -- 172.21.15.102:0/3376100893 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdd14005d10 con 0x7fdd1c15baa0 2024-03-18T16:32:31.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.881+0000 7fdd18c00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcfc06bc20 0x7fdcfc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.881+0000 7fdd20a00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcfc06bc20 0x7fdcfc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.882+0000 7fdd20a00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcfc06bc20 0x7fdcfc06e0e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fdd10000f60 tx=0x7fdd10007ea0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.884+0000 7fdd18c00640 1 -- 172.21.15.102:0/3376100893 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fdd14058240 con 0x7fdd1c15baa0 2024-03-18T16:32:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:31.896+0000 7fdd18c00640 1 -- 172.21.15.102:0/3376100893 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdd1c15e6c0 con 0x7fdd1c15baa0 2024-03-18T16:32:32.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.186+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd1c074fd0 con 0x7fdd1c15baa0 2024-03-18T16:32:32.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.209+0000 7fdd18c00640 1 -- 172.21.15.102:0/3376100893 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fdd14022030 con 0x7fdd1c15baa0 2024-03-18T16:32:32.211 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:32.212 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:32.212 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:32.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcfc06bc20 msgr2=0x7fdcfc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:32.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcfc06bc20 0x7fdcfc06e0e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fdd10000f60 tx=0x7fdd10007ea0 comp rx=0 tx=0).stop 2024-03-18T16:32:32.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15baa0 msgr2=0x7fdd1c0c7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:32.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15baa0 0x7fdd1c0c7c60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009d90 tx=0x7fdd1400c490 comp rx=0 tx=0).stop 2024-03-18T16:32:32.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 shutdown_connections 2024-03-18T16:32:32.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdcfc06bc20 0x7fdcfc06e0e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:32.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdd1c15baa0 0x7fdd1c0c7c60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:32.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd1c15a6f0 0x7fdd1c0c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:32.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 --2- 172.21.15.102:0/3376100893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd1c15a310 0x7fdd1c1f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:32.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.213+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 >> 172.21.15.102:0/3376100893 conn(0x7fdd1c08c1a0 msgr2=0x7fdd1c153960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:32.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.214+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 shutdown_connections 2024-03-18T16:32:32.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:32.214+0000 7fdd21e00640 1 -- 172.21.15.102:0/3376100893 wait complete. 2024-03-18T16:32:32.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:32.837 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:33.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.008+0000 7fc3e5c00640 1 Processor -- start 2024-03-18T16:32:34.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.008+0000 7fc3e5c00640 1 -- start start 2024-03-18T16:32:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.009+0000 7fc3e5c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc3e006b470 0x7fc3e015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.009+0000 7fc3e5c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006de50 0x7fc3e006b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.009+0000 7fc3e5c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e00c94b0 con 0x7fc3e006bd90 2024-03-18T16:32:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.009+0000 7fc3e5c00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc3e00ca0a0 con 0x7fc3e006b470 2024-03-18T16:32:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.009+0000 7fc3e5c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e00c9770 con 0x7fc3e006de50 2024-03-18T16:32:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.010+0000 7fc3df400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006de50 0x7fc3e006b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.010+0000 7fc3dfe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc3e006b470 0x7fc3e015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.010+0000 7fc3e5200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc3e006bd90 0x7fc3e0154150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36934/0 (socket says 172.21.15.102:36934) 2024-03-18T16:32:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.010+0000 7fc3e5200640 1 -- 172.21.15.102:0/1850164295 learned_addr learned my addr 172.21.15.102:0/1850164295 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.013+0000 7fc3df400640 1 -- 172.21.15.102:0/1850164295 >> v2:172.21.15.154:3300/0 conn(0x7fc3e006b470 msgr2=0x7fc3e015ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.013+0000 7fc3df400640 1 --2- 172.21.15.102:0/1850164295 >> v2:172.21.15.154:3300/0 conn(0x7fc3e006b470 0x7fc3e015ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.013+0000 7fc3df400640 1 -- 172.21.15.102:0/1850164295 >> v1:172.21.15.102:6789/0 conn(0x7fc3e006bd90 legacy=0x7fc3e0154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:34.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.013+0000 7fc3df400640 1 -- 172.21.15.102:0/1850164295 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e00c9a50 con 0x7fc3e006de50 2024-03-18T16:32:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.016+0000 7fc3df400640 1 --2- 172.21.15.102:0/1850164295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006de50 0x7fc3e006b850 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0097a0 tx=0x7fc3cc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9af2abb77806f406 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:34.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.019+0000 7fc3dea00640 1 -- 172.21.15.102:0/1850164295 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc035020 con 0x7fc3e006de50 2024-03-18T16:32:34.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.020+0000 7fc3dea00640 1 -- 172.21.15.102:0/1850164295 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3cc031810 con 0x7fc3e006de50 2024-03-18T16:32:34.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.020+0000 7fc3dea00640 1 -- 172.21.15.102:0/1850164295 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc031be0 con 0x7fc3e006de50 2024-03-18T16:32:34.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.020+0000 7fc3e5c00640 1 -- 172.21.15.102:0/1850164295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006de50 msgr2=0x7fc3e006b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:34.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.020+0000 7fc3e5c00640 1 --2- 172.21.15.102:0/1850164295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006de50 0x7fc3e006b850 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0097a0 tx=0x7fc3cc02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.020+0000 7fc3e5c00640 1 -- 172.21.15.102:0/1850164295 shutdown_connections 2024-03-18T16:32:34.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.020+0000 7fc3e5c00640 1 --2- 172.21.15.102:0/1850164295 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006de50 0x7fc3e006b850 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.020+0000 7fc3e5c00640 1 --2- 172.21.15.102:0/1850164295 >> v2:172.21.15.154:3300/0 conn(0x7fc3e006b470 0x7fc3e015ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.020+0000 7fc3e5c00640 1 -- 172.21.15.102:0/1850164295 >> 172.21.15.102:0/1850164295 conn(0x7fc3e00727e0 msgr2=0x7fc3e0072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:34.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.021+0000 7fc3e5c00640 1 -- 172.21.15.102:0/1850164295 shutdown_connections 2024-03-18T16:32:34.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.021+0000 7fc3e5c00640 1 -- 172.21.15.102:0/1850164295 wait complete. 2024-03-18T16:32:34.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.022+0000 7fc3e5c00640 1 Processor -- start 2024-03-18T16:32:34.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.022+0000 7fc3e5c00640 1 -- start start 2024-03-18T16:32:34.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.023+0000 7fc3e5c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006b470 0x7fc3e01f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:34.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.023+0000 7fc3e5c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e006bd90 0x7fc3e0155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:34.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.023+0000 7fc3e5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e006de50 0x7fc3e0159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:34.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.023+0000 7fc3e5c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e01549e0 con 0x7fc3e006de50 2024-03-18T16:32:34.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.023+0000 7fc3e5c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e0154860 con 0x7fc3e006bd90 2024-03-18T16:32:34.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.023+0000 7fc3e5c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e00c5190 con 0x7fc3e006b470 2024-03-18T16:32:34.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.024+0000 7fc3df400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e006bd90 0x7fc3e0155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:34.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.024+0000 7fc3dfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006b470 0x7fc3e01f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:34.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.024+0000 7fc3df400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e006bd90 0x7fc3e0155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53748/0 (socket says 172.21.15.102:53748) 2024-03-18T16:32:34.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.024+0000 7fc3df400640 1 -- 172.21.15.102:0/2083930032 learned_addr learned my addr 172.21.15.102:0/2083930032 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:34.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.026+0000 7fc3e5200640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e006de50 0x7fc3e0159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.028+0000 7fc3df400640 1 -- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006b470 msgr2=0x7fc3e01f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.028+0000 7fc3df400640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006b470 0x7fc3e01f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.028+0000 7fc3df400640 1 -- 172.21.15.102:0/2083930032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e006de50 msgr2=0x7fc3e0159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.028+0000 7fc3df400640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e006de50 0x7fc3e0159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.028+0000 7fc3df400640 1 -- 172.21.15.102:0/2083930032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e01f34f0 con 0x7fc3e006bd90 2024-03-18T16:32:34.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.030+0000 7fc3df400640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e006bd90 0x7fc3e0155520 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc02bec0 tx=0x7fc3cc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:34.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.033+0000 7fc3d7e00640 1 -- 172.21.15.102:0/2083930032 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc035040 con 0x7fc3e006bd90 2024-03-18T16:32:34.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.033+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e01f37d0 con 0x7fc3e006bd90 2024-03-18T16:32:34.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.033+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e0154d20 con 0x7fc3e006bd90 2024-03-18T16:32:34.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.034+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e0074fd0 con 0x7fc3e006bd90 2024-03-18T16:32:34.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.035+0000 7fc3d7e00640 1 -- 172.21.15.102:0/2083930032 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3cc031810 con 0x7fc3e006bd90 2024-03-18T16:32:34.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.035+0000 7fc3d7e00640 1 -- 172.21.15.102:0/2083930032 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc039ce0 con 0x7fc3e006bd90 2024-03-18T16:32:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.316+0000 7fc3d7e00640 1 -- 172.21.15.102:0/2083930032 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc3cc038020 con 0x7fc3e006bd90 2024-03-18T16:32:34.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.318+0000 7fc3d7e00640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc3c006bbd0 0x7fc3c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:34.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.319+0000 7fc3dfe00640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc3c006bbd0 0x7fc3c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:34.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.319+0000 7fc3dfe00640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc3c006bbd0 0x7fc3c006e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009b40 tx=0x7fc3d0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.321+0000 7fc3d7e00640 1 -- 172.21.15.102:0/2083930032 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fc3cc0791b0 con 0x7fc3e006bd90 2024-03-18T16:32:34.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.337+0000 7fc3d7e00640 1 -- 172.21.15.102:0/2083930032 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc3cc07b110 con 0x7fc3e006bd90 2024-03-18T16:32:34.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.571+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3e00c6720 con 0x7fc3e006bd90 2024-03-18T16:32:34.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.584+0000 7fc3d7e00640 1 -- 172.21.15.102:0/2083930032 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fc3cc07f1b0 con 0x7fc3e006bd90 2024-03-18T16:32:34.586 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:34.586 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:34.586 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:34.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc3c006bbd0 msgr2=0x7fc3c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:34.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc3c006bbd0 0x7fc3c006e090 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009b40 tx=0x7fc3d0009190 comp rx=0 tx=0).stop 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e006bd90 msgr2=0x7fc3e0155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e006bd90 0x7fc3e0155520 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc02bec0 tx=0x7fc3cc0046c0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 shutdown_connections 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc3c006bbd0 0x7fc3c006e090 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e006de50 0x7fc3e0159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3e006bd90 0x7fc3e0155520 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 --2- 172.21.15.102:0/2083930032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc3e006b470 0x7fc3e01f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 >> 172.21.15.102:0/2083930032 conn(0x7fc3e00727e0 msgr2=0x7fc3e00cb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 shutdown_connections 2024-03-18T16:32:34.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:34.587+0000 7fc3e5c00640 1 -- 172.21.15.102:0/2083930032 wait complete. 2024-03-18T16:32:35.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:35.216 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:36.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:36.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.391+0000 7efe72200640 1 Processor -- start 2024-03-18T16:32:36.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.391+0000 7efe72200640 1 -- start start 2024-03-18T16:32:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.393+0000 7efe72200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efe6c06b470 0x7efe6c15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.393+0000 7efe72200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06de50 0x7efe6c06b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:36.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.393+0000 7efe72200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe6c06bd90 0x7efe6c154150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:36.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.393+0000 7efe72200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7efe6c0ca0a0 con 0x7efe6c06b470 2024-03-18T16:32:36.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.393+0000 7efe72200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efe6c0c94b0 con 0x7efe6c06bd90 2024-03-18T16:32:36.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.393+0000 7efe72200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efe6c0c9770 con 0x7efe6c06de50 2024-03-18T16:32:36.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.394+0000 7efe71800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe6c06bd90 0x7efe6c154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:36.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.394+0000 7efe6be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06de50 0x7efe6c06b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:36.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.394+0000 7efe71800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efe6c06bd90 0x7efe6c154150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53762/0 (socket says 172.21.15.102:53762) 2024-03-18T16:32:36.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.394+0000 7efe71800640 1 -- 172.21.15.102:0/764225863 learned_addr learned my addr 172.21.15.102:0/764225863 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:36.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.397+0000 7efe71800640 1 -- 172.21.15.102:0/764225863 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06de50 msgr2=0x7efe6c06b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:36.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.397+0000 7efe71800640 1 --2- 172.21.15.102:0/764225863 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06de50 0x7efe6c06b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.397+0000 7efe71800640 1 -- 172.21.15.102:0/764225863 >> v2:172.21.15.102:3300/0 conn(0x7efe6c06b470 msgr2=0x7efe6c15ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:36.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.397+0000 7efe71800640 1 --2- 172.21.15.102:0/764225863 >> v2:172.21.15.102:3300/0 conn(0x7efe6c06b470 0x7efe6c15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.397+0000 7efe71800640 1 -- 172.21.15.102:0/764225863 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c0c9a50 con 0x7efe6c06bd90 2024-03-18T16:32:36.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.399+0000 7efe71800640 1 --2- 172.21.15.102:0/764225863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06bd90 0x7efe6c154150 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7efe6c06f800 tx=0x7efe6402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc9e0d1b0667fd2e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:36.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.404+0000 7efe6b400640 1 -- 172.21.15.102:0/764225863 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6402ca80 con 0x7efe6c06bd90 2024-03-18T16:32:36.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.404+0000 7efe6b400640 1 -- 172.21.15.102:0/764225863 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe6402cc20 con 0x7efe6c06bd90 2024-03-18T16:32:36.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.404+0000 7efe6b400640 1 -- 172.21.15.102:0/764225863 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe640343f0 con 0x7efe6c06bd90 2024-03-18T16:32:36.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.404+0000 7efe72200640 1 -- 172.21.15.102:0/764225863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06bd90 msgr2=0x7efe6c154150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:36.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.404+0000 7efe72200640 1 --2- 172.21.15.102:0/764225863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06bd90 0x7efe6c154150 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7efe6c06f800 tx=0x7efe6402bee0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.405+0000 7efe72200640 1 -- 172.21.15.102:0/764225863 shutdown_connections 2024-03-18T16:32:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.405+0000 7efe72200640 1 --2- 172.21.15.102:0/764225863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06bd90 0x7efe6c154150 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.405+0000 7efe72200640 1 --2- 172.21.15.102:0/764225863 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06de50 0x7efe6c06b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.405+0000 7efe72200640 1 --2- 172.21.15.102:0/764225863 >> v2:172.21.15.102:3300/0 conn(0x7efe6c06b470 0x7efe6c15ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.405+0000 7efe72200640 1 -- 172.21.15.102:0/764225863 >> 172.21.15.102:0/764225863 conn(0x7efe6c0727e0 msgr2=0x7efe6c072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.405+0000 7efe72200640 1 -- 172.21.15.102:0/764225863 shutdown_connections 2024-03-18T16:32:36.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.405+0000 7efe72200640 1 -- 172.21.15.102:0/764225863 wait complete. 2024-03-18T16:32:36.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.406+0000 7efe72200640 1 Processor -- start 2024-03-18T16:32:36.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.407+0000 7efe72200640 1 -- start start 2024-03-18T16:32:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.407+0000 7efe72200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06b470 0x7efe6c15d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.407+0000 7efe72200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06bd90 0x7efe6c1f1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.407+0000 7efe72200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe6c06de50 0x7efe6c1f1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.407+0000 7efe72200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c154810 con 0x7efe6c06de50 2024-03-18T16:32:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.407+0000 7efe72200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c154690 con 0x7efe6c06b470 2024-03-18T16:32:36.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.407+0000 7efe72200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efe6c154990 con 0x7efe6c06bd90 2024-03-18T16:32:36.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.408+0000 7efe70e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06b470 0x7efe6c15d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:36.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.408+0000 7efe6be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06bd90 0x7efe6c1f1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:36.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.408+0000 7efe70e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06b470 0x7efe6c15d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53764/0 (socket says 172.21.15.102:53764) 2024-03-18T16:32:36.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.408+0000 7efe70e00640 1 -- 172.21.15.102:0/1425640760 learned_addr learned my addr 172.21.15.102:0/1425640760 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:36.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.412+0000 7efe70e00640 1 -- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06bd90 msgr2=0x7efe6c1f1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:36.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.412+0000 7efe70e00640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06bd90 0x7efe6c1f1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.412+0000 7efe70e00640 1 -- 172.21.15.102:0/1425640760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe6c06de50 msgr2=0x7efe6c1f1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:36.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.412+0000 7efe70e00640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe6c06de50 0x7efe6c1f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.412+0000 7efe70e00640 1 -- 172.21.15.102:0/1425640760 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c158480 con 0x7efe6c06b470 2024-03-18T16:32:36.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.415+0000 7efe70e00640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06b470 0x7efe6c15d850 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7efe6000b440 tx=0x7efe6000b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.420+0000 7efe68c00640 1 -- 172.21.15.102:0/1425640760 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe60015020 con 0x7efe6c06b470 2024-03-18T16:32:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.420+0000 7efe68c00640 1 -- 172.21.15.102:0/1425640760 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe60011890 con 0x7efe6c06b470 2024-03-18T16:32:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.420+0000 7efe68c00640 1 -- 172.21.15.102:0/1425640760 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe60011c80 con 0x7efe6c06b470 2024-03-18T16:32:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.420+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe6c15b4a0 con 0x7efe6c06b470 2024-03-18T16:32:36.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.420+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe6c15ba20 con 0x7efe6c06b470 2024-03-18T16:32:36.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.421+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe6c074fd0 con 0x7efe6c06b470 2024-03-18T16:32:36.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.448+0000 7efe68c00640 1 -- 172.21.15.102:0/1425640760 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efe60018020 con 0x7efe6c06b470 2024-03-18T16:32:36.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.449+0000 7efe68c00640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe4c06bc00 0x7efe4c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:36.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.450+0000 7efe6be00640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe4c06bc00 0x7efe4c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:36.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.450+0000 7efe6be00640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe4c06bc00 0x7efe4c06e0c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efe6c06bd10 tx=0x7efe58009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:36.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.453+0000 7efe68c00640 1 -- 172.21.15.102:0/1425640760 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7efe60057510 con 0x7efe6c06b470 2024-03-18T16:32:36.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.724+0000 7efe68c00640 1 -- 172.21.15.102:0/1425640760 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efe6001f020 con 0x7efe6c06b470 2024-03-18T16:32:36.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.967+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe6c152d20 con 0x7efe6c06b470 2024-03-18T16:32:36.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.989+0000 7efe68c00640 1 -- 172.21.15.102:0/1425640760 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7efe6c158480 con 0x7efe6c06b470 2024-03-18T16:32:36.991 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:36.991 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:36.991 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe4c06bc00 msgr2=0x7efe4c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe4c06bc00 0x7efe4c06e0c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efe6c06bd10 tx=0x7efe58009190 comp rx=0 tx=0).stop 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06b470 msgr2=0x7efe6c15d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06b470 0x7efe6c15d850 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7efe6000b440 tx=0x7efe6000b900 comp rx=0 tx=0).stop 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 shutdown_connections 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe4c06bc00 0x7efe4c06e0c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe6c06de50 0x7efe6c1f1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe6c06bd90 0x7efe6c1f1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 --2- 172.21.15.102:0/1425640760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe6c06b470 0x7efe6c15d850 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 >> 172.21.15.102:0/1425640760 conn(0x7efe6c0727e0 msgr2=0x7efe6c0cb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 shutdown_connections 2024-03-18T16:32:36.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:36.992+0000 7efe72200640 1 -- 172.21.15.102:0/1425640760 wait complete. 2024-03-18T16:32:37.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:37.616 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:38.617 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:38.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.791+0000 7f0358a00640 1 Processor -- start 2024-03-18T16:32:38.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.791+0000 7f0358a00640 1 -- start start 2024-03-18T16:32:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.792+0000 7f0358a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f035406de50 0x7f035415ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.792+0000 7f0358a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f03541545a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.792+0000 7f0358a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f035406bf60 con 0x7f035406b470 2024-03-18T16:32:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.792+0000 7f0358a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f03540ca0a0 con 0x7f035406de50 2024-03-18T16:32:38.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.792+0000 7f0358a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f03540c94b0 con 0x7f0354154150 2024-03-18T16:32:38.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.793+0000 7f0353e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f03541545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:38.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.793+0000 7f0352a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f035406de50 0x7f035415ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:38.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.793+0000 7f0353400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f035406b470 0x7f035406b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36944/0 (socket says 172.21.15.102:36944) 2024-03-18T16:32:38.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.793+0000 7f0353400640 1 -- 172.21.15.102:0/4029798978 learned_addr learned my addr 172.21.15.102:0/4029798978 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:38.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.796+0000 7f0353e00640 1 -- 172.21.15.102:0/4029798978 >> v2:172.21.15.154:3300/0 conn(0x7f035406de50 msgr2=0x7f035415ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:38.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.796+0000 7f0353e00640 1 --2- 172.21.15.102:0/4029798978 >> v2:172.21.15.154:3300/0 conn(0x7f035406de50 0x7f035415ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:38.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.796+0000 7f0353e00640 1 -- 172.21.15.102:0/4029798978 >> v1:172.21.15.102:6789/0 conn(0x7f035406b470 legacy=0x7f035406b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:38.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.796+0000 7f0353e00640 1 -- 172.21.15.102:0/4029798978 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03540c9770 con 0x7f0354154150 2024-03-18T16:32:38.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.798+0000 7f0353e00640 1 --2- 172.21.15.102:0/4029798978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f03541545a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f034c009d90 tx=0x7f034c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af9fe1f294d928ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:38.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.802+0000 7f0352000640 1 -- 172.21.15.102:0/4029798978 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c039070 con 0x7f0354154150 2024-03-18T16:32:38.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.802+0000 7f0352000640 1 -- 172.21.15.102:0/4029798978 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034c02cc60 con 0x7f0354154150 2024-03-18T16:32:38.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.802+0000 7f0352000640 1 -- 172.21.15.102:0/4029798978 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c034660 con 0x7f0354154150 2024-03-18T16:32:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.803+0000 7f0358a00640 1 -- 172.21.15.102:0/4029798978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 msgr2=0x7f03541545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.803+0000 7f0358a00640 1 --2- 172.21.15.102:0/4029798978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f03541545a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f034c009d90 tx=0x7f034c02beb0 comp rx=0 tx=0).stop 2024-03-18T16:32:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.803+0000 7f0358a00640 1 -- 172.21.15.102:0/4029798978 shutdown_connections 2024-03-18T16:32:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.803+0000 7f0358a00640 1 --2- 172.21.15.102:0/4029798978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f03541545a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.803+0000 7f0358a00640 1 --2- 172.21.15.102:0/4029798978 >> v2:172.21.15.154:3300/0 conn(0x7f035406de50 0x7f035415ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.803+0000 7f0358a00640 1 -- 172.21.15.102:0/4029798978 >> 172.21.15.102:0/4029798978 conn(0x7f03540727e0 msgr2=0x7f0354072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.803+0000 7f0358a00640 1 -- 172.21.15.102:0/4029798978 shutdown_connections 2024-03-18T16:32:38.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.803+0000 7f0358a00640 1 -- 172.21.15.102:0/4029798978 wait complete. 2024-03-18T16:32:38.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.806+0000 7f0358a00640 1 Processor -- start 2024-03-18T16:32:38.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.806+0000 7f0358a00640 1 -- start start 2024-03-18T16:32:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.807+0000 7f0358a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f035406b470 0x7f035415c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.807+0000 7f0358a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f035406de50 0x7f035414e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.807+0000 7f0358a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f0354158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.807+0000 7f0358a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f03540c5190 con 0x7f035406de50 2024-03-18T16:32:38.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.807+0000 7f0358a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f03540c9d30 con 0x7f035406b470 2024-03-18T16:32:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.807+0000 7f0358a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f03540c5310 con 0x7f0354154150 2024-03-18T16:32:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.808+0000 7f0353400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f035406b470 0x7f035415c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.808+0000 7f0353e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f0354158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.808+0000 7f0352a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f035406de50 0x7f035414e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.808+0000 7f0353e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f0354158f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:59586/0 (socket says 172.21.15.102:59586) 2024-03-18T16:32:38.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.808+0000 7f0353e00640 1 -- 172.21.15.102:0/3378646742 learned_addr learned my addr 172.21.15.102:0/3378646742 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.812+0000 7f0353e00640 1 -- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f035406b470 msgr2=0x7f035415c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.812+0000 7f0353e00640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f035406b470 0x7f035415c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.812+0000 7f0353e00640 1 -- 172.21.15.102:0/3378646742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f035406de50 msgr2=0x7f035414e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.812+0000 7f0353e00640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f035406de50 0x7f035414e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:38.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.812+0000 7f0353e00640 1 -- 172.21.15.102:0/3378646742 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035415c7e0 con 0x7f0354154150 2024-03-18T16:32:38.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.815+0000 7f0353e00640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f0354158f50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f034c009eb0 tx=0x7f034c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:38.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.818+0000 7f034b400640 1 -- 172.21.15.102:0/3378646742 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c039050 con 0x7f0354154150 2024-03-18T16:32:38.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.818+0000 7f034b400640 1 -- 172.21.15.102:0/3378646742 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034c045070 con 0x7f0354154150 2024-03-18T16:32:38.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.818+0000 7f034b400640 1 -- 172.21.15.102:0/3378646742 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c009450 con 0x7f0354154150 2024-03-18T16:32:38.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.818+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03541eff60 con 0x7f0354154150 2024-03-18T16:32:38.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.819+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f03541f0520 con 0x7f0354154150 2024-03-18T16:32:38.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.820+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0354074fd0 con 0x7f0354154150 2024-03-18T16:32:38.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.839+0000 7f034b400640 1 -- 172.21.15.102:0/3378646742 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f034c03e850 con 0x7f0354154150 2024-03-18T16:32:38.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.841+0000 7f034b400640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f033406bbd0 0x7f033406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:38.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.841+0000 7f0353400640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f033406bbd0 0x7f033406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:38.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:38.842+0000 7f0353400640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f033406bbd0 0x7f033406e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0344007fd0 tx=0x7f0344007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:39.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.084+0000 7f034b400640 1 -- 172.21.15.102:0/3378646742 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f034c07a7d0 con 0x7f0354154150 2024-03-18T16:32:39.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.098+0000 7f034b400640 1 -- 172.21.15.102:0/3378646742 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f034c07c730 con 0x7f0354154150 2024-03-18T16:32:39.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.402+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03541538a0 con 0x7f0354154150 2024-03-18T16:32:39.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.416+0000 7f034b400640 1 -- 172.21.15.102:0/3378646742 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f034c0807d0 con 0x7f0354154150 2024-03-18T16:32:39.418 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:39.418 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:39.418 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:39.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f033406bbd0 msgr2=0x7f033406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:39.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f033406bbd0 0x7f033406e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0344007fd0 tx=0x7f0344007e80 comp rx=0 tx=0).stop 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 msgr2=0x7f0354158f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f0354158f50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f034c009eb0 tx=0x7f034c004a40 comp rx=0 tx=0).stop 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 shutdown_connections 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f033406bbd0 0x7f033406e090 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0354154150 0x7f0354158f50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f035406de50 0x7f035414e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 --2- 172.21.15.102:0/3378646742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f035406b470 0x7f035415c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 >> 172.21.15.102:0/3378646742 conn(0x7f03540727e0 msgr2=0x7f035406b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 shutdown_connections 2024-03-18T16:32:39.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:39.418+0000 7f0358a00640 1 -- 172.21.15.102:0/3378646742 wait complete. 2024-03-18T16:32:40.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:40.041 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:41.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:41.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.218+0000 7fd5f5800640 1 Processor -- start 2024-03-18T16:32:41.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.219+0000 7fd5f5800640 1 -- start start 2024-03-18T16:32:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.220+0000 7fd5f5800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd5f00ca960 0x7fd5f006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.221+0000 7fd5f5800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f015a1a0 0x7fd5f00705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.221+0000 7fd5f5800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f006c9c0 con 0x7fd5f00cad40 2024-03-18T16:32:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.221+0000 7fd5f5800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd5f00c8680 con 0x7fd5f00ca960 2024-03-18T16:32:41.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.221+0000 7fd5f5800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f006cca0 con 0x7fd5f015a1a0 2024-03-18T16:32:41.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.222+0000 7fd5f4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f015a1a0 0x7fd5f00705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:41.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.222+0000 7fd5efe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd5f00ca960 0x7fd5f006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:41.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.222+0000 7fd5ef400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd5f00cad40 0x7fd5f015e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54832/0 (socket says 172.21.15.102:54832) 2024-03-18T16:32:41.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.222+0000 7fd5ef400640 1 -- 172.21.15.102:0/2294900789 learned_addr learned my addr 172.21.15.102:0/2294900789 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:41.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.224+0000 7fd5f4e00640 1 -- 172.21.15.102:0/2294900789 >> v2:172.21.15.102:3300/0 conn(0x7fd5f00ca960 msgr2=0x7fd5f006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:41.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.224+0000 7fd5f4e00640 1 --2- 172.21.15.102:0/2294900789 >> v2:172.21.15.102:3300/0 conn(0x7fd5f00ca960 0x7fd5f006c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:41.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.224+0000 7fd5f4e00640 1 -- 172.21.15.102:0/2294900789 >> v1:172.21.15.102:6789/0 conn(0x7fd5f00cad40 legacy=0x7fd5f015e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:41.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.224+0000 7fd5f4e00640 1 -- 172.21.15.102:0/2294900789 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f006cf80 con 0x7fd5f015a1a0 2024-03-18T16:32:41.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.227+0000 7fd5f4e00640 1 --2- 172.21.15.102:0/2294900789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f015a1a0 0x7fd5f00705a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009d90 tx=0x7fd5e802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be03466819fca3ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:41.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.231+0000 7fd5eea00640 1 -- 172.21.15.102:0/2294900789 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8039070 con 0x7fd5f015a1a0 2024-03-18T16:32:41.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.231+0000 7fd5eea00640 1 -- 172.21.15.102:0/2294900789 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e802cce0 con 0x7fd5f015a1a0 2024-03-18T16:32:41.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.231+0000 7fd5eea00640 1 -- 172.21.15.102:0/2294900789 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8034660 con 0x7fd5f015a1a0 2024-03-18T16:32:41.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.232+0000 7fd5f5800640 1 -- 172.21.15.102:0/2294900789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f015a1a0 msgr2=0x7fd5f00705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:41.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.232+0000 7fd5f5800640 1 --2- 172.21.15.102:0/2294900789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f015a1a0 0x7fd5f00705a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009d90 tx=0x7fd5e802bf30 comp rx=0 tx=0).stop 2024-03-18T16:32:41.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.232+0000 7fd5f5800640 1 -- 172.21.15.102:0/2294900789 shutdown_connections 2024-03-18T16:32:41.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.232+0000 7fd5f5800640 1 --2- 172.21.15.102:0/2294900789 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f015a1a0 0x7fd5f00705a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:41.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.232+0000 7fd5f5800640 1 --2- 172.21.15.102:0/2294900789 >> v2:172.21.15.102:3300/0 conn(0x7fd5f00ca960 0x7fd5f006c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:41.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.232+0000 7fd5f5800640 1 -- 172.21.15.102:0/2294900789 >> 172.21.15.102:0/2294900789 conn(0x7fd5f0067190 msgr2=0x7fd5f00675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:41.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.232+0000 7fd5f5800640 1 -- 172.21.15.102:0/2294900789 shutdown_connections 2024-03-18T16:32:41.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.233+0000 7fd5f5800640 1 -- 172.21.15.102:0/2294900789 wait complete. 2024-03-18T16:32:41.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.235+0000 7fd5f5800640 1 Processor -- start 2024-03-18T16:32:41.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.235+0000 7fd5f5800640 1 -- start start 2024-03-18T16:32:41.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.237+0000 7fd5f5800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f00ca960 0x7fd5f01f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:41.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.237+0000 7fd5f5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5f00cad40 0x7fd5f00cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:41.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.237+0000 7fd5f5800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5f015a1a0 0x7fd5f0158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:41.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.237+0000 7fd5f5800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f0070ae0 con 0x7fd5f00cad40 2024-03-18T16:32:41.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.237+0000 7fd5f5800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f015dc10 con 0x7fd5f015a1a0 2024-03-18T16:32:41.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.237+0000 7fd5f5800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f0070c60 con 0x7fd5f00ca960 2024-03-18T16:32:41.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.237+0000 7fd5efe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f00ca960 0x7fd5f01f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:41.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.238+0000 7fd5f4e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5f015a1a0 0x7fd5f0158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:41.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.238+0000 7fd5ef400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5f00cad40 0x7fd5f00cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:41.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.238+0000 7fd5efe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f00ca960 0x7fd5f01f0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43528/0 (socket says 172.21.15.102:43528) 2024-03-18T16:32:41.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.238+0000 7fd5efe00640 1 -- 172.21.15.102:0/3841297869 learned_addr learned my addr 172.21.15.102:0/3841297869 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:41.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.243+0000 7fd5f4e00640 1 -- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f00ca960 msgr2=0x7fd5f01f0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:41.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.243+0000 7fd5f4e00640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f00ca960 0x7fd5f01f0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:41.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.243+0000 7fd5f4e00640 1 -- 172.21.15.102:0/3841297869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5f00cad40 msgr2=0x7fd5f00cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:41.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.243+0000 7fd5f4e00640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5f00cad40 0x7fd5f00cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:41.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.243+0000 7fd5f4e00640 1 -- 172.21.15.102:0/3841297869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f0157630 con 0x7fd5f015a1a0 2024-03-18T16:32:41.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.246+0000 7fd5f4e00640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5f015a1a0 0x7fd5f0158d90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8033000 tx=0x7fd5e8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:41.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.248+0000 7fd5e7e00640 1 -- 172.21.15.102:0/3841297869 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8039050 con 0x7fd5f015a1a0 2024-03-18T16:32:41.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.248+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f0157910 con 0x7fd5f015a1a0 2024-03-18T16:32:41.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.248+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f01f36e0 con 0x7fd5f015a1a0 2024-03-18T16:32:41.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.249+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f006dd30 con 0x7fd5f015a1a0 2024-03-18T16:32:41.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.511+0000 7fd5e7e00640 1 -- 172.21.15.102:0/3841297869 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e8045070 con 0x7fd5f015a1a0 2024-03-18T16:32:41.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.511+0000 7fd5e7e00640 1 -- 172.21.15.102:0/3841297869 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8009450 con 0x7fd5f015a1a0 2024-03-18T16:32:41.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.513+0000 7fd5e7e00640 1 -- 172.21.15.102:0/3841297869 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd5e803e890 con 0x7fd5f015a1a0 2024-03-18T16:32:41.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.516+0000 7fd5e7e00640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd5d006bbd0 0x7fd5d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:41.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.516+0000 7fd5e7e00640 1 -- 172.21.15.102:0/3841297869 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd5e80ae4c0 con 0x7fd5f015a1a0 2024-03-18T16:32:41.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.517+0000 7fd5efe00640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd5d006bbd0 0x7fd5d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:41.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.517+0000 7fd5efe00640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd5d006bbd0 0x7fd5d006e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009b60 tx=0x7fd5e0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:41.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.526+0000 7fd5e7e00640 1 -- 172.21.15.102:0/3841297869 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd5f006dd30 con 0x7fd5f015a1a0 2024-03-18T16:32:41.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.816+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5f0074fd0 con 0x7fd5f015a1a0 2024-03-18T16:32:41.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.830+0000 7fd5e7e00640 1 -- 172.21.15.102:0/3841297869 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd5e807c510 con 0x7fd5f015a1a0 2024-03-18T16:32:41.832 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:41.832 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:41.833 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:41.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd5d006bbd0 msgr2=0x7fd5d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:41.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd5d006bbd0 0x7fd5d006e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009b60 tx=0x7fd5e0009190 comp rx=0 tx=0).stop 2024-03-18T16:32:41.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5f015a1a0 msgr2=0x7fd5f0158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:41.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5f015a1a0 0x7fd5f0158d90 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8033000 tx=0x7fd5e8004a40 comp rx=0 tx=0).stop 2024-03-18T16:32:41.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 shutdown_connections 2024-03-18T16:32:41.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd5d006bbd0 0x7fd5d006e090 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:41.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5f015a1a0 0x7fd5f0158d90 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:41.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5f00cad40 0x7fd5f00cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:41.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 --2- 172.21.15.102:0/3841297869 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd5f00ca960 0x7fd5f01f0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:41.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 >> 172.21.15.102:0/3841297869 conn(0x7fd5f0067190 msgr2=0x7fd5f0142500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:41.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 shutdown_connections 2024-03-18T16:32:41.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:41.834+0000 7fd5f5800640 1 -- 172.21.15.102:0/3841297869 wait complete. 2024-03-18T16:32:42.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:42.465 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:43.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:43.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.639+0000 7f345a200640 1 Processor -- start 2024-03-18T16:32:43.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.639+0000 7f345a200640 1 -- start start 2024-03-18T16:32:43.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.641+0000 7f345a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406de50 0x7f345415ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:43.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.641+0000 7f345a200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3454154150 0x7f34541545c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:43.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.641+0000 7f345a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f345406bf60 con 0x7f345406b470 2024-03-18T16:32:43.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.641+0000 7f345a200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f34540ca0a0 con 0x7f3454154150 2024-03-18T16:32:43.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.641+0000 7f345a200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f34540c94b0 con 0x7f345406de50 2024-03-18T16:32:43.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.642+0000 7f3453e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406de50 0x7f345415ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:43.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.642+0000 7f3453e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406de50 0x7f345415ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43534/0 (socket says 172.21.15.102:43534) 2024-03-18T16:32:43.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.642+0000 7f3453e00640 1 -- 172.21.15.102:0/1363410271 learned_addr learned my addr 172.21.15.102:0/1363410271 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:43.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.642+0000 7f3459800640 1 --2- 172.21.15.102:0/1363410271 >> v2:172.21.15.102:3300/0 conn(0x7f3454154150 0x7f34541545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:43.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.644+0000 7f3453e00640 1 -- 172.21.15.102:0/1363410271 >> v2:172.21.15.102:3300/0 conn(0x7f3454154150 msgr2=0x7f34541545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:43.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.644+0000 7f3453e00640 1 --2- 172.21.15.102:0/1363410271 >> v2:172.21.15.102:3300/0 conn(0x7f3454154150 0x7f34541545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:43.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.644+0000 7f3453e00640 1 -- 172.21.15.102:0/1363410271 >> v1:172.21.15.102:6789/0 conn(0x7f345406b470 legacy=0x7f345406b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:43.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.644+0000 7f3453e00640 1 -- 172.21.15.102:0/1363410271 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34540c9770 con 0x7f345406de50 2024-03-18T16:32:43.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.647+0000 7f3453e00640 1 --2- 172.21.15.102:0/1363410271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406de50 0x7f345415ad60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3440009870 tx=0x7f344002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b1b69e55d1d6a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.650+0000 7f3453400640 1 -- 172.21.15.102:0/1363410271 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3440034070 con 0x7f345406de50 2024-03-18T16:32:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.650+0000 7f3453400640 1 -- 172.21.15.102:0/1363410271 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3440030840 con 0x7f345406de50 2024-03-18T16:32:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.650+0000 7f3453400640 1 -- 172.21.15.102:0/1363410271 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3440030c10 con 0x7f345406de50 2024-03-18T16:32:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.651+0000 7f345a200640 1 -- 172.21.15.102:0/1363410271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406de50 msgr2=0x7f345415ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.651+0000 7f345a200640 1 --2- 172.21.15.102:0/1363410271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406de50 0x7f345415ad60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3440009870 tx=0x7f344002b980 comp rx=0 tx=0).stop 2024-03-18T16:32:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.652+0000 7f345a200640 1 -- 172.21.15.102:0/1363410271 shutdown_connections 2024-03-18T16:32:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.652+0000 7f345a200640 1 --2- 172.21.15.102:0/1363410271 >> v2:172.21.15.102:3300/0 conn(0x7f3454154150 0x7f34541545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.652+0000 7f345a200640 1 --2- 172.21.15.102:0/1363410271 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406de50 0x7f345415ad60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.652+0000 7f345a200640 1 -- 172.21.15.102:0/1363410271 >> 172.21.15.102:0/1363410271 conn(0x7f34540727e0 msgr2=0x7f3454072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.652+0000 7f345a200640 1 -- 172.21.15.102:0/1363410271 shutdown_connections 2024-03-18T16:32:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.652+0000 7f345a200640 1 -- 172.21.15.102:0/1363410271 wait complete. 2024-03-18T16:32:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.654+0000 7f345a200640 1 Processor -- start 2024-03-18T16:32:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.655+0000 7f345a200640 1 -- start start 2024-03-18T16:32:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.656+0000 7f345a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406b470 0x7f34541557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.656+0000 7f345a200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345406de50 0x7f34541e1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.656+0000 7f345a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454154150 0x7f3454159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.656+0000 7f345a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f34540c5190 con 0x7f3454154150 2024-03-18T16:32:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.656+0000 7f345a200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34540c9d30 con 0x7f345406de50 2024-03-18T16:32:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.656+0000 7f345a200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f34540c5310 con 0x7f345406b470 2024-03-18T16:32:43.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.657+0000 7f3458e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406b470 0x7f34541557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:43.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.657+0000 7f3459800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454154150 0x7f3454159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:43.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.657+0000 7f3453e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345406de50 0x7f34541e1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:43.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.657+0000 7f3458e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406b470 0x7f34541557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43550/0 (socket says 172.21.15.102:43550) 2024-03-18T16:32:43.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.657+0000 7f3458e00640 1 -- 172.21.15.102:0/1795632640 learned_addr learned my addr 172.21.15.102:0/1795632640 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:43.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.661+0000 7f3458e00640 1 -- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345406de50 msgr2=0x7f34541e1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:43.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.661+0000 7f3458e00640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345406de50 0x7f34541e1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:43.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.661+0000 7f3458e00640 1 -- 172.21.15.102:0/1795632640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454154150 msgr2=0x7f3454159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:43.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.661+0000 7f3458e00640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454154150 0x7f3454159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:43.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.661+0000 7f3458e00640 1 -- 172.21.15.102:0/1795632640 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34541577d0 con 0x7f345406b470 2024-03-18T16:32:43.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.663+0000 7f3458e00640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406b470 0x7f34541557f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f344800c430 tx=0x7f344800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:43.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.667+0000 7f3450c00640 1 -- 172.21.15.102:0/1795632640 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34480052d0 con 0x7f345406b470 2024-03-18T16:32:43.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.667+0000 7f3450c00640 1 -- 172.21.15.102:0/1795632640 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3448005490 con 0x7f345406b470 2024-03-18T16:32:43.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.667+0000 7f3450c00640 1 -- 172.21.15.102:0/1795632640 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3448007260 con 0x7f345406b470 2024-03-18T16:32:43.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.667+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34541e1040 con 0x7f345406b470 2024-03-18T16:32:43.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.668+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34541e0560 con 0x7f345406b470 2024-03-18T16:32:43.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.669+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3454074fd0 con 0x7f345406b470 2024-03-18T16:32:43.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.930+0000 7f3450c00640 1 -- 172.21.15.102:0/1795632640 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3448007400 con 0x7f345406b470 2024-03-18T16:32:43.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.931+0000 7f3450c00640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f343406bc20 0x7f343406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.931+0000 7f3453e00640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f343406bc20 0x7f343406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:43.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.932+0000 7f3453e00640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f343406bc20 0x7f343406e0e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3440036000 tx=0x7f3440008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:43.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.932+0000 7f3450c00640 1 -- 172.21.15.102:0/1795632640 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3448056980 con 0x7f345406b470 2024-03-18T16:32:43.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:43.944+0000 7f3450c00640 1 -- 172.21.15.102:0/1795632640 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f34480588e0 con 0x7f345406b470 2024-03-18T16:32:44.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.279+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34540c3e40 con 0x7f345406b470 2024-03-18T16:32:44.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.291+0000 7f3450c00640 1 -- 172.21.15.102:0/1795632640 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f344805c980 con 0x7f345406b470 2024-03-18T16:32:44.294 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:44.294 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:44.295 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f343406bc20 msgr2=0x7f343406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f343406bc20 0x7f343406e0e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3440036000 tx=0x7f3440008030 comp rx=0 tx=0).stop 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406b470 msgr2=0x7f34541557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406b470 0x7f34541557f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f344800c430 tx=0x7f344800c8f0 comp rx=0 tx=0).stop 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 shutdown_connections 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f343406bc20 0x7f343406e0e0 secure :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3440036000 tx=0x7f3440008030 comp rx=0 tx=0).stop 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454154150 0x7f3454159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f345406de50 0x7f34541e1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 --2- 172.21.15.102:0/1795632640 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f345406b470 0x7f34541557f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 >> 172.21.15.102:0/1795632640 conn(0x7f34540727e0 msgr2=0x7f34540c67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 shutdown_connections 2024-03-18T16:32:44.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:44.294+0000 7f345a200640 1 -- 172.21.15.102:0/1795632640 wait complete. 2024-03-18T16:32:44.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:44.918 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:45.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:46.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.098+0000 7fefe8e00640 1 Processor -- start 2024-03-18T16:32:46.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.098+0000 7fefe8e00640 1 -- start start 2024-03-18T16:32:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.099+0000 7fefe8e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40cfe70 0x7fefe415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.099+0000 7fefe8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fefe40cc2f0 0x7fefe40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.099+0000 7fefe8e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe40ccca0 con 0x7fefe40d0250 2024-03-18T16:32:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.100+0000 7fefe8e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fefe40cabd0 con 0x7fefe40cc2f0 2024-03-18T16:32:46.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.100+0000 7fefe8e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fefe40caea0 con 0x7fefe40cfe70 2024-03-18T16:32:46.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.100+0000 7fefe2a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fefe40d0250 0x7fefe40c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:43542/0 (socket says 172.21.15.102:43542) 2024-03-18T16:32:46.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.100+0000 7fefe2a00640 1 -- 172.21.15.102:0/4101289063 learned_addr learned my addr 172.21.15.102:0/4101289063 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:46.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.100+0000 7fefe3400640 1 --2- 172.21.15.102:0/4101289063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40cfe70 0x7fefe415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:46.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.101+0000 7fefe3e00640 1 --2- 172.21.15.102:0/4101289063 >> v2:172.21.15.102:3300/0 conn(0x7fefe40cc2f0 0x7fefe40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:46.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.103+0000 7fefe3400640 1 -- 172.21.15.102:0/4101289063 >> v2:172.21.15.102:3300/0 conn(0x7fefe40cc2f0 msgr2=0x7fefe40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:46.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.103+0000 7fefe3400640 1 --2- 172.21.15.102:0/4101289063 >> v2:172.21.15.102:3300/0 conn(0x7fefe40cc2f0 0x7fefe40cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:46.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.103+0000 7fefe3400640 1 -- 172.21.15.102:0/4101289063 >> v1:172.21.15.154:6789/0 conn(0x7fefe40d0250 legacy=0x7fefe40c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:46.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.103+0000 7fefe3400640 1 -- 172.21.15.102:0/4101289063 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe40cb180 con 0x7fefe40cfe70 2024-03-18T16:32:46.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.106+0000 7fefe3400640 1 --2- 172.21.15.102:0/4101289063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40cfe70 0x7fefe415ae40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fefd40097a0 tx=0x7fefd402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeadec84fe512dd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:46.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.109+0000 7fefe2000640 1 -- 172.21.15.102:0/4101289063 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4034070 con 0x7fefe40cfe70 2024-03-18T16:32:46.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.110+0000 7fefe2000640 1 -- 172.21.15.102:0/4101289063 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefd4030800 con 0x7fefe40cfe70 2024-03-18T16:32:46.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.110+0000 7fefe2000640 1 -- 172.21.15.102:0/4101289063 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4030bd0 con 0x7fefe40cfe70 2024-03-18T16:32:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.110+0000 7fefe8e00640 1 -- 172.21.15.102:0/4101289063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40cfe70 msgr2=0x7fefe415ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.110+0000 7fefe8e00640 1 --2- 172.21.15.102:0/4101289063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40cfe70 0x7fefe415ae40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fefd40097a0 tx=0x7fefd402b940 comp rx=0 tx=0).stop 2024-03-18T16:32:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.110+0000 7fefe8e00640 1 -- 172.21.15.102:0/4101289063 shutdown_connections 2024-03-18T16:32:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.110+0000 7fefe8e00640 1 --2- 172.21.15.102:0/4101289063 >> v2:172.21.15.102:3300/0 conn(0x7fefe40cc2f0 0x7fefe40cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.110+0000 7fefe8e00640 1 --2- 172.21.15.102:0/4101289063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40cfe70 0x7fefe415ae40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.110+0000 7fefe8e00640 1 -- 172.21.15.102:0/4101289063 >> 172.21.15.102:0/4101289063 conn(0x7fefe408c3e0 msgr2=0x7fefe406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.110+0000 7fefe8e00640 1 -- 172.21.15.102:0/4101289063 shutdown_connections 2024-03-18T16:32:46.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.111+0000 7fefe8e00640 1 -- 172.21.15.102:0/4101289063 wait complete. 2024-03-18T16:32:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.113+0000 7fefe8e00640 1 Processor -- start 2024-03-18T16:32:46.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.113+0000 7fefe8e00640 1 -- start start 2024-03-18T16:32:46.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.115+0000 7fefe8e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40cc2f0 0x7fefe4156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:46.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.115+0000 7fefe8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefe40cfe70 0x7fefe4156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:46.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.115+0000 7fefe8e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40d0250 0x7fefe41f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:46.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.115+0000 7fefe8e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fefe40cba20 con 0x7fefe40cfe70 2024-03-18T16:32:46.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.115+0000 7fefe8e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefe40cb740 con 0x7fefe40cc2f0 2024-03-18T16:32:46.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.115+0000 7fefe8e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fefe40cbd00 con 0x7fefe40d0250 2024-03-18T16:32:46.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.115+0000 7fefe3400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40cc2f0 0x7fefe4156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:46.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.116+0000 7fefe3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40d0250 0x7fefe41f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:46.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.116+0000 7fefe2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefe40cfe70 0x7fefe4156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:46.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.116+0000 7fefe3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40d0250 0x7fefe41f3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43564/0 (socket says 172.21.15.102:43564) 2024-03-18T16:32:46.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.116+0000 7fefe3e00640 1 -- 172.21.15.102:0/3952786482 learned_addr learned my addr 172.21.15.102:0/3952786482 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:46.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.119+0000 7fefe3e00640 1 -- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40cc2f0 msgr2=0x7fefe4156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:46.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.119+0000 7fefe3e00640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40cc2f0 0x7fefe4156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:46.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.119+0000 7fefe3e00640 1 -- 172.21.15.102:0/3952786482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefe40cfe70 msgr2=0x7fefe4156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:46.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.119+0000 7fefe3e00640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefe40cfe70 0x7fefe4156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:46.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.119+0000 7fefe3e00640 1 -- 172.21.15.102:0/3952786482 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe41e1960 con 0x7fefe40d0250 2024-03-18T16:32:46.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.122+0000 7fefe3e00640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40d0250 0x7fefe41f3540 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00efc0 tx=0x7fefdc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:46.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.125+0000 7fefdb400640 1 -- 172.21.15.102:0/3952786482 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc018020 con 0x7fefe40d0250 2024-03-18T16:32:46.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.125+0000 7fefdb400640 1 -- 172.21.15.102:0/3952786482 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefdc014480 con 0x7fefe40d0250 2024-03-18T16:32:46.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.125+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe41e1c40 con 0x7fefe40d0250 2024-03-18T16:32:46.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.125+0000 7fefdb400640 1 -- 172.21.15.102:0/3952786482 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc014870 con 0x7fefe40d0250 2024-03-18T16:32:46.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.125+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe4156290 con 0x7fefe40d0250 2024-03-18T16:32:46.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.126+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe40735d0 con 0x7fefe40d0250 2024-03-18T16:32:46.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.387+0000 7fefdb400640 1 -- 172.21.15.102:0/3952786482 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fefdc005d10 con 0x7fefe40d0250 2024-03-18T16:32:46.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.389+0000 7fefdb400640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefc406bc20 0x7fefc406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:46.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.389+0000 7fefe3400640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefc406bc20 0x7fefc406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:46.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.390+0000 7fefe3400640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefc406bc20 0x7fefc406e0e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fefd4036000 tx=0x7fefd403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:46.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.390+0000 7fefdb400640 1 -- 172.21.15.102:0/3952786482 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fefdc0584e0 con 0x7fefe40d0250 2024-03-18T16:32:46.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.401+0000 7fefdb400640 1 -- 172.21.15.102:0/3952786482 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fefe40735d0 con 0x7fefe40d0250 2024-03-18T16:32:46.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.656+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefe4074fd0 con 0x7fefe40d0250 2024-03-18T16:32:46.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.668+0000 7fefdb400640 1 -- 172.21.15.102:0/3952786482 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fefdc022020 con 0x7fefe40d0250 2024-03-18T16:32:46.671 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:46.672 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:46.672 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:46.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefc406bc20 msgr2=0x7fefc406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:46.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefc406bc20 0x7fefc406e0e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fefd4036000 tx=0x7fefd403b040 comp rx=0 tx=0).stop 2024-03-18T16:32:46.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40d0250 msgr2=0x7fefe41f3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:46.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40d0250 0x7fefe41f3540 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00efc0 tx=0x7fefdc00c490 comp rx=0 tx=0).stop 2024-03-18T16:32:46.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 shutdown_connections 2024-03-18T16:32:46.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefc406bc20 0x7fefc406e0e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:46.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fefe40d0250 0x7fefe41f3540 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:46.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefe40cfe70 0x7fefe4156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:46.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 --2- 172.21.15.102:0/3952786482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe40cc2f0 0x7fefe4156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:46.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 >> 172.21.15.102:0/3952786482 conn(0x7fefe408c3e0 msgr2=0x7fefe40c63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:46.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 shutdown_connections 2024-03-18T16:32:46.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:46.671+0000 7fefe8e00640 1 -- 172.21.15.102:0/3952786482 wait complete. 2024-03-18T16:32:47.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:47.295 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:48.295 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:48.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.470+0000 7f70f9000640 1 Processor -- start 2024-03-18T16:32:48.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.470+0000 7f70f9000640 1 -- start start 2024-03-18T16:32:48.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.471+0000 7f70f9000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40cfe70 0x7f70f415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:48.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.471+0000 7f70f9000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f70f40d0250 0x7f70f40c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:48.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.471+0000 7f70f9000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f70f40cc2f0 0x7f70f40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:48.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.471+0000 7f70f9000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f70f40ccca0 con 0x7f70f40cc2f0 2024-03-18T16:32:48.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.471+0000 7f70f9000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f70f40cabd0 con 0x7f70f40d0250 2024-03-18T16:32:48.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.471+0000 7f70f9000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f70f40caea0 con 0x7f70f40cfe70 2024-03-18T16:32:48.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.472+0000 7f70f2a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f70f40d0250 0x7f70f40c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:48.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.472+0000 7f70f3400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40cfe70 0x7f70f415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:48.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.473+0000 7f70f2a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f70f40d0250 0x7f70f40c65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48634/0 (socket says 172.21.15.102:48634) 2024-03-18T16:32:48.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.473+0000 7f70f2a00640 1 -- 172.21.15.102:0/1208456837 learned_addr learned my addr 172.21.15.102:0/1208456837 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:48.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.475+0000 7f70f2a00640 1 -- 172.21.15.102:0/1208456837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40cfe70 msgr2=0x7f70f415ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:48.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.475+0000 7f70f2a00640 1 --2- 172.21.15.102:0/1208456837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40cfe70 0x7f70f415ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:48.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.475+0000 7f70f2a00640 1 -- 172.21.15.102:0/1208456837 >> v2:172.21.15.102:3300/0 conn(0x7f70f40cc2f0 msgr2=0x7f70f40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:48.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.475+0000 7f70f2a00640 1 --2- 172.21.15.102:0/1208456837 >> v2:172.21.15.102:3300/0 conn(0x7f70f40cc2f0 0x7f70f40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:48.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.475+0000 7f70f2a00640 1 -- 172.21.15.102:0/1208456837 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f40cb180 con 0x7f70f40d0250 2024-03-18T16:32:48.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.477+0000 7f70f2a00640 1 --2- 172.21.15.102:0/1208456837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40d0250 0x7f70f40c65e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f70f40cd830 tx=0x7f70e002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53d61a694b68fb97 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.481+0000 7f70f2000640 1 -- 172.21.15.102:0/1208456837 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0034070 con 0x7f70f40d0250 2024-03-18T16:32:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.481+0000 7f70f2000640 1 -- 172.21.15.102:0/1208456837 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70e0030860 con 0x7f70f40d0250 2024-03-18T16:32:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.481+0000 7f70f2000640 1 -- 172.21.15.102:0/1208456837 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0030c30 con 0x7f70f40d0250 2024-03-18T16:32:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.482+0000 7f70f9000640 1 -- 172.21.15.102:0/1208456837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40d0250 msgr2=0x7f70f40c65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:48.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.482+0000 7f70f9000640 1 --2- 172.21.15.102:0/1208456837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40d0250 0x7f70f40c65e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f70f40cd830 tx=0x7f70e002b9a0 comp rx=0 tx=0).stop 2024-03-18T16:32:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.482+0000 7f70f9000640 1 -- 172.21.15.102:0/1208456837 shutdown_connections 2024-03-18T16:32:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.482+0000 7f70f9000640 1 --2- 172.21.15.102:0/1208456837 >> v2:172.21.15.102:3300/0 conn(0x7f70f40cc2f0 0x7f70f40cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.482+0000 7f70f9000640 1 --2- 172.21.15.102:0/1208456837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40d0250 0x7f70f40c65e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.482+0000 7f70f9000640 1 --2- 172.21.15.102:0/1208456837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40cfe70 0x7f70f415ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.482+0000 7f70f9000640 1 -- 172.21.15.102:0/1208456837 >> 172.21.15.102:0/1208456837 conn(0x7f70f408c3e0 msgr2=0x7f70f406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.482+0000 7f70f9000640 1 -- 172.21.15.102:0/1208456837 shutdown_connections 2024-03-18T16:32:48.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.482+0000 7f70f9000640 1 -- 172.21.15.102:0/1208456837 wait complete. 2024-03-18T16:32:48.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.484+0000 7f70f9000640 1 Processor -- start 2024-03-18T16:32:48.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.484+0000 7f70f9000640 1 -- start start 2024-03-18T16:32:48.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.485+0000 7f70f9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70f40cc2f0 0x7f70f41e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:48.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.485+0000 7f70f9000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40cfe70 0x7f70f414fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:48.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.485+0000 7f70f9000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40d0250 0x7f70f41e1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:48.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.485+0000 7f70f9000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f70f40cba20 con 0x7f70f40cc2f0 2024-03-18T16:32:48.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.485+0000 7f70f9000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70f40cb740 con 0x7f70f40cfe70 2024-03-18T16:32:48.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.485+0000 7f70f9000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f70f40cbd00 con 0x7f70f40d0250 2024-03-18T16:32:48.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.486+0000 7f70f3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40d0250 0x7f70f41e1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:48.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.486+0000 7f70f2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40cfe70 0x7f70f414fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:48.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.486+0000 7f70f3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40d0250 0x7f70f41e1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:43574/0 (socket says 172.21.15.102:43574) 2024-03-18T16:32:48.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.486+0000 7f70f3e00640 1 -- 172.21.15.102:0/362849805 learned_addr learned my addr 172.21.15.102:0/362849805 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:48.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.490+0000 7f70f2a00640 1 -- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40d0250 msgr2=0x7f70f41e1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:48.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.490+0000 7f70f2a00640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40d0250 0x7f70f41e1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:48.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.490+0000 7f70f2a00640 1 -- 172.21.15.102:0/362849805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70f40cc2f0 msgr2=0x7f70f41e03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:48.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.490+0000 7f70f2a00640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70f40cc2f0 0x7f70f41e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:48.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.490+0000 7f70f2a00640 1 -- 172.21.15.102:0/362849805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f41f30f0 con 0x7f70f40cfe70 2024-03-18T16:32:48.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.493+0000 7f70f2a00640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40cfe70 0x7f70f414fab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f70e0033000 tx=0x7f70e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:48.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.497+0000 7f70eb400640 1 -- 172.21.15.102:0/362849805 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0034040 con 0x7f70f40cfe70 2024-03-18T16:32:48.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.497+0000 7f70eb400640 1 -- 172.21.15.102:0/362849805 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70e003ada0 con 0x7f70f40cfe70 2024-03-18T16:32:48.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.497+0000 7f70eb400640 1 -- 172.21.15.102:0/362849805 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0044610 con 0x7f70f40cfe70 2024-03-18T16:32:48.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.497+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f41f33d0 con 0x7f70f40cfe70 2024-03-18T16:32:48.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.497+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f41f3990 con 0x7f70f40cfe70 2024-03-18T16:32:48.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.499+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70f4074fd0 con 0x7f70f40cfe70 2024-03-18T16:32:48.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.758+0000 7f70eb400640 1 -- 172.21.15.102:0/362849805 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f70e004b060 con 0x7f70f40cfe70 2024-03-18T16:32:48.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.760+0000 7f70eb400640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70d406bc50 0x7f70d406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:48.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.760+0000 7f70eb400640 1 -- 172.21.15.102:0/362849805 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f70e00ad000 con 0x7f70f40cfe70 2024-03-18T16:32:48.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.760+0000 7f70f3400640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70d406bc50 0x7f70d406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:48.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.761+0000 7f70f3400640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70d406bc50 0x7f70d406e110 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f70e4006a90 tx=0x7f70e4006a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:48.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:48.770+0000 7f70eb400640 1 -- 172.21.15.102:0/362849805 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f70e003f070 con 0x7f70f40cfe70 2024-03-18T16:32:49.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.133+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70f415b130 con 0x7f70f40cfe70 2024-03-18T16:32:49.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.146+0000 7f70eb400640 1 -- 172.21.15.102:0/362849805 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f70e007b770 con 0x7f70f40cfe70 2024-03-18T16:32:49.149 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:49.149 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:49.149 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:49.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.149+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70d406bc50 msgr2=0x7f70d406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:49.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.149+0000 7f70f9000640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70d406bc50 0x7f70d406e110 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f70e4006a90 tx=0x7f70e4006a20 comp rx=0 tx=0).stop 2024-03-18T16:32:49.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.149+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40cfe70 msgr2=0x7f70f414fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.149+0000 7f70f9000640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40cfe70 0x7f70f414fab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f70e0033000 tx=0x7f70e00046c0 comp rx=0 tx=0).stop 2024-03-18T16:32:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.150+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 shutdown_connections 2024-03-18T16:32:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.150+0000 7f70f9000640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70d406bc50 0x7f70d406e110 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.150+0000 7f70f9000640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f70f40d0250 0x7f70f41e1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.150+0000 7f70f9000640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70f40cfe70 0x7f70f414fab0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.150+0000 7f70f9000640 1 --2- 172.21.15.102:0/362849805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70f40cc2f0 0x7f70f41e03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.150+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 >> 172.21.15.102:0/362849805 conn(0x7f70f408c3e0 msgr2=0x7f70f40cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.150+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 shutdown_connections 2024-03-18T16:32:49.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:49.150+0000 7f70f9000640 1 -- 172.21.15.102:0/362849805 wait complete. 2024-03-18T16:32:49.772 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:49.772 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:50.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:50.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.945+0000 7fa0cfe00640 1 Processor -- start 2024-03-18T16:32:50.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.945+0000 7fa0cfe00640 1 -- start start 2024-03-18T16:32:50.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.946+0000 7fa0cfe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0d015c3a0 0x7fa0d0163cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:50.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.946+0000 7fa0cfe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d00c72d0 con 0x7fa0d0158420 2024-03-18T16:32:50.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.946+0000 7fa0cfe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d0158840 con 0x7fa0d0141650 2024-03-18T16:32:50.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.946+0000 7fa0cfe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d0159370 con 0x7fa0d015c3a0 2024-03-18T16:32:50.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.947+0000 7fa0cf400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa0d015c3a0 0x7fa0d0163cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:50.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.947+0000 7fa0ce000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa0d0158420 0x7fa0d00c4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58800/0 (socket says 172.21.15.102:58800) 2024-03-18T16:32:50.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.947+0000 7fa0ce000640 1 -- 172.21.15.102:0/2103904946 learned_addr learned my addr 172.21.15.102:0/2103904946 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.950+0000 7fa0cd600640 1 -- 172.21.15.102:0/2103904946 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1042915022 0 0) 0x7fa0d00c72d0 con 0x7fa0d0158420 2024-03-18T16:32:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.950+0000 7fa0cd600640 1 -- 172.21.15.102:0/2103904946 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0b4003520 con 0x7fa0d0158420 2024-03-18T16:32:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.950+0000 7fa0cf400640 1 -- 172.21.15.102:0/2103904946 >> v1:172.21.15.154:6789/0 conn(0x7fa0d0141650 legacy=0x7fa0d00c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.950+0000 7fa0cf400640 1 -- 172.21.15.102:0/2103904946 >> v1:172.21.15.102:6789/0 conn(0x7fa0d0158420 legacy=0x7fa0d00c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:50.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.950+0000 7fa0cf400640 1 -- 172.21.15.102:0/2103904946 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d0158ad0 con 0x7fa0d015c3a0 2024-03-18T16:32:50.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.954+0000 7fa0cf400640 1 --2- 172.21.15.102:0/2103904946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d015c3a0 0x7fa0d0163cd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009d90 tx=0x7fa0c802bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=600a00b6e1434302 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:50.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.958+0000 7fa0cd600640 1 -- 172.21.15.102:0/2103904946 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8039070 con 0x7fa0d015c3a0 2024-03-18T16:32:50.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.958+0000 7fa0cd600640 1 -- 172.21.15.102:0/2103904946 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c802ccf0 con 0x7fa0d015c3a0 2024-03-18T16:32:50.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.958+0000 7fa0cd600640 1 -- 172.21.15.102:0/2103904946 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8034660 con 0x7fa0d015c3a0 2024-03-18T16:32:50.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.959+0000 7fa0cfe00640 1 -- 172.21.15.102:0/2103904946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d015c3a0 msgr2=0x7fa0d0163cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:50.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.959+0000 7fa0cfe00640 1 --2- 172.21.15.102:0/2103904946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d015c3a0 0x7fa0d0163cd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009d90 tx=0x7fa0c802bf40 comp rx=0 tx=0).stop 2024-03-18T16:32:50.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.959+0000 7fa0cfe00640 1 -- 172.21.15.102:0/2103904946 shutdown_connections 2024-03-18T16:32:50.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.959+0000 7fa0cfe00640 1 --2- 172.21.15.102:0/2103904946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d015c3a0 0x7fa0d0163cd0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:50.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.959+0000 7fa0cfe00640 1 -- 172.21.15.102:0/2103904946 >> 172.21.15.102:0/2103904946 conn(0x7fa0d00cb890 msgr2=0x7fa0d00cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:50.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.959+0000 7fa0cfe00640 1 -- 172.21.15.102:0/2103904946 shutdown_connections 2024-03-18T16:32:50.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.959+0000 7fa0cfe00640 1 -- 172.21.15.102:0/2103904946 wait complete. 2024-03-18T16:32:50.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.961+0000 7fa0cfe00640 1 Processor -- start 2024-03-18T16:32:50.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.962+0000 7fa0cfe00640 1 -- start start 2024-03-18T16:32:50.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.963+0000 7fa0cfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d0141650 0x7fa0d01e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:50.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.963+0000 7fa0cfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d0158420 0x7fa0d01e19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:50.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.963+0000 7fa0cfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0d015c3a0 0x7fa0d01f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:50.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.963+0000 7fa0cfe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d00c63c0 con 0x7fa0d015c3a0 2024-03-18T16:32:50.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.963+0000 7fa0cfe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d0159090 con 0x7fa0d0158420 2024-03-18T16:32:50.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.963+0000 7fa0cfe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d00c66a0 con 0x7fa0d0141650 2024-03-18T16:32:50.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.964+0000 7fa0ce000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d0158420 0x7fa0d01e19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:50.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.964+0000 7fa0cea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d0141650 0x7fa0d01e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:50.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.964+0000 7fa0cea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d0141650 0x7fa0d01e2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56900/0 (socket says 172.21.15.102:56900) 2024-03-18T16:32:50.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.964+0000 7fa0cea00640 1 -- 172.21.15.102:0/645999018 learned_addr learned my addr 172.21.15.102:0/645999018 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:50.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.965+0000 7fa0cf400640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0d015c3a0 0x7fa0d01f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:50.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.971+0000 7fa0cea00640 1 -- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d0158420 msgr2=0x7fa0d01e19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:50.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.971+0000 7fa0cea00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d0158420 0x7fa0d01e19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:50.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.971+0000 7fa0cea00640 1 -- 172.21.15.102:0/645999018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0d015c3a0 msgr2=0x7fa0d01f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:50.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.971+0000 7fa0cea00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0d015c3a0 0x7fa0d01f0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:50.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.971+0000 7fa0cea00640 1 -- 172.21.15.102:0/645999018 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d0154510 con 0x7fa0d0141650 2024-03-18T16:32:50.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.975+0000 7fa0cea00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d0141650 0x7fa0d01e2c90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c430 tx=0x7fa0c000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.980+0000 7fa0c6a00640 1 -- 172.21.15.102:0/645999018 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c00052d0 con 0x7fa0d0141650 2024-03-18T16:32:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.980+0000 7fa0c6a00640 1 -- 172.21.15.102:0/645999018 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c0005490 con 0x7fa0d0141650 2024-03-18T16:32:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.980+0000 7fa0c6a00640 1 -- 172.21.15.102:0/645999018 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c0007260 con 0x7fa0d0141650 2024-03-18T16:32:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.980+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d01547f0 con 0x7fa0d0141650 2024-03-18T16:32:50.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.980+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d0155870 con 0x7fa0d0141650 2024-03-18T16:32:50.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:50.982+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d015c810 con 0x7fa0d0141650 2024-03-18T16:32:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.021+0000 7fa0c6a00640 1 -- 172.21.15.102:0/645999018 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa0c0007400 con 0x7fa0d0141650 2024-03-18T16:32:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.022+0000 7fa0c6a00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0b006bc50 0x7fa0b006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:51.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.022+0000 7fa0ce000640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0b006bc50 0x7fa0b006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:51.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.023+0000 7fa0ce000640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0b006bc50 0x7fa0b006e110 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa0d01f1a30 tx=0x7fa0bc007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:51.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.319+0000 7fa0c6a00640 1 -- 172.21.15.102:0/645999018 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa0c0056ca0 con 0x7fa0d0141650 2024-03-18T16:32:51.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.330+0000 7fa0c6a00640 1 -- 172.21.15.102:0/645999018 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa0d015c810 con 0x7fa0d0141650 2024-03-18T16:32:51.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.577+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0d00c5f80 con 0x7fa0d0141650 2024-03-18T16:32:51.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.590+0000 7fa0c6a00640 1 -- 172.21.15.102:0/645999018 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa0d00c5f80 con 0x7fa0d0141650 2024-03-18T16:32:51.593 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:51.593 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:51.593 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:51.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.593+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0b006bc50 msgr2=0x7fa0b006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:51.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.593+0000 7fa0cfe00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0b006bc50 0x7fa0b006e110 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa0d01f1a30 tx=0x7fa0bc007d30 comp rx=0 tx=0).stop 2024-03-18T16:32:51.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.593+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d0141650 msgr2=0x7fa0d01e2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:51.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.593+0000 7fa0cfe00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d0141650 0x7fa0d01e2c90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa0c000c430 tx=0x7fa0c000c8f0 comp rx=0 tx=0).stop 2024-03-18T16:32:51.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.593+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 shutdown_connections 2024-03-18T16:32:51.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.593+0000 7fa0cfe00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa0b006bc50 0x7fa0b006e110 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:51.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.593+0000 7fa0cfe00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0d015c3a0 0x7fa0d01f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:51.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.594+0000 7fa0cfe00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0d0158420 0x7fa0d01e19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:51.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.594+0000 7fa0cfe00640 1 --2- 172.21.15.102:0/645999018 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa0d0141650 0x7fa0d01e2c90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:51.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.594+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 >> 172.21.15.102:0/645999018 conn(0x7fa0d00cb890 msgr2=0x7fa0d0157ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:51.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.594+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 shutdown_connections 2024-03-18T16:32:51.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:51.594+0000 7fa0cfe00640 1 -- 172.21.15.102:0/645999018 wait complete. 2024-03-18T16:32:52.225 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:52.225 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:53.226 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.466+0000 7fa188a00640 1 Processor -- start 2024-03-18T16:32:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.466+0000 7fa188a00640 1 -- start start 2024-03-18T16:32:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.467+0000 7fa188a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa18415a310 0x7fa18406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.467+0000 7fa188a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415baa0 0x7fa18415bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.467+0000 7fa188a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1840ca500 con 0x7fa18415a6f0 2024-03-18T16:32:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.467+0000 7fa188a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa1840cd410 con 0x7fa18415a310 2024-03-18T16:32:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.467+0000 7fa188a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa18415c490 con 0x7fa18415baa0 2024-03-18T16:32:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.468+0000 7fa183e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415baa0 0x7fa18415bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.468+0000 7fa183400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa18415a310 0x7fa18406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.468+0000 7fa183e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415baa0 0x7fa18415bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56908/0 (socket says 172.21.15.102:56908) 2024-03-18T16:32:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.468+0000 7fa183e00640 1 -- 172.21.15.102:0/1570120078 learned_addr learned my addr 172.21.15.102:0/1570120078 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.468+0000 7fa182a00640 1 --1- 172.21.15.102:0/1570120078 >> v1:172.21.15.154:6789/0 conn(0x7fa18415a6f0 0x7fa18415ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:38704/0 (socket says 172.21.15.102:38704) 2024-03-18T16:32:53.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.471+0000 7fa183400640 1 -- 172.21.15.102:0/1570120078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415baa0 msgr2=0x7fa18415bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.471+0000 7fa183400640 1 --2- 172.21.15.102:0/1570120078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415baa0 0x7fa18415bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.471+0000 7fa183400640 1 -- 172.21.15.102:0/1570120078 >> v1:172.21.15.154:6789/0 conn(0x7fa18415a6f0 legacy=0x7fa18415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.471+0000 7fa183400640 1 -- 172.21.15.102:0/1570120078 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1840ca7a0 con 0x7fa18415a310 2024-03-18T16:32:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.474+0000 7fa183400640 1 --2- 172.21.15.102:0/1570120078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa18415a310 0x7fa18406d3e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa1740097a0 tx=0x7fa17402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38fb96db3724d4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:53.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.476+0000 7fa182000640 1 -- 172.21.15.102:0/1570120078 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174035020 con 0x7fa18415a310 2024-03-18T16:32:53.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.479+0000 7fa182000640 1 -- 172.21.15.102:0/1570120078 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa174031810 con 0x7fa18415a310 2024-03-18T16:32:53.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.479+0000 7fa182000640 1 -- 172.21.15.102:0/1570120078 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174031be0 con 0x7fa18415a310 2024-03-18T16:32:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.480+0000 7fa188a00640 1 -- 172.21.15.102:0/1570120078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa18415a310 msgr2=0x7fa18406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.480+0000 7fa188a00640 1 --2- 172.21.15.102:0/1570120078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa18415a310 0x7fa18406d3e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa1740097a0 tx=0x7fa17402b8b0 comp rx=0 tx=0).stop 2024-03-18T16:32:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.480+0000 7fa188a00640 1 -- 172.21.15.102:0/1570120078 shutdown_connections 2024-03-18T16:32:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.480+0000 7fa188a00640 1 --2- 172.21.15.102:0/1570120078 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415baa0 0x7fa18415bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.480+0000 7fa188a00640 1 --2- 172.21.15.102:0/1570120078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa18415a310 0x7fa18406d3e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.480+0000 7fa188a00640 1 -- 172.21.15.102:0/1570120078 >> 172.21.15.102:0/1570120078 conn(0x7fa18408c1a0 msgr2=0x7fa184066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.480+0000 7fa188a00640 1 -- 172.21.15.102:0/1570120078 shutdown_connections 2024-03-18T16:32:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.480+0000 7fa188a00640 1 -- 172.21.15.102:0/1570120078 wait complete. 2024-03-18T16:32:53.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.482+0000 7fa188a00640 1 Processor -- start 2024-03-18T16:32:53.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.482+0000 7fa188a00640 1 -- start start 2024-03-18T16:32:53.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.483+0000 7fa188a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415a310 0x7fa18414b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:53.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.483+0000 7fa188a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa18415a6f0 0x7fa1841f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:53.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.483+0000 7fa188a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa18415baa0 0x7fa184158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:53.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.483+0000 7fa188a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa1840cd130 con 0x7fa18415baa0 2024-03-18T16:32:53.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.483+0000 7fa188a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1840cce50 con 0x7fa18415a6f0 2024-03-18T16:32:53.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.483+0000 7fa188a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa18406d920 con 0x7fa18415a310 2024-03-18T16:32:53.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.484+0000 7fa183400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415a310 0x7fa18414b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:53.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.484+0000 7fa182a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa18415a6f0 0x7fa1841f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:53.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.484+0000 7fa183e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa18415baa0 0x7fa184158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:53.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.484+0000 7fa183400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415a310 0x7fa18414b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56914/0 (socket says 172.21.15.102:56914) 2024-03-18T16:32:53.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.484+0000 7fa183400640 1 -- 172.21.15.102:0/1779560467 learned_addr learned my addr 172.21.15.102:0/1779560467 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:53.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.489+0000 7fa183400640 1 -- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa18415a6f0 msgr2=0x7fa1841f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:53.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.489+0000 7fa183400640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa18415a6f0 0x7fa1841f3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:53.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.489+0000 7fa183400640 1 -- 172.21.15.102:0/1779560467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa18415baa0 msgr2=0x7fa184158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:53.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.489+0000 7fa183400640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa18415baa0 0x7fa184158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:53.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.489+0000 7fa183400640 1 -- 172.21.15.102:0/1779560467 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa184154c90 con 0x7fa18415a310 2024-03-18T16:32:53.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.493+0000 7fa183400640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415a310 0x7fa18414b900 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa1740098c0 tx=0x7fa1740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:53.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.495+0000 7fa17b400640 1 -- 172.21.15.102:0/1779560467 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174035040 con 0x7fa18415a310 2024-03-18T16:32:53.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.495+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa184159d80 con 0x7fa18415a310 2024-03-18T16:32:53.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.495+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa184155210 con 0x7fa18415a310 2024-03-18T16:32:53.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.497+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa184074fd0 con 0x7fa18415a310 2024-03-18T16:32:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.497+0000 7fa17b400640 1 -- 172.21.15.102:0/1779560467 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa174031810 con 0x7fa18415a310 2024-03-18T16:32:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.497+0000 7fa17b400640 1 -- 172.21.15.102:0/1779560467 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa174044430 con 0x7fa18415a310 2024-03-18T16:32:53.542 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:32:53.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:32:53.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.759+0000 7fa17b400640 1 -- 172.21.15.102:0/1779560467 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa174038020 con 0x7fa18415a310 2024-03-18T16:32:53.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.761+0000 7fa17b400640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa16406bc00 0x7fa16406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:53.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.762+0000 7fa182a00640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa16406bc00 0x7fa16406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.762+0000 7fa182a00640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa16406bc00 0x7fa16406e0c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa18406d8c0 tx=0x7fa170000ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.763+0000 7fa17b400640 1 -- 172.21.15.102:0/1779560467 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa174079020 con 0x7fa18415a310 2024-03-18T16:32:53.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:53.774+0000 7fa17b400640 1 -- 172.21.15.102:0/1779560467 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa17407af80 con 0x7fa18415a310 2024-03-18T16:32:54.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.040+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1840caed0 con 0x7fa18415a310 2024-03-18T16:32:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.053+0000 7fa17b400640 1 -- 172.21.15.102:0/1779560467 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa17407f020 con 0x7fa18415a310 2024-03-18T16:32:54.055 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:54.055 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:54.056 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa16406bc00 msgr2=0x7fa16406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa16406bc00 0x7fa16406e0c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa18406d8c0 tx=0x7fa170000ed0 comp rx=0 tx=0).stop 2024-03-18T16:32:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415a310 msgr2=0x7fa18414b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415a310 0x7fa18414b900 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa1740098c0 tx=0x7fa1740046c0 comp rx=0 tx=0).stop 2024-03-18T16:32:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 shutdown_connections 2024-03-18T16:32:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa16406bc00 0x7fa16406e0c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa18415baa0 0x7fa184158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa18415a6f0 0x7fa1841f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 --2- 172.21.15.102:0/1779560467 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa18415a310 0x7fa18414b900 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 >> 172.21.15.102:0/1779560467 conn(0x7fa18408c1a0 msgr2=0x7fa1840cee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.057+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 shutdown_connections 2024-03-18T16:32:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:54.058+0000 7fa188a00640 1 -- 172.21.15.102:0/1779560467 wait complete. 2024-03-18T16:32:54.688 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:54.689 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:55.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:55.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.861+0000 7f7275600640 1 Processor -- start 2024-03-18T16:32:55.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.861+0000 7f7275600640 1 -- start start 2024-03-18T16:32:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.863+0000 7f7275600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7270158420 0x7f727015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.863+0000 7f7275600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72700c72d0 con 0x7f72700c4fa0 2024-03-18T16:32:55.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.863+0000 7f7275600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7270158840 con 0x7f7270141650 2024-03-18T16:32:55.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.863+0000 7f7275600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f727015c8e0 con 0x7f7270158420 2024-03-18T16:32:55.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.863+0000 7f726f400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7270158420 0x7f727015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:55.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.864+0000 7f7274c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f72700c4fa0 0x7f7270162cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:58806/0 (socket says 172.21.15.102:58806) 2024-03-18T16:32:55.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.864+0000 7f7274c00640 1 -- 172.21.15.102:0/896011760 learned_addr learned my addr 172.21.15.102:0/896011760 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:32:55.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.867+0000 7f726f400640 1 -- 172.21.15.102:0/896011760 >> v1:172.21.15.154:6789/0 conn(0x7f7270141650 legacy=0x7f72700c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:55.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.867+0000 7f726f400640 1 -- 172.21.15.102:0/896011760 >> v1:172.21.15.102:6789/0 conn(0x7f72700c4fa0 legacy=0x7f7270162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:55.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.867+0000 7f726f400640 1 -- 172.21.15.102:0/896011760 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7270158ad0 con 0x7f7270158420 2024-03-18T16:32:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.871+0000 7f726f400640 1 --2- 172.21.15.102:0/896011760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270158420 0x7f727015c3a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f725c0097a0 tx=0x7f725c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17eee8eeac41fc4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.875+0000 7f726ea00640 1 -- 172.21.15.102:0/896011760 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c034070 con 0x7f7270158420 2024-03-18T16:32:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.875+0000 7f726ea00640 1 -- 172.21.15.102:0/896011760 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725c030610 con 0x7f7270158420 2024-03-18T16:32:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.875+0000 7f726ea00640 1 -- 172.21.15.102:0/896011760 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c0309e0 con 0x7f7270158420 2024-03-18T16:32:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.876+0000 7f7275600640 1 -- 172.21.15.102:0/896011760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270158420 msgr2=0x7f727015c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.876+0000 7f7275600640 1 --2- 172.21.15.102:0/896011760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270158420 0x7f727015c3a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f725c0097a0 tx=0x7f725c02b960 comp rx=0 tx=0).stop 2024-03-18T16:32:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.876+0000 7f7275600640 1 -- 172.21.15.102:0/896011760 shutdown_connections 2024-03-18T16:32:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.876+0000 7f7275600640 1 --2- 172.21.15.102:0/896011760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270158420 0x7f727015c3a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.876+0000 7f7275600640 1 -- 172.21.15.102:0/896011760 >> 172.21.15.102:0/896011760 conn(0x7f72700cb890 msgr2=0x7f72700cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.876+0000 7f7275600640 1 -- 172.21.15.102:0/896011760 shutdown_connections 2024-03-18T16:32:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.876+0000 7f7275600640 1 -- 172.21.15.102:0/896011760 wait complete. 2024-03-18T16:32:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.878+0000 7f7275600640 1 Processor -- start 2024-03-18T16:32:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.878+0000 7f7275600640 1 -- start start 2024-03-18T16:32:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.879+0000 7f7275600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72700c4fa0 0x7f72700caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.879+0000 7f7275600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270141650 0x7f727015a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.879+0000 7f7275600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7270158420 0x7f727006ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.879+0000 7f7275600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7270159090 con 0x7f7270158420 2024-03-18T16:32:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.879+0000 7f7275600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7270159370 con 0x7f7270141650 2024-03-18T16:32:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.879+0000 7f7275600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f72700c63c0 con 0x7f72700c4fa0 2024-03-18T16:32:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.880+0000 7f726f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270141650 0x7f727015a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.880+0000 7f7274c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7270158420 0x7f727006ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.880+0000 7f726fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72700c4fa0 0x7f72700caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.880+0000 7f726f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270141650 0x7f727015a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57354/0 (socket says 172.21.15.102:57354) 2024-03-18T16:32:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.880+0000 7f726f400640 1 -- 172.21.15.102:0/2349622297 learned_addr learned my addr 172.21.15.102:0/2349622297 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.884+0000 7f726fe00640 1 -- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270141650 msgr2=0x7f727015a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.884+0000 7f726fe00640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270141650 0x7f727015a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.884+0000 7f726fe00640 1 -- 172.21.15.102:0/2349622297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7270158420 msgr2=0x7f727006ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.884+0000 7f726fe00640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7270158420 0x7f727006ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.884+0000 7f726fe00640 1 -- 172.21.15.102:0/2349622297 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72701f1f80 con 0x7f72700c4fa0 2024-03-18T16:32:55.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.886+0000 7f726fe00640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72700c4fa0 0x7f72700caf20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f726000c340 tx=0x7f726000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.890+0000 7f7267e00640 1 -- 172.21.15.102:0/2349622297 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72600052d0 con 0x7f72700c4fa0 2024-03-18T16:32:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.890+0000 7f7267e00640 1 -- 172.21.15.102:0/2349622297 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7260005490 con 0x7f72700c4fa0 2024-03-18T16:32:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.890+0000 7f7267e00640 1 -- 172.21.15.102:0/2349622297 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260007260 con 0x7f72700c4fa0 2024-03-18T16:32:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.890+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72701e28d0 con 0x7f72700c4fa0 2024-03-18T16:32:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.890+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72701e2e90 con 0x7f72700c4fa0 2024-03-18T16:32:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.892+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72700b7f00 con 0x7f72700c4fa0 2024-03-18T16:32:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.914+0000 7f7267e00640 1 -- 172.21.15.102:0/2349622297 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7260007400 con 0x7f72700c4fa0 2024-03-18T16:32:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.915+0000 7f7267e00640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f725006bc20 0x7f725006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.915+0000 7f726f400640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f725006bc20 0x7f725006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:55.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.916+0000 7f726f400640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f725006bc20 0x7f725006e0e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7270158100 tx=0x7f725c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:55.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:55.917+0000 7f7267e00640 1 -- 172.21.15.102:0/2349622297 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7260056870 con 0x7f72700c4fa0 2024-03-18T16:32:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.170+0000 7f7267e00640 1 -- 172.21.15.102:0/2349622297 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f72600587d0 con 0x7f72700c4fa0 2024-03-18T16:32:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.469+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f727013e080 con 0x7f72700c4fa0 2024-03-18T16:32:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.481+0000 7f7267e00640 1 -- 172.21.15.102:0/2349622297 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f726005c870 con 0x7f72700c4fa0 2024-03-18T16:32:56.483 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:56.483 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:56.483 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:56.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f725006bc20 msgr2=0x7f725006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f725006bc20 0x7f725006e0e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7270158100 tx=0x7f725c03b040 comp rx=0 tx=0).stop 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72700c4fa0 msgr2=0x7f72700caf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72700c4fa0 0x7f72700caf20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f726000c340 tx=0x7f726000c800 comp rx=0 tx=0).stop 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 shutdown_connections 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f725006bc20 0x7f725006e0e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7270158420 0x7f727006ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7270141650 0x7f727015a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 --2- 172.21.15.102:0/2349622297 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f72700c4fa0 0x7f72700caf20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 >> 172.21.15.102:0/2349622297 conn(0x7f72700cb890 msgr2=0x7f72700747f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 shutdown_connections 2024-03-18T16:32:56.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:56.484+0000 7f7275600640 1 -- 172.21.15.102:0/2349622297 wait complete. 2024-03-18T16:32:57.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:57.107 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:32:58.108 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:32:58.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.277+0000 7f4e3e400640 1 Processor -- start 2024-03-18T16:32:58.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.277+0000 7f4e3e400640 1 -- start start 2024-03-18T16:32:58.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.278+0000 7f4e3e400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e3815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:58.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.278+0000 7f4e3e400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e380d0250 0x7f4e380c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:58.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.278+0000 7f4e3e400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e380cabd0 con 0x7f4e380cc2f0 2024-03-18T16:32:58.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.278+0000 7f4e3e400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4e380caea0 con 0x7f4e380d0250 2024-03-18T16:32:58.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.278+0000 7f4e3e400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4e380cb180 con 0x7f4e380cfe70 2024-03-18T16:32:58.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.279+0000 7f4e3d000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e3815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:58.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.279+0000 7f4e37e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e380d0250 0x7f4e380c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:58.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.280+0000 7f4e3d000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e3815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56926/0 (socket says 172.21.15.102:56926) 2024-03-18T16:32:58.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.280+0000 7f4e3d000640 1 -- 172.21.15.102:0/2013839462 learned_addr learned my addr 172.21.15.102:0/2013839462 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:58.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.282+0000 7f4e3d000640 1 -- 172.21.15.102:0/2013839462 >> v2:172.21.15.154:3300/0 conn(0x7f4e380d0250 msgr2=0x7f4e380c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:32:58.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.282+0000 7f4e3d000640 1 --2- 172.21.15.102:0/2013839462 >> v2:172.21.15.154:3300/0 conn(0x7f4e380d0250 0x7f4e380c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:58.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.282+0000 7f4e3d000640 1 -- 172.21.15.102:0/2013839462 >> v1:172.21.15.102:6789/0 conn(0x7f4e380cc2f0 legacy=0x7f4e380cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:58.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.282+0000 7f4e3d000640 1 -- 172.21.15.102:0/2013839462 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e380cb460 con 0x7f4e380cfe70 2024-03-18T16:32:58.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.285+0000 7f4e3d000640 1 --2- 172.21.15.102:0/2013839462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e3815ae40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009870 tx=0x7f4e2c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15013835b21efdd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:58.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.288+0000 7f4e37400640 1 -- 172.21.15.102:0/2013839462 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c02f830 con 0x7f4e380cfe70 2024-03-18T16:32:58.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.288+0000 7f4e37400640 1 -- 172.21.15.102:0/2013839462 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e2c02f9d0 con 0x7f4e380cfe70 2024-03-18T16:32:58.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.288+0000 7f4e37400640 1 -- 172.21.15.102:0/2013839462 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c02fda0 con 0x7f4e380cfe70 2024-03-18T16:32:58.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.289+0000 7f4e3e400640 1 -- 172.21.15.102:0/2013839462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 msgr2=0x7f4e3815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:58.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.289+0000 7f4e3e400640 1 --2- 172.21.15.102:0/2013839462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e3815ae40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009870 tx=0x7f4e2c02ba10 comp rx=0 tx=0).stop 2024-03-18T16:32:58.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.289+0000 7f4e3e400640 1 -- 172.21.15.102:0/2013839462 shutdown_connections 2024-03-18T16:32:58.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.289+0000 7f4e3e400640 1 --2- 172.21.15.102:0/2013839462 >> v2:172.21.15.154:3300/0 conn(0x7f4e380d0250 0x7f4e380c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:58.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.289+0000 7f4e3e400640 1 --2- 172.21.15.102:0/2013839462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e3815ae40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:58.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.289+0000 7f4e3e400640 1 -- 172.21.15.102:0/2013839462 >> 172.21.15.102:0/2013839462 conn(0x7f4e3808c3e0 msgr2=0x7f4e3806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:58.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.289+0000 7f4e3e400640 1 -- 172.21.15.102:0/2013839462 shutdown_connections 2024-03-18T16:32:58.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.289+0000 7f4e3e400640 1 -- 172.21.15.102:0/2013839462 wait complete. 2024-03-18T16:32:58.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.292+0000 7f4e3e400640 1 Processor -- start 2024-03-18T16:32:58.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.292+0000 7f4e3e400640 1 -- start start 2024-03-18T16:32:58.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.294+0000 7f4e3e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e380cc2f0 0x7f4e38159340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:58.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.294+0000 7f4e3e400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e381f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:58.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.294+0000 7f4e3e400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380d0250 0x7f4e38158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.294+0000 7f4e3e400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4e380cbd00 con 0x7f4e380cc2f0 2024-03-18T16:32:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.294+0000 7f4e3e400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4e380cba20 con 0x7f4e380d0250 2024-03-18T16:32:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.294+0000 7f4e3e400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4e380c6b20 con 0x7f4e380cfe70 2024-03-18T16:32:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.294+0000 7f4e37e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e381f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:58.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.295+0000 7f4e3d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e380cc2f0 0x7f4e38159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.295+0000 7f4e37e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e381f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56936/0 (socket says 172.21.15.102:56936) 2024-03-18T16:32:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.295+0000 7f4e37e00640 1 -- 172.21.15.102:0/490875893 learned_addr learned my addr 172.21.15.102:0/490875893 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:32:58.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.295+0000 7f4e3da00640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380d0250 0x7f4e38158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:58.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.298+0000 7f4e37e00640 1 -- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380d0250 msgr2=0x7f4e38158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:58.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.298+0000 7f4e37e00640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380d0250 0x7f4e38158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:58.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.298+0000 7f4e37e00640 1 -- 172.21.15.102:0/490875893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e380cc2f0 msgr2=0x7f4e38159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:58.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.298+0000 7f4e37e00640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e380cc2f0 0x7f4e38159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:58.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.298+0000 7f4e37e00640 1 -- 172.21.15.102:0/490875893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e381efe90 con 0x7f4e380cfe70 2024-03-18T16:32:58.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.300+0000 7f4e37e00640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e381f3e10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400e8a0 tx=0x7f4e2400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:58.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.304+0000 7f4e34c00640 1 -- 172.21.15.102:0/490875893 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24017070 con 0x7f4e380cfe70 2024-03-18T16:32:58.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.304+0000 7f4e34c00640 1 -- 172.21.15.102:0/490875893 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e2400c940 con 0x7f4e380cfe70 2024-03-18T16:32:58.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.304+0000 7f4e34c00640 1 -- 172.21.15.102:0/490875893 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2400cd30 con 0x7f4e380cfe70 2024-03-18T16:32:58.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.304+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e38159a40 con 0x7f4e380cfe70 2024-03-18T16:32:58.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.304+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e381f0150 con 0x7f4e380cfe70 2024-03-18T16:32:58.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.305+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e38074fd0 con 0x7f4e380cfe70 2024-03-18T16:32:58.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.325+0000 7f4e34c00640 1 -- 172.21.15.102:0/490875893 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4e2401a070 con 0x7f4e380cfe70 2024-03-18T16:32:58.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.326+0000 7f4e34c00640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e1806bbd0 0x7f4e1806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:32:58.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.326+0000 7f4e3d000640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e1806bbd0 0x7f4e1806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:32:58.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.327+0000 7f4e3d000640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e1806bbd0 0x7f4e1806e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c036000 tx=0x7f4e2c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:32:58.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.571+0000 7f4e34c00640 1 -- 172.21.15.102:0/490875893 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f4e24057540 con 0x7f4e380cfe70 2024-03-18T16:32:58.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.582+0000 7f4e34c00640 1 -- 172.21.15.102:0/490875893 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4e240594a0 con 0x7f4e380cfe70 2024-03-18T16:32:58.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.822+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e3815b270 con 0x7f4e380cfe70 2024-03-18T16:32:58.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.836+0000 7f4e34c00640 1 -- 172.21.15.102:0/490875893 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f4e2405d540 con 0x7f4e380cfe70 2024-03-18T16:32:58.838 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:32:58.838 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:32:58.838 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:32:58.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e1806bbd0 msgr2=0x7f4e1806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e1806bbd0 0x7f4e1806e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c036000 tx=0x7f4e2c008030 comp rx=0 tx=0).stop 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 msgr2=0x7f4e381f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e381f3e10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400e8a0 tx=0x7f4e2400ed60 comp rx=0 tx=0).stop 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 shutdown_connections 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e1806bbd0 0x7f4e1806e090 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e380d0250 0x7f4e38158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4e380cfe70 0x7f4e381f3e10 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400e8a0 tx=0x7f4e2400ed60 comp rx=0 tx=0).stop 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 --2- 172.21.15.102:0/490875893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e380cc2f0 0x7f4e38159340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.839+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 >> 172.21.15.102:0/490875893 conn(0x7f4e3808c3e0 msgr2=0x7f4e380ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.840+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 shutdown_connections 2024-03-18T16:32:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:32:58.840+0000 7f4e3e400640 1 -- 172.21.15.102:0/490875893 wait complete. 2024-03-18T16:32:59.470 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:32:59.471 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:00.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.657+0000 7f54a1400640 1 Processor -- start 2024-03-18T16:33:00.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.657+0000 7f54a1400640 1 -- start start 2024-03-18T16:33:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.658+0000 7f54a1400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f549c158420 0x7f549c15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.658+0000 7f54a1400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c0c4fa0 0x7f549c0c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.658+0000 7f54a1400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549c0c72d0 con 0x7f549c141650 2024-03-18T16:33:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.658+0000 7f54a1400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f549c15c8e0 con 0x7f549c158420 2024-03-18T16:33:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.658+0000 7f54a1400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f549c159370 con 0x7f549c0c4fa0 2024-03-18T16:33:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.659+0000 7f54a0a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c0c4fa0 0x7f549c0c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.659+0000 7f549b400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f549c158420 0x7f549c15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:00.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.659+0000 7f54a0a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c0c4fa0 0x7f549c0c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:56950/0 (socket says 172.21.15.102:56950) 2024-03-18T16:33:00.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.659+0000 7f54a0a00640 1 -- 172.21.15.102:0/2652830390 learned_addr learned my addr 172.21.15.102:0/2652830390 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.662+0000 7f54a0a00640 1 -- 172.21.15.102:0/2652830390 >> v2:172.21.15.154:3300/0 conn(0x7f549c158420 msgr2=0x7f549c15c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.662+0000 7f54a0a00640 1 --2- 172.21.15.102:0/2652830390 >> v2:172.21.15.154:3300/0 conn(0x7f549c158420 0x7f549c15c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.662+0000 7f54a0a00640 1 -- 172.21.15.102:0/2652830390 >> v1:172.21.15.154:6789/0 conn(0x7f549c141650 legacy=0x7f549c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.662+0000 7f54a0a00640 1 -- 172.21.15.102:0/2652830390 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c158840 con 0x7f549c0c4fa0 2024-03-18T16:33:00.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.664+0000 7f54a0a00640 1 --2- 172.21.15.102:0/2652830390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c0c4fa0 0x7f549c0c5430 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5494009e50 tx=0x7f549402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5003cb3c921a0da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:00.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.668+0000 7f549aa00640 1 -- 172.21.15.102:0/2652830390 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549402ca70 con 0x7f549c0c4fa0 2024-03-18T16:33:00.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.668+0000 7f549aa00640 1 -- 172.21.15.102:0/2652830390 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f549402cc10 con 0x7f549c0c4fa0 2024-03-18T16:33:00.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.668+0000 7f549aa00640 1 -- 172.21.15.102:0/2652830390 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54940343f0 con 0x7f549c0c4fa0 2024-03-18T16:33:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.669+0000 7f54a1400640 1 -- 172.21.15.102:0/2652830390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c0c4fa0 msgr2=0x7f549c0c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.669+0000 7f54a1400640 1 --2- 172.21.15.102:0/2652830390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c0c4fa0 0x7f549c0c5430 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5494009e50 tx=0x7f549402bed0 comp rx=0 tx=0).stop 2024-03-18T16:33:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.669+0000 7f54a1400640 1 -- 172.21.15.102:0/2652830390 shutdown_connections 2024-03-18T16:33:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.669+0000 7f54a1400640 1 --2- 172.21.15.102:0/2652830390 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c0c4fa0 0x7f549c0c5430 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.669+0000 7f54a1400640 1 --2- 172.21.15.102:0/2652830390 >> v2:172.21.15.154:3300/0 conn(0x7f549c158420 0x7f549c15c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.669+0000 7f54a1400640 1 -- 172.21.15.102:0/2652830390 >> 172.21.15.102:0/2652830390 conn(0x7f549c0cb890 msgr2=0x7f549c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.669+0000 7f54a1400640 1 -- 172.21.15.102:0/2652830390 shutdown_connections 2024-03-18T16:33:00.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.669+0000 7f54a1400640 1 -- 172.21.15.102:0/2652830390 wait complete. 2024-03-18T16:33:00.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.671+0000 7f54a1400640 1 Processor -- start 2024-03-18T16:33:00.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.671+0000 7f54a1400640 1 -- start start 2024-03-18T16:33:00.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.672+0000 7f54a1400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c0c4fa0 0x7f549c1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:00.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.672+0000 7f54a1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f549c141650 0x7f549c0c2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:00.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.672+0000 7f54a1400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c158420 0x7f549c0c2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.672+0000 7f54a1400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f549c159090 con 0x7f549c141650 2024-03-18T16:33:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.672+0000 7f54a1400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f549c158db0 con 0x7f549c0c4fa0 2024-03-18T16:33:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.672+0000 7f54a1400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f549c0c63c0 con 0x7f549c158420 2024-03-18T16:33:00.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.673+0000 7f549be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c0c4fa0 0x7f549c1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:00.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.673+0000 7f54a0a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c158420 0x7f549c0c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:00.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.673+0000 7f549b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f549c141650 0x7f549c0c2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:00.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.673+0000 7f549be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c0c4fa0 0x7f549c1551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57386/0 (socket says 172.21.15.102:57386) 2024-03-18T16:33:00.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.673+0000 7f549be00640 1 -- 172.21.15.102:0/2161966365 learned_addr learned my addr 172.21.15.102:0/2161966365 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:00.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.677+0000 7f549be00640 1 -- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c158420 msgr2=0x7f549c0c2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:00.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.677+0000 7f549be00640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c158420 0x7f549c0c2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:00.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.677+0000 7f549be00640 1 -- 172.21.15.102:0/2161966365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f549c141650 msgr2=0x7f549c0c2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:00.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.677+0000 7f549be00640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f549c141650 0x7f549c0c2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:00.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.677+0000 7f549be00640 1 -- 172.21.15.102:0/2161966365 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c0c47a0 con 0x7f549c0c4fa0 2024-03-18T16:33:00.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.677+0000 7f54a0a00640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c158420 0x7f549c0c2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:33:00.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.679+0000 7f549be00640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c0c4fa0 0x7f549c1551a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f548c00c430 tx=0x7f548c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:00.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.683+0000 7f5493e00640 1 -- 172.21.15.102:0/2161966365 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c011790 con 0x7f549c0c4fa0 2024-03-18T16:33:00.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.683+0000 7f5493e00640 1 -- 172.21.15.102:0/2161966365 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f548c017070 con 0x7f549c0c4fa0 2024-03-18T16:33:00.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.683+0000 7f5493e00640 1 -- 172.21.15.102:0/2161966365 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c011bb0 con 0x7f549c0c4fa0 2024-03-18T16:33:00.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.684+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549c0c4d20 con 0x7f549c0c4fa0 2024-03-18T16:33:00.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.684+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f549c0c4a40 con 0x7f549c0c4fa0 2024-03-18T16:33:00.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.685+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549c0c7b60 con 0x7f549c0c4fa0 2024-03-18T16:33:00.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.705+0000 7f5493e00640 1 -- 172.21.15.102:0/2161966365 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f548c01c460 con 0x7f549c0c4fa0 2024-03-18T16:33:00.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.706+0000 7f5493e00640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f547c06bc20 0x7f547c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:00.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.706+0000 7f549b400640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f547c06bc20 0x7f547c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:00.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.707+0000 7f549b400640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f547c06bc20 0x7f547c06e0e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f549c0c4f70 tx=0x7f5488009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:00.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.707+0000 7f5493e00640 1 -- 172.21.15.102:0/2161966365 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f548c057910 con 0x7f549c0c4fa0 2024-03-18T16:33:00.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:00.959+0000 7f5493e00640 1 -- 172.21.15.102:0/2161966365 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f549c0c7b60 con 0x7f549c0c4fa0 2024-03-18T16:33:01.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.259+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f549c074fd0 con 0x7f549c0c4fa0 2024-03-18T16:33:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.272+0000 7f5493e00640 1 -- 172.21.15.102:0/2161966365 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f548c059870 con 0x7f549c0c4fa0 2024-03-18T16:33:01.274 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:01.274 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f547c06bc20 msgr2=0x7f547c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f547c06bc20 0x7f547c06e0e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f549c0c4f70 tx=0x7f5488009190 comp rx=0 tx=0).stop 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c0c4fa0 msgr2=0x7f549c1551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c0c4fa0 0x7f549c1551a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f548c00c430 tx=0x7f548c00c8f0 comp rx=0 tx=0).stop 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 shutdown_connections 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f547c06bc20 0x7f547c06e0e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f549c158420 0x7f549c0c2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f549c141650 0x7f549c0c2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 --2- 172.21.15.102:0/2161966365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c0c4fa0 0x7f549c1551a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 >> 172.21.15.102:0/2161966365 conn(0x7f549c0cb890 msgr2=0x7f549c13e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 shutdown_connections 2024-03-18T16:33:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:01.276+0000 7f54a1400640 1 -- 172.21.15.102:0/2161966365 wait complete. 2024-03-18T16:33:01.904 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:01.904 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:02.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:03.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.080+0000 7f5313000640 1 Processor -- start 2024-03-18T16:33:03.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.080+0000 7f5313000640 1 -- start start 2024-03-18T16:33:03.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.082+0000 7f5313000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f530c157d90 0x7f530c0c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:03.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.082+0000 7f5313000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c072810 0x7f530c072c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:03.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.082+0000 7f5313000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f530c073530 con 0x7f530c0c7f30 2024-03-18T16:33:03.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.082+0000 7f5313000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f530c0c5d80 con 0x7f530c157d90 2024-03-18T16:33:03.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.082+0000 7f5313000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f530c157900 con 0x7f530c072810 2024-03-18T16:33:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.113+0000 7f5312600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c072810 0x7f530c072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:03.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.113+0000 7f5312600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c072810 0x7f530c072c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53202/0 (socket says 172.21.15.102:53202) 2024-03-18T16:33:03.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.113+0000 7f5312600640 1 -- 172.21.15.102:0/3792844784 learned_addr learned my addr 172.21.15.102:0/3792844784 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:03.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.115+0000 7f5312600640 1 -- 172.21.15.102:0/3792844784 >> v2:172.21.15.102:3300/0 conn(0x7f530c157d90 msgr2=0x7f530c0c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:03.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.115+0000 7f5312600640 1 --2- 172.21.15.102:0/3792844784 >> v2:172.21.15.102:3300/0 conn(0x7f530c157d90 0x7f530c0c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:03.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.115+0000 7f5312600640 1 -- 172.21.15.102:0/3792844784 >> v1:172.21.15.102:6789/0 conn(0x7f530c0c7f30 legacy=0x7f530c15b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:03.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.115+0000 7f5312600640 1 -- 172.21.15.102:0/3792844784 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c0c57e0 con 0x7f530c072810 2024-03-18T16:33:03.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.118+0000 7f5312600640 1 --2- 172.21.15.102:0/3792844784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c072810 0x7f530c072c20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5304009e50 tx=0x7f530402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d0f90189d71848 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:03.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.122+0000 7f530be00640 1 -- 172.21.15.102:0/3792844784 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530402cb00 con 0x7f530c072810 2024-03-18T16:33:03.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.122+0000 7f530be00640 1 -- 172.21.15.102:0/3792844784 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530402cca0 con 0x7f530c072810 2024-03-18T16:33:03.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.122+0000 7f530be00640 1 -- 172.21.15.102:0/3792844784 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304034610 con 0x7f530c072810 2024-03-18T16:33:03.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.123+0000 7f5313000640 1 -- 172.21.15.102:0/3792844784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c072810 msgr2=0x7f530c072c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:03.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.123+0000 7f5313000640 1 --2- 172.21.15.102:0/3792844784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c072810 0x7f530c072c20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5304009e50 tx=0x7f530402bf60 comp rx=0 tx=0).stop 2024-03-18T16:33:03.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.123+0000 7f5313000640 1 -- 172.21.15.102:0/3792844784 shutdown_connections 2024-03-18T16:33:03.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.123+0000 7f5313000640 1 --2- 172.21.15.102:0/3792844784 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c072810 0x7f530c072c20 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:03.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.123+0000 7f5313000640 1 --2- 172.21.15.102:0/3792844784 >> v2:172.21.15.102:3300/0 conn(0x7f530c157d90 0x7f530c0c2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:03.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.123+0000 7f5313000640 1 -- 172.21.15.102:0/3792844784 >> 172.21.15.102:0/3792844784 conn(0x7f530c0cb890 msgr2=0x7f530c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:03.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.124+0000 7f5313000640 1 -- 172.21.15.102:0/3792844784 shutdown_connections 2024-03-18T16:33:03.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.124+0000 7f5313000640 1 -- 172.21.15.102:0/3792844784 wait complete. 2024-03-18T16:33:03.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.126+0000 7f5313000640 1 Processor -- start 2024-03-18T16:33:03.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.126+0000 7f5313000640 1 -- start start 2024-03-18T16:33:03.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.128+0000 7f5313000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f530c072810 0x7f530c15af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:03.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.128+0000 7f5313000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0c7f30 0x7f530c1f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:03.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.128+0000 7f5313000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c157d90 0x7f530c154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:03.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.128+0000 7f5313000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f530c157590 con 0x7f530c072810 2024-03-18T16:33:03.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.128+0000 7f5313000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1572b0 con 0x7f530c0c7f30 2024-03-18T16:33:03.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.128+0000 7f5313000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f530c0c2940 con 0x7f530c157d90 2024-03-18T16:33:03.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.129+0000 7f5312600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c157d90 0x7f530c154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:03.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.129+0000 7f5311200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0c7f30 0x7f530c1f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:03.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.129+0000 7f5311200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0c7f30 0x7f530c1f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59980/0 (socket says 172.21.15.102:59980) 2024-03-18T16:33:03.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.129+0000 7f5311200640 1 -- 172.21.15.102:0/2206861400 learned_addr learned my addr 172.21.15.102:0/2206861400 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:03.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.131+0000 7f5311c00640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f530c072810 0x7f530c15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:03.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.134+0000 7f5312600640 1 -- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0c7f30 msgr2=0x7f530c1f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:03.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.134+0000 7f5312600640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0c7f30 0x7f530c1f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:03.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.134+0000 7f5312600640 1 -- 172.21.15.102:0/2206861400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f530c072810 msgr2=0x7f530c15af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:03.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.134+0000 7f5312600640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f530c072810 0x7f530c15af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:03.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.134+0000 7f5312600640 1 -- 172.21.15.102:0/2206861400 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c0c3670 con 0x7f530c157d90 2024-03-18T16:33:03.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.138+0000 7f5312600640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c157d90 0x7f530c154160 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5304032000 tx=0x7f5304004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:03.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.148+0000 7f5309600640 1 -- 172.21.15.102:0/2206861400 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5304043030 con 0x7f530c157d90 2024-03-18T16:33:03.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.148+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530c150b40 con 0x7f530c157d90 2024-03-18T16:33:03.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.148+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f530c158370 con 0x7f530c157d90 2024-03-18T16:33:03.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.149+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f530c074fd0 con 0x7f530c157d90 2024-03-18T16:33:03.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.424+0000 7f5309600640 1 -- 172.21.15.102:0/2206861400 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530402ce50 con 0x7f530c157d90 2024-03-18T16:33:03.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.425+0000 7f5309600640 1 -- 172.21.15.102:0/2206861400 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530402cb10 con 0x7f530c157d90 2024-03-18T16:33:03.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.427+0000 7f5309600640 1 -- 172.21.15.102:0/2206861400 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5304046070 con 0x7f530c157d90 2024-03-18T16:33:03.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.428+0000 7f5309600640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52ec06bc00 0x7f52ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:03.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.428+0000 7f5311c00640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52ec06bc00 0x7f52ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:03.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.429+0000 7f5311c00640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52ec06bc00 0x7f52ec06e0c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f53000015d0 tx=0x7f5300009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:03.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.434+0000 7f5309600640 1 -- 172.21.15.102:0/2206861400 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5304079030 con 0x7f530c157d90 2024-03-18T16:33:03.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.448+0000 7f5309600640 1 -- 172.21.15.102:0/2206861400 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f530404e020 con 0x7f530c157d90 2024-03-18T16:33:03.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.698+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f530c0cc470 con 0x7f530c157d90 2024-03-18T16:33:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.711+0000 7f5309600640 1 -- 172.21.15.102:0/2206861400 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f530407b690 con 0x7f530c157d90 2024-03-18T16:33:03.713 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:03.713 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.714+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52ec06bc00 msgr2=0x7f52ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.714+0000 7f5313000640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52ec06bc00 0x7f52ec06e0c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f53000015d0 tx=0x7f5300009040 comp rx=0 tx=0).stop 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.714+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c157d90 msgr2=0x7f530c154160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.714+0000 7f5313000640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c157d90 0x7f530c154160 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5304032000 tx=0x7f5304004a40 comp rx=0 tx=0).stop 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.715+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 shutdown_connections 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.715+0000 7f5313000640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52ec06bc00 0x7f52ec06e0c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.715+0000 7f5313000640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f530c157d90 0x7f530c154160 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.715+0000 7f5313000640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0c7f30 0x7f530c1f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.715+0000 7f5313000640 1 --2- 172.21.15.102:0/2206861400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f530c072810 0x7f530c15af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.715+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 >> 172.21.15.102:0/2206861400 conn(0x7f530c0cb890 msgr2=0x7f530c0c2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.715+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 shutdown_connections 2024-03-18T16:33:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:03.715+0000 7f5313000640 1 -- 172.21.15.102:0/2206861400 wait complete. 2024-03-18T16:33:04.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:04.346 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:05.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:05.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.521+0000 7f6aad200640 1 Processor -- start 2024-03-18T16:33:05.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.521+0000 7f6aad200640 1 -- start start 2024-03-18T16:33:05.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.522+0000 7f6aad200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6aa8141650 0x7f6aa8158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:05.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.522+0000 7f6aad200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa80c72d0 con 0x7f6aa815c3a0 2024-03-18T16:33:05.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.522+0000 7f6aad200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa8158ad0 con 0x7f6aa80c4fa0 2024-03-18T16:33:05.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.522+0000 7f6aad200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6aa8159370 con 0x7f6aa8141650 2024-03-18T16:33:05.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.523+0000 7f6aa7400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6aa8141650 0x7f6aa8158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:05.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.523+0000 7f6aa7400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6aa8141650 0x7f6aa8158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59990/0 (socket says 172.21.15.102:59990) 2024-03-18T16:33:05.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.523+0000 7f6aa7400640 1 -- 172.21.15.102:0/2223376724 learned_addr learned my addr 172.21.15.102:0/2223376724 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:05.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.527+0000 7f6aa7400640 1 -- 172.21.15.102:0/2223376724 >> v1:172.21.15.154:6789/0 conn(0x7f6aa80c4fa0 legacy=0x7f6aa8162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:05.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.527+0000 7f6aa7400640 1 -- 172.21.15.102:0/2223376724 >> v1:172.21.15.102:6789/0 conn(0x7f6aa815c3a0 legacy=0x7f6aa80c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:05.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.527+0000 7f6aa7400640 1 -- 172.21.15.102:0/2223376724 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa8158db0 con 0x7f6aa8141650 2024-03-18T16:33:05.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.531+0000 7f6aa7400640 1 --2- 172.21.15.102:0/2223376724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa8141650 0x7f6aa8158420 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009870 tx=0x7f6a9802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b10f2dabf2f117 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:05.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.535+0000 7f6aa6000640 1 -- 172.21.15.102:0/2223376724 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9802f850 con 0x7f6aa8141650 2024-03-18T16:33:05.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.535+0000 7f6aa6000640 1 -- 172.21.15.102:0/2223376724 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a9802f9f0 con 0x7f6aa8141650 2024-03-18T16:33:05.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.535+0000 7f6aa6000640 1 -- 172.21.15.102:0/2223376724 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a9802fdc0 con 0x7f6aa8141650 2024-03-18T16:33:05.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.536+0000 7f6aad200640 1 -- 172.21.15.102:0/2223376724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa8141650 msgr2=0x7f6aa8158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:05.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.536+0000 7f6aad200640 1 --2- 172.21.15.102:0/2223376724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa8141650 0x7f6aa8158420 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009870 tx=0x7f6a9802ba30 comp rx=0 tx=0).stop 2024-03-18T16:33:05.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.536+0000 7f6aad200640 1 -- 172.21.15.102:0/2223376724 shutdown_connections 2024-03-18T16:33:05.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.536+0000 7f6aad200640 1 --2- 172.21.15.102:0/2223376724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa8141650 0x7f6aa8158420 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:05.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.536+0000 7f6aad200640 1 -- 172.21.15.102:0/2223376724 >> 172.21.15.102:0/2223376724 conn(0x7f6aa80cb890 msgr2=0x7f6aa80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:05.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.536+0000 7f6aad200640 1 -- 172.21.15.102:0/2223376724 shutdown_connections 2024-03-18T16:33:05.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.536+0000 7f6aad200640 1 -- 172.21.15.102:0/2223376724 wait complete. 2024-03-18T16:33:05.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.538+0000 7f6aad200640 1 Processor -- start 2024-03-18T16:33:05.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.538+0000 7f6aad200640 1 -- start start 2024-03-18T16:33:05.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.539+0000 7f6aad200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa80c4fa0 0x7f6aa8157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:05.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.539+0000 7f6aad200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa8141650 0x7f6aa81e40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:05.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.539+0000 7f6aad200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa815c3a0 0x7f6aa81e2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:05.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.539+0000 7f6aad200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa80c66a0 con 0x7f6aa8141650 2024-03-18T16:33:05.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.539+0000 7f6aad200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa80c63c0 con 0x7f6aa815c3a0 2024-03-18T16:33:05.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.539+0000 7f6aad200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa815c7c0 con 0x7f6aa80c4fa0 2024-03-18T16:33:05.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.540+0000 7f6aa7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa815c3a0 0x7f6aa81e2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:05.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.540+0000 7f6aa7400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa80c4fa0 0x7f6aa8157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:05.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.540+0000 7f6aa6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa8141650 0x7f6aa81e40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:05.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.540+0000 7f6aa7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa815c3a0 0x7f6aa81e2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:59994/0 (socket says 172.21.15.102:59994) 2024-03-18T16:33:05.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.540+0000 7f6aa7e00640 1 -- 172.21.15.102:0/2835715563 learned_addr learned my addr 172.21.15.102:0/2835715563 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:05.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.540+0000 7f6aa7400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa80c4fa0 0x7f6aa8157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53226/0 (socket says 172.21.15.102:53226) 2024-03-18T16:33:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.544+0000 7f6aa6a00640 1 -- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa80c4fa0 msgr2=0x7f6aa8157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.544+0000 7f6aa6a00640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa80c4fa0 0x7f6aa8157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.544+0000 7f6aa6a00640 1 -- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa815c3a0 msgr2=0x7f6aa81e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.544+0000 7f6aa6a00640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa815c3a0 0x7f6aa81e2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.544+0000 7f6aa6a00640 1 -- 172.21.15.102:0/2835715563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa815b340 con 0x7f6aa8141650 2024-03-18T16:33:05.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.547+0000 7f6aa6a00640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa8141650 0x7f6aa81e40f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400c3f0 tx=0x7f6a9400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:05.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.551+0000 7f6a9f400640 1 -- 172.21.15.102:0/2835715563 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a940052d0 con 0x7f6aa8141650 2024-03-18T16:33:05.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.552+0000 7f6a9f400640 1 -- 172.21.15.102:0/2835715563 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a94005490 con 0x7f6aa8141650 2024-03-18T16:33:05.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.552+0000 7f6a9f400640 1 -- 172.21.15.102:0/2835715563 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94007260 con 0x7f6aa8141650 2024-03-18T16:33:05.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.552+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa815b5e0 con 0x7f6aa8141650 2024-03-18T16:33:05.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.552+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa815a1f0 con 0x7f6aa8141650 2024-03-18T16:33:05.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.553+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa8074fd0 con 0x7f6aa8141650 2024-03-18T16:33:05.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.587+0000 7f6a9f400640 1 -- 172.21.15.102:0/2835715563 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6a94007400 con 0x7f6aa8141650 2024-03-18T16:33:05.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.589+0000 7f6a9f400640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:05.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.589+0000 7f6aa7400640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:05.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.590+0000 7f6aa7400640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6a98036000 tx=0x7f6a98008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:05.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.790+0000 7f6a9f400640 1 -- 172.21.15.102:0/2835715563 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6a94056820 con 0x7f6aa8141650 2024-03-18T16:33:05.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:05.867+0000 7f6a9f400640 1 -- 172.21.15.102:0/2835715563 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6a94058780 con 0x7f6aa8141650 2024-03-18T16:33:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.107+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6aa813ea20 con 0x7f6aa8141650 2024-03-18T16:33:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.129+0000 7f6a9f400640 1 -- 172.21.15.102:0/2835715563 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f6a9405c820 con 0x7f6aa8141650 2024-03-18T16:33:06.131 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:06.131 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:06.131 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:06.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.131+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 msgr2=0x7f6a8806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.131+0000 7f6aad200640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6a98036000 tx=0x7f6a98008030 comp rx=0 tx=0).stop 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.131+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa8141650 msgr2=0x7f6aa81e40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.131+0000 7f6aad200640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa8141650 0x7f6aa81e40f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400c3f0 tx=0x7f6a9400c8b0 comp rx=0 tx=0).stop 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.131+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 shutdown_connections 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.131+0000 7f6aad200640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6a8806bc20 0x7f6a8806e0e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.132+0000 7f6aad200640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6aa815c3a0 0x7f6aa81e2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.132+0000 7f6aad200640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6aa8141650 0x7f6aa81e40f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.132+0000 7f6aad200640 1 --2- 172.21.15.102:0/2835715563 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6aa80c4fa0 0x7f6aa8157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.132+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 >> 172.21.15.102:0/2835715563 conn(0x7f6aa80cb890 msgr2=0x7f6aa813ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.132+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 shutdown_connections 2024-03-18T16:33:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:06.132+0000 7f6aad200640 1 -- 172.21.15.102:0/2835715563 wait complete. 2024-03-18T16:33:06.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:06.755 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:07.756 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:07.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.931+0000 7fc645600640 1 Processor -- start 2024-03-18T16:33:07.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.931+0000 7fc645600640 1 -- start start 2024-03-18T16:33:07.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.932+0000 7fc645600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:07.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.932+0000 7fc645600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc6401589f0 0x7fc6400704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:07.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.932+0000 7fc645600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc640158e60 con 0x7fc640152120 2024-03-18T16:33:07.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.932+0000 7fc645600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc640158560 con 0x7fc6401589f0 2024-03-18T16:33:07.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.932+0000 7fc645600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc6400c3150 con 0x7fc64015b4d0 2024-03-18T16:33:07.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.933+0000 7fc63f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:07.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.933+0000 7fc644c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc6401589f0 0x7fc6400704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:07.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.933+0000 7fc63f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400cacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53236/0 (socket says 172.21.15.102:53236) 2024-03-18T16:33:07.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.933+0000 7fc63f400640 1 -- 172.21.15.102:0/257128626 learned_addr learned my addr 172.21.15.102:0/257128626 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:07.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.935+0000 7fc63f400640 1 -- 172.21.15.102:0/257128626 >> v2:172.21.15.102:3300/0 conn(0x7fc6401589f0 msgr2=0x7fc6400704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:07.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.935+0000 7fc63f400640 1 --2- 172.21.15.102:0/257128626 >> v2:172.21.15.102:3300/0 conn(0x7fc6401589f0 0x7fc6400704c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:07.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.935+0000 7fc63f400640 1 -- 172.21.15.102:0/257128626 >> v1:172.21.15.102:6789/0 conn(0x7fc640152120 legacy=0x7fc6400c2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:07.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.936+0000 7fc63f400640 1 -- 172.21.15.102:0/257128626 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6400c3430 con 0x7fc64015b4d0 2024-03-18T16:33:07.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.938+0000 7fc63f400640 1 --2- 172.21.15.102:0/257128626 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400cacc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009870 tx=0x7fc62c02ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=450af9031299d149 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:07.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.943+0000 7fc63ea00640 1 -- 172.21.15.102:0/257128626 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c02f830 con 0x7fc64015b4d0 2024-03-18T16:33:07.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.943+0000 7fc63ea00640 1 -- 172.21.15.102:0/257128626 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc62c02f9d0 con 0x7fc64015b4d0 2024-03-18T16:33:07.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.943+0000 7fc63ea00640 1 -- 172.21.15.102:0/257128626 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c02fda0 con 0x7fc64015b4d0 2024-03-18T16:33:07.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.944+0000 7fc645600640 1 -- 172.21.15.102:0/257128626 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 msgr2=0x7fc6400cacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:07.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.944+0000 7fc645600640 1 --2- 172.21.15.102:0/257128626 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400cacc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009870 tx=0x7fc62c02ba10 comp rx=0 tx=0).stop 2024-03-18T16:33:07.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.944+0000 7fc645600640 1 -- 172.21.15.102:0/257128626 shutdown_connections 2024-03-18T16:33:07.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.944+0000 7fc645600640 1 --2- 172.21.15.102:0/257128626 >> v2:172.21.15.102:3300/0 conn(0x7fc6401589f0 0x7fc6400704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:07.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.944+0000 7fc645600640 1 --2- 172.21.15.102:0/257128626 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400cacc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:07.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.944+0000 7fc645600640 1 -- 172.21.15.102:0/257128626 >> 172.21.15.102:0/257128626 conn(0x7fc640067190 msgr2=0x7fc6400675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:07.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.944+0000 7fc645600640 1 -- 172.21.15.102:0/257128626 shutdown_connections 2024-03-18T16:33:07.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.945+0000 7fc645600640 1 -- 172.21.15.102:0/257128626 wait complete. 2024-03-18T16:33:07.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.946+0000 7fc645600640 1 Processor -- start 2024-03-18T16:33:07.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.946+0000 7fc645600640 1 -- start start 2024-03-18T16:33:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.947+0000 7fc645600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc640152120 0x7fc640142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.947+0000 7fc645600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6401589f0 0x7fc6400c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.947+0000 7fc645600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.947+0000 7fc645600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc640157ee0 con 0x7fc640152120 2024-03-18T16:33:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.947+0000 7fc645600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6401581f0 con 0x7fc6401589f0 2024-03-18T16:33:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.947+0000 7fc645600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc640152500 con 0x7fc64015b4d0 2024-03-18T16:33:07.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.948+0000 7fc644c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:07.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.948+0000 7fc63f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6401589f0 0x7fc6400c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:07.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.948+0000 7fc63fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc640152120 0x7fc640142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:07.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.948+0000 7fc63f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6401589f0 0x7fc6400c80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:60006/0 (socket says 172.21.15.102:60006) 2024-03-18T16:33:07.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.948+0000 7fc63f400640 1 -- 172.21.15.102:0/1585364721 learned_addr learned my addr 172.21.15.102:0/1585364721 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:07.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.952+0000 7fc63f400640 1 -- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 msgr2=0x7fc6400c85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:07.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.952+0000 7fc63f400640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400c85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:07.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.952+0000 7fc63f400640 1 -- 172.21.15.102:0/1585364721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc640152120 msgr2=0x7fc640142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:07.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.952+0000 7fc63f400640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc640152120 0x7fc640142940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:07.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.952+0000 7fc63f400640 1 -- 172.21.15.102:0/1585364721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc640142e80 con 0x7fc6401589f0 2024-03-18T16:33:07.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.954+0000 7fc63f400640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6401589f0 0x7fc6400c80b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc62c000c00 tx=0x7fc62c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:07.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.958+0000 7fc637e00640 1 -- 172.21.15.102:0/1585364721 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c037c40 con 0x7fc6401589f0 2024-03-18T16:33:07.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.958+0000 7fc637e00640 1 -- 172.21.15.102:0/1585364721 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc62c046070 con 0x7fc6401589f0 2024-03-18T16:33:07.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.958+0000 7fc637e00640 1 -- 172.21.15.102:0/1585364721 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c041400 con 0x7fc6401589f0 2024-03-18T16:33:07.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.958+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc64014aae0 con 0x7fc6401589f0 2024-03-18T16:33:07.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.958+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6400c7750 con 0x7fc6401589f0 2024-03-18T16:33:07.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.959+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc64006f900 con 0x7fc6401589f0 2024-03-18T16:33:07.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.985+0000 7fc637e00640 1 -- 172.21.15.102:0/1585364721 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc62c040070 con 0x7fc6401589f0 2024-03-18T16:33:07.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.986+0000 7fc637e00640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc62006bbd0 0x7fc62006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:07.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.987+0000 7fc63fe00640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc62006bbd0 0x7fc62006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:07.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:07.988+0000 7fc63fe00640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc62006bbd0 0x7fc62006e090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc630001780 tx=0x7fc630009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:08.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.247+0000 7fc637e00640 1 -- 172.21.15.102:0/1585364721 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fc62c079f00 con 0x7fc6401589f0 2024-03-18T16:33:08.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.257+0000 7fc637e00640 1 -- 172.21.15.102:0/1585364721 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc64006f900 con 0x7fc6401589f0 2024-03-18T16:33:08.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.504+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc640074fd0 con 0x7fc6401589f0 2024-03-18T16:33:08.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.516+0000 7fc637e00640 1 -- 172.21.15.102:0/1585364721 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fc62c080e20 con 0x7fc6401589f0 2024-03-18T16:33:08.518 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:08.519 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:08.519 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:08.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc62006bbd0 msgr2=0x7fc62006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:08.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc62006bbd0 0x7fc62006e090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc630001780 tx=0x7fc630009040 comp rx=0 tx=0).stop 2024-03-18T16:33:08.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6401589f0 msgr2=0x7fc6400c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:08.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6401589f0 0x7fc6400c80b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc62c000c00 tx=0x7fc62c004730 comp rx=0 tx=0).stop 2024-03-18T16:33:08.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 shutdown_connections 2024-03-18T16:33:08.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc62006bbd0 0x7fc62006e090 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:08.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc64015b4d0 0x7fc6400c85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:08.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6401589f0 0x7fc6400c80b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:08.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 --2- 172.21.15.102:0/1585364721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc640152120 0x7fc640142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:08.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 >> 172.21.15.102:0/1585364721 conn(0x7fc640067190 msgr2=0x7fc64014b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:08.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 shutdown_connections 2024-03-18T16:33:08.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:08.519+0000 7fc645600640 1 -- 172.21.15.102:0/1585364721 wait complete. 2024-03-18T16:33:09.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:09.141 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:10.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:10.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.316+0000 7f7bdb000640 1 Processor -- start 2024-03-18T16:33:10.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.316+0000 7f7bdb000640 1 -- start start 2024-03-18T16:33:10.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.317+0000 7f7bdb000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cfe70 0x7f7bd415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:10.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.318+0000 7f7bdb000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7bd40d0250 0x7f7bd40c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:10.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.318+0000 7f7bdb000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bd40cabd0 con 0x7f7bd40cc2f0 2024-03-18T16:33:10.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.318+0000 7f7bdb000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd40caea0 con 0x7f7bd40d0250 2024-03-18T16:33:10.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.318+0000 7f7bdb000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd40cb180 con 0x7f7bd40cfe70 2024-03-18T16:33:10.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.318+0000 7f7bd9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cfe70 0x7f7bd415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:10.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.319+0000 7f7bd9200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7bd40d0250 0x7f7bd40c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:10.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.319+0000 7f7bd9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cfe70 0x7f7bd415ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53252/0 (socket says 172.21.15.102:53252) 2024-03-18T16:33:10.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.319+0000 7f7bd9c00640 1 -- 172.21.15.102:0/745853016 learned_addr learned my addr 172.21.15.102:0/745853016 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:10.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.321+0000 7f7bd9c00640 1 -- 172.21.15.102:0/745853016 >> v2:172.21.15.102:3300/0 conn(0x7f7bd40d0250 msgr2=0x7f7bd40c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:10.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.321+0000 7f7bd9c00640 1 --2- 172.21.15.102:0/745853016 >> v2:172.21.15.102:3300/0 conn(0x7f7bd40d0250 0x7f7bd40c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:10.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.321+0000 7f7bd9c00640 1 -- 172.21.15.102:0/745853016 >> v1:172.21.15.102:6789/0 conn(0x7f7bd40cc2f0 legacy=0x7f7bd40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:10.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.321+0000 7f7bd9c00640 1 -- 172.21.15.102:0/745853016 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd40cb460 con 0x7f7bd40cfe70 2024-03-18T16:33:10.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.324+0000 7f7bd9c00640 1 --2- 172.21.15.102:0/745853016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cfe70 0x7f7bd415ae40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009870 tx=0x7f7bc802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4797952dc328b7af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:10.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.329+0000 7f7bd3e00640 1 -- 172.21.15.102:0/745853016 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc802f830 con 0x7f7bd40cfe70 2024-03-18T16:33:10.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.329+0000 7f7bd3e00640 1 -- 172.21.15.102:0/745853016 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bc802f9d0 con 0x7f7bd40cfe70 2024-03-18T16:33:10.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.329+0000 7f7bd3e00640 1 -- 172.21.15.102:0/745853016 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc802fda0 con 0x7f7bd40cfe70 2024-03-18T16:33:10.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.330+0000 7f7bdb000640 1 -- 172.21.15.102:0/745853016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cfe70 msgr2=0x7f7bd415ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:10.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.330+0000 7f7bdb000640 1 --2- 172.21.15.102:0/745853016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cfe70 0x7f7bd415ae40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009870 tx=0x7f7bc802ba10 comp rx=0 tx=0).stop 2024-03-18T16:33:10.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.330+0000 7f7bdb000640 1 -- 172.21.15.102:0/745853016 shutdown_connections 2024-03-18T16:33:10.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.330+0000 7f7bdb000640 1 --2- 172.21.15.102:0/745853016 >> v2:172.21.15.102:3300/0 conn(0x7f7bd40d0250 0x7f7bd40c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:10.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.330+0000 7f7bdb000640 1 --2- 172.21.15.102:0/745853016 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cfe70 0x7f7bd415ae40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:10.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.330+0000 7f7bdb000640 1 -- 172.21.15.102:0/745853016 >> 172.21.15.102:0/745853016 conn(0x7f7bd408c3e0 msgr2=0x7f7bd406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:10.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.330+0000 7f7bdb000640 1 -- 172.21.15.102:0/745853016 shutdown_connections 2024-03-18T16:33:10.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.330+0000 7f7bdb000640 1 -- 172.21.15.102:0/745853016 wait complete. 2024-03-18T16:33:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.333+0000 7f7bdb000640 1 Processor -- start 2024-03-18T16:33:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.333+0000 7f7bdb000640 1 -- start start 2024-03-18T16:33:10.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.334+0000 7f7bdb000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cc2f0 0x7f7bd41f3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:10.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.334+0000 7f7bdb000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bd40cfe70 0x7f7bd41e18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:10.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.334+0000 7f7bdb000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bd40d0250 0x7f7bd4151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:10.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.334+0000 7f7bdb000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd40cbd00 con 0x7f7bd40cfe70 2024-03-18T16:33:10.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.334+0000 7f7bdb000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd40cba20 con 0x7f7bd40d0250 2024-03-18T16:33:10.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.334+0000 7f7bdb000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd40c6b20 con 0x7f7bd40cc2f0 2024-03-18T16:33:10.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.335+0000 7f7bd9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cc2f0 0x7f7bd41f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:10.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.335+0000 7f7bda600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bd40d0250 0x7f7bd4151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:10.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.335+0000 7f7bd9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bd40cfe70 0x7f7bd41e18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:10.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.335+0000 7f7bd9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cc2f0 0x7f7bd41f3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53264/0 (socket says 172.21.15.102:53264) 2024-03-18T16:33:10.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.335+0000 7f7bd9c00640 1 -- 172.21.15.102:0/4099039589 learned_addr learned my addr 172.21.15.102:0/4099039589 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:10.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.339+0000 7f7bda600640 1 -- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cc2f0 msgr2=0x7f7bd41f3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:10.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.339+0000 7f7bda600640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cc2f0 0x7f7bd41f3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:10.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.339+0000 7f7bda600640 1 -- 172.21.15.102:0/4099039589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bd40cfe70 msgr2=0x7f7bd41e18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:10.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.339+0000 7f7bda600640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bd40cfe70 0x7f7bd41e18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:10.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.339+0000 7f7bda600640 1 -- 172.21.15.102:0/4099039589 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd41f3b60 con 0x7f7bd40d0250 2024-03-18T16:33:10.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.342+0000 7f7bda600640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bd40d0250 0x7f7bd4151140 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00c6c0 tx=0x7f7bcc00cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:10.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.346+0000 7f7bd1600640 1 -- 172.21.15.102:0/4099039589 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc003b70 con 0x7f7bd40d0250 2024-03-18T16:33:10.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.346+0000 7f7bd1600640 1 -- 172.21.15.102:0/4099039589 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bcc003d10 con 0x7f7bd40d0250 2024-03-18T16:33:10.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.346+0000 7f7bd1600640 1 -- 172.21.15.102:0/4099039589 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc0052f0 con 0x7f7bd40d0250 2024-03-18T16:33:10.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.346+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd41f3e40 con 0x7f7bd40d0250 2024-03-18T16:33:10.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.346+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd41e10b0 con 0x7f7bd40d0250 2024-03-18T16:33:10.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.348+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd4074fd0 con 0x7f7bd40d0250 2024-03-18T16:33:10.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.372+0000 7f7bd1600640 1 -- 172.21.15.102:0/4099039589 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7bcc005490 con 0x7f7bd40d0250 2024-03-18T16:33:10.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.373+0000 7f7bd1600640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bb406bc20 0x7f7bb406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:10.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.374+0000 7f7bd9c00640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bb406bc20 0x7f7bb406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:10.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.374+0000 7f7bd9c00640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bb406bc20 0x7f7bb406e0e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009870 tx=0x7f7bc8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:10.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.630+0000 7f7bd1600640 1 -- 172.21.15.102:0/4099039589 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7bcc057810 con 0x7f7bd40d0250 2024-03-18T16:33:10.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.641+0000 7f7bd1600640 1 -- 172.21.15.102:0/4099039589 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7bcc059770 con 0x7f7bd40d0250 2024-03-18T16:33:10.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.886+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7bd40c6670 con 0x7f7bd40d0250 2024-03-18T16:33:10.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.899+0000 7f7bd1600640 1 -- 172.21.15.102:0/4099039589 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7bcc05d810 con 0x7f7bd40d0250 2024-03-18T16:33:10.901 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:10.901 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:10.901 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:10.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.902+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bb406bc20 msgr2=0x7f7bb406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.902+0000 7f7bdb000640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bb406bc20 0x7f7bb406e0e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7bc8009870 tx=0x7f7bc8008030 comp rx=0 tx=0).stop 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.902+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bd40d0250 msgr2=0x7f7bd4151140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.902+0000 7f7bdb000640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bd40d0250 0x7f7bd4151140 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc00c6c0 tx=0x7f7bcc00cb80 comp rx=0 tx=0).stop 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.903+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 shutdown_connections 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.903+0000 7f7bdb000640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7bb406bc20 0x7f7bb406e0e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.903+0000 7f7bdb000640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bd40d0250 0x7f7bd4151140 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.903+0000 7f7bdb000640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7bd40cfe70 0x7f7bd41e18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.903+0000 7f7bdb000640 1 --2- 172.21.15.102:0/4099039589 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7bd40cc2f0 0x7f7bd41f3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.903+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 >> 172.21.15.102:0/4099039589 conn(0x7f7bd408c3e0 msgr2=0x7f7bd4066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.903+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 shutdown_connections 2024-03-18T16:33:10.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:10.903+0000 7f7bdb000640 1 -- 172.21.15.102:0/4099039589 wait complete. 2024-03-18T16:33:11.525 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:11.525 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:12.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.717+0000 7fd179200640 1 Processor -- start 2024-03-18T16:33:12.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.717+0000 7fd179200640 1 -- start start 2024-03-18T16:33:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.718+0000 7fd179200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd1741524c0 0x7fd174072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.719+0000 7fd179200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd174158ee0 0x7fd17406e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.719+0000 7fd179200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd17415c200 con 0x7fd17415b8e0 2024-03-18T16:33:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.719+0000 7fd179200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd174159300 con 0x7fd1741524c0 2024-03-18T16:33:12.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.719+0000 7fd179200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd17415c4e0 con 0x7fd174158ee0 2024-03-18T16:33:12.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.719+0000 7fd171e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd1741524c0 0x7fd174072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:12.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.720+0000 7fd171400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd174158ee0 0x7fd17406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:12.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.720+0000 7fd171400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd174158ee0 0x7fd17406e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42950/0 (socket says 172.21.15.102:42950) 2024-03-18T16:33:12.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.720+0000 7fd171400640 1 -- 172.21.15.102:0/3523462988 learned_addr learned my addr 172.21.15.102:0/3523462988 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:12.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.722+0000 7fd171e00640 1 -- 172.21.15.102:0/3523462988 >> v2:172.21.15.154:3300/0 conn(0x7fd174158ee0 msgr2=0x7fd17406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:12.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.722+0000 7fd171e00640 1 --2- 172.21.15.102:0/3523462988 >> v2:172.21.15.154:3300/0 conn(0x7fd174158ee0 0x7fd17406e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:12.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.723+0000 7fd171e00640 1 -- 172.21.15.102:0/3523462988 >> v1:172.21.15.154:6789/0 conn(0x7fd17415b8e0 legacy=0x7fd1740c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:12.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.723+0000 7fd171e00640 1 -- 172.21.15.102:0/3523462988 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17415c7c0 con 0x7fd1741524c0 2024-03-18T16:33:12.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.725+0000 7fd171e00640 1 --2- 172.21.15.102:0/3523462988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1741524c0 0x7fd174072700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd1740c9670 tx=0x7fd16402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec43a930eb6409a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:12.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.730+0000 7fd170a00640 1 -- 172.21.15.102:0/3523462988 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164035020 con 0x7fd1741524c0 2024-03-18T16:33:12.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.730+0000 7fd170a00640 1 -- 172.21.15.102:0/3523462988 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd164031830 con 0x7fd1741524c0 2024-03-18T16:33:12.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.730+0000 7fd170a00640 1 -- 172.21.15.102:0/3523462988 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164031c00 con 0x7fd1741524c0 2024-03-18T16:33:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.731+0000 7fd179200640 1 -- 172.21.15.102:0/3523462988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1741524c0 msgr2=0x7fd174072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.731+0000 7fd179200640 1 --2- 172.21.15.102:0/3523462988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1741524c0 0x7fd174072700 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd1740c9670 tx=0x7fd16402b8d0 comp rx=0 tx=0).stop 2024-03-18T16:33:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.731+0000 7fd179200640 1 -- 172.21.15.102:0/3523462988 shutdown_connections 2024-03-18T16:33:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.731+0000 7fd179200640 1 --2- 172.21.15.102:0/3523462988 >> v2:172.21.15.154:3300/0 conn(0x7fd174158ee0 0x7fd17406e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.731+0000 7fd179200640 1 --2- 172.21.15.102:0/3523462988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1741524c0 0x7fd174072700 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.731+0000 7fd179200640 1 -- 172.21.15.102:0/3523462988 >> 172.21.15.102:0/3523462988 conn(0x7fd1740700d0 msgr2=0x7fd17406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:12.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.731+0000 7fd179200640 1 -- 172.21.15.102:0/3523462988 shutdown_connections 2024-03-18T16:33:12.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.732+0000 7fd179200640 1 -- 172.21.15.102:0/3523462988 wait complete. 2024-03-18T16:33:12.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.734+0000 7fd179200640 1 Processor -- start 2024-03-18T16:33:12.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.734+0000 7fd179200640 1 -- start start 2024-03-18T16:33:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.736+0000 7fd179200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1741524c0 0x7fd17414c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.736+0000 7fd179200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd174158ee0 0x7fd1741595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:12.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.736+0000 7fd179200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17415b8e0 0x7fd174159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.736+0000 7fd179200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd1741589c0 con 0x7fd17415b8e0 2024-03-18T16:33:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.736+0000 7fd179200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1741586e0 con 0x7fd174158ee0 2024-03-18T16:33:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.736+0000 7fd179200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd17406e880 con 0x7fd1741524c0 2024-03-18T16:33:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.737+0000 7fd171400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd174158ee0 0x7fd1741595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.737+0000 7fd171e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1741524c0 0x7fd17414c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:12.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.737+0000 7fd172800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17415b8e0 0x7fd174159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:12.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.737+0000 7fd171400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd174158ee0 0x7fd1741595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42960/0 (socket says 172.21.15.102:42960) 2024-03-18T16:33:12.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.737+0000 7fd171400640 1 -- 172.21.15.102:0/2149152755 learned_addr learned my addr 172.21.15.102:0/2149152755 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.741+0000 7fd171e00640 1 -- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd174158ee0 msgr2=0x7fd1741595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.741+0000 7fd171e00640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd174158ee0 0x7fd1741595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:12.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.741+0000 7fd171e00640 1 -- 172.21.15.102:0/2149152755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17415b8e0 msgr2=0x7fd174159af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.741+0000 7fd171e00640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17415b8e0 0x7fd174159af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:12.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.741+0000 7fd171e00640 1 -- 172.21.15.102:0/2149152755 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd174071b80 con 0x7fd1741524c0 2024-03-18T16:33:12.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.744+0000 7fd171e00640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1741524c0 0x7fd17414c780 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd1740c5d10 tx=0x7fd1640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:12.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.747+0000 7fd16a000640 1 -- 172.21.15.102:0/2149152755 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164041070 con 0x7fd1741524c0 2024-03-18T16:33:12.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.747+0000 7fd16a000640 1 -- 172.21.15.102:0/2149152755 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd16403f070 con 0x7fd1741524c0 2024-03-18T16:33:12.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.747+0000 7fd16a000640 1 -- 172.21.15.102:0/2149152755 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164035040 con 0x7fd1741524c0 2024-03-18T16:33:12.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.747+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1740c81b0 con 0x7fd1741524c0 2024-03-18T16:33:12.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.748+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1740c6c30 con 0x7fd1741524c0 2024-03-18T16:33:12.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.749+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd174074fd0 con 0x7fd1741524c0 2024-03-18T16:33:12.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.770+0000 7fd16a000640 1 -- 172.21.15.102:0/2149152755 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd164038020 con 0x7fd1741524c0 2024-03-18T16:33:12.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.771+0000 7fd16a000640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd15406bc50 0x7fd15406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:12.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.771+0000 7fd171400640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd15406bc50 0x7fd15406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:12.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:12.772+0000 7fd171400640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd15406bc50 0x7fd15406e110 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd17415acf0 tx=0x7fd160009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:13.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.006+0000 7fd16a000640 1 -- 172.21.15.102:0/2149152755 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd164078dd0 con 0x7fd1741524c0 2024-03-18T16:33:13.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.017+0000 7fd16a000640 1 -- 172.21.15.102:0/2149152755 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd164054020 con 0x7fd1741524c0 2024-03-18T16:33:13.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.262+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd174156560 con 0x7fd1741524c0 2024-03-18T16:33:13.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.275+0000 7fd16a000640 1 -- 172.21.15.102:0/2149152755 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd164031800 con 0x7fd1741524c0 2024-03-18T16:33:13.277 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:13.277 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:13.277 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd15406bc50 msgr2=0x7fd15406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd15406bc50 0x7fd15406e110 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd17415acf0 tx=0x7fd160009240 comp rx=0 tx=0).stop 2024-03-18T16:33:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1741524c0 msgr2=0x7fd17414c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:13.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1741524c0 0x7fd17414c780 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd1740c5d10 tx=0x7fd1640046c0 comp rx=0 tx=0).stop 2024-03-18T16:33:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 shutdown_connections 2024-03-18T16:33:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd15406bc50 0x7fd15406e110 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd17415b8e0 0x7fd174159af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd174158ee0 0x7fd1741595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 --2- 172.21.15.102:0/2149152755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd1741524c0 0x7fd17414c780 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.277+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 >> 172.21.15.102:0/2149152755 conn(0x7fd1740700d0 msgr2=0x7fd1740c51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.278+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 shutdown_connections 2024-03-18T16:33:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:13.278+0000 7fd179200640 1 -- 172.21.15.102:0/2149152755 wait complete. 2024-03-18T16:33:13.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:13.899 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:14.415 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:40.674 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:33:14.419 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:40.679 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:33:14.474 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:12:40.820 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:33:14.476 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:12:40.821 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:33:14.514 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:40.859 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:33:14.518 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:40.863 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:33:14.521 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:12:40.781 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:33:14.550 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:12:40.896 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:33:14.589 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:12:40.934 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:33:14.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:15.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.087+0000 7fa63d200640 1 Processor -- start 2024-03-18T16:33:15.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.088+0000 7fa63d200640 1 -- start start 2024-03-18T16:33:15.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.089+0000 7fa63d200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa63815a6f0 0x7fa63815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:15.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.089+0000 7fa63d200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa63815baa0 0x7fa63815bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:15.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.089+0000 7fa63d200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6380ca500 con 0x7fa63815a310 2024-03-18T16:33:15.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.089+0000 7fa63d200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa6380cd410 con 0x7fa63815a6f0 2024-03-18T16:33:15.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.089+0000 7fa63d200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa63815b080 con 0x7fa63815baa0 2024-03-18T16:33:15.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.090+0000 7fa636a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa63815a6f0 0x7fa63815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:15.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.090+0000 7fa637e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa63815baa0 0x7fa63815bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:15.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.090+0000 7fa636a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa63815a6f0 0x7fa63815ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47552/0 (socket says 172.21.15.102:47552) 2024-03-18T16:33:15.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.090+0000 7fa636a00640 1 -- 172.21.15.102:0/3749005947 learned_addr learned my addr 172.21.15.102:0/3749005947 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.094+0000 7fa637e00640 1 -- 172.21.15.102:0/3749005947 >> v2:172.21.15.102:3300/0 conn(0x7fa63815a6f0 msgr2=0x7fa63815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.094+0000 7fa637e00640 1 --2- 172.21.15.102:0/3749005947 >> v2:172.21.15.102:3300/0 conn(0x7fa63815a6f0 0x7fa63815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.094+0000 7fa637e00640 1 -- 172.21.15.102:0/3749005947 >> v1:172.21.15.102:6789/0 conn(0x7fa63815a310 legacy=0x7fa63806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:15.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.094+0000 7fa637e00640 1 -- 172.21.15.102:0/3749005947 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63815c430 con 0x7fa63815baa0 2024-03-18T16:33:15.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.097+0000 7fa637e00640 1 --2- 172.21.15.102:0/3749005947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 0x7fa63815bef0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa638067f40 tx=0x7fa63002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe4bf66761100766 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:15.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.102+0000 7fa636000640 1 -- 172.21.15.102:0/3749005947 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630039070 con 0x7fa63815baa0 2024-03-18T16:33:15.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.102+0000 7fa636000640 1 -- 172.21.15.102:0/3749005947 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa63002cc80 con 0x7fa63815baa0 2024-03-18T16:33:15.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.102+0000 7fa636000640 1 -- 172.21.15.102:0/3749005947 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630034660 con 0x7fa63815baa0 2024-03-18T16:33:15.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.102+0000 7fa63d200640 1 -- 172.21.15.102:0/3749005947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 msgr2=0x7fa63815bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:15.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.102+0000 7fa63d200640 1 --2- 172.21.15.102:0/3749005947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 0x7fa63815bef0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa638067f40 tx=0x7fa63002bed0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.103+0000 7fa63d200640 1 -- 172.21.15.102:0/3749005947 shutdown_connections 2024-03-18T16:33:15.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.103+0000 7fa63d200640 1 --2- 172.21.15.102:0/3749005947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 0x7fa63815bef0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.103+0000 7fa63d200640 1 --2- 172.21.15.102:0/3749005947 >> v2:172.21.15.102:3300/0 conn(0x7fa63815a6f0 0x7fa63815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.103+0000 7fa63d200640 1 -- 172.21.15.102:0/3749005947 >> 172.21.15.102:0/3749005947 conn(0x7fa63808c1a0 msgr2=0x7fa638066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:15.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.103+0000 7fa63d200640 1 -- 172.21.15.102:0/3749005947 shutdown_connections 2024-03-18T16:33:15.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.103+0000 7fa63d200640 1 -- 172.21.15.102:0/3749005947 wait complete. 2024-03-18T16:33:15.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.104+0000 7fa63d200640 1 Processor -- start 2024-03-18T16:33:15.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.105+0000 7fa63d200640 1 -- start start 2024-03-18T16:33:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.105+0000 7fa63d200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa63815a310 0x7fa6380c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.105+0000 7fa63d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa63815a6f0 0x7fa638159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.105+0000 7fa63d200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 0x7fa6380c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.105+0000 7fa63d200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380cce50 con 0x7fa63815a6f0 2024-03-18T16:33:15.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.105+0000 7fa63d200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380caa80 con 0x7fa63815baa0 2024-03-18T16:33:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.105+0000 7fa63d200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa6380cd130 con 0x7fa63815a310 2024-03-18T16:33:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.106+0000 7fa636a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa63815a6f0 0x7fa638159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.106+0000 7fa637e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 0x7fa6380c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.107+0000 7fa637400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa63815a310 0x7fa6380c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.107+0000 7fa637e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 0x7fa6380c7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42982/0 (socket says 172.21.15.102:42982) 2024-03-18T16:33:15.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.107+0000 7fa637e00640 1 -- 172.21.15.102:0/343491797 learned_addr learned my addr 172.21.15.102:0/343491797 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.111+0000 7fa637e00640 1 -- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa63815a310 msgr2=0x7fa6380c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.111+0000 7fa637e00640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa63815a310 0x7fa6380c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.111+0000 7fa637e00640 1 -- 172.21.15.102:0/343491797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa63815a6f0 msgr2=0x7fa638159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.111+0000 7fa637e00640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa63815a6f0 0x7fa638159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.111+0000 7fa637e00640 1 -- 172.21.15.102:0/343491797 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa638156ef0 con 0x7fa63815baa0 2024-03-18T16:33:15.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.113+0000 7fa637e00640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 0x7fa6380c7310 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa630009e80 tx=0x7fa630004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.117+0000 7fa62f400640 1 -- 172.21.15.102:0/343491797 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630039050 con 0x7fa63815baa0 2024-03-18T16:33:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.117+0000 7fa62f400640 1 -- 172.21.15.102:0/343491797 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa630045070 con 0x7fa63815baa0 2024-03-18T16:33:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.117+0000 7fa62f400640 1 -- 172.21.15.102:0/343491797 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630009450 con 0x7fa63815baa0 2024-03-18T16:33:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.117+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6381f0440 con 0x7fa63815baa0 2024-03-18T16:33:15.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.117+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6381f09c0 con 0x7fa63815baa0 2024-03-18T16:33:15.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.118+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6380cac90 con 0x7fa63815baa0 2024-03-18T16:33:15.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.142+0000 7fa62f400640 1 -- 172.21.15.102:0/343491797 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa63003e850 con 0x7fa63815baa0 2024-03-18T16:33:15.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.143+0000 7fa62f400640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa61806bc20 0x7fa61806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:15.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.143+0000 7fa637400640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa61806bc20 0x7fa61806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:15.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.144+0000 7fa637400640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa61806bc20 0x7fa61806e0e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa628000f60 tx=0x7fa628009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:15.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.387+0000 7fa62f400640 1 -- 172.21.15.102:0/343491797 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa63007a5a0 con 0x7fa63815baa0 2024-03-18T16:33:15.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.398+0000 7fa62f400640 1 -- 172.21.15.102:0/343491797 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa6380cac90 con 0x7fa63815baa0 2024-03-18T16:33:15.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.647+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6380caed0 con 0x7fa63815baa0 2024-03-18T16:33:15.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.667+0000 7fa62f400640 1 -- 172.21.15.102:0/343491797 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa63007c500 con 0x7fa63815baa0 2024-03-18T16:33:15.669 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:15.669 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:15.669 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:15.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.670+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa61806bc20 msgr2=0x7fa61806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:15.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.670+0000 7fa63d200640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa61806bc20 0x7fa61806e0e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa628000f60 tx=0x7fa628009040 comp rx=0 tx=0).stop 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 msgr2=0x7fa6380c7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 0x7fa6380c7310 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa630009e80 tx=0x7fa630004a40 comp rx=0 tx=0).stop 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 shutdown_connections 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa61806bc20 0x7fa61806e0e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa63815baa0 0x7fa6380c7310 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa63815a6f0 0x7fa638159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 --2- 172.21.15.102:0/343491797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa63815a310 0x7fa6380c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 >> 172.21.15.102:0/343491797 conn(0x7fa63808c1a0 msgr2=0x7fa638153b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 shutdown_connections 2024-03-18T16:33:15.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:15.671+0000 7fa63d200640 1 -- 172.21.15.102:0/343491797 wait complete. 2024-03-18T16:33:16.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:16.300 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:17.301 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:17.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.478+0000 7f3c4c200640 1 Processor -- start 2024-03-18T16:33:17.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.478+0000 7f3c4c200640 1 -- start start 2024-03-18T16:33:17.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.479+0000 7f3c4c200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3c44072700 0x7f3c44158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:17.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.480+0000 7f3c4c200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:17.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.480+0000 7f3c4c200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c4415c200 con 0x7f3c441524c0 2024-03-18T16:33:17.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.480+0000 7f3c4c200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3c44072b20 con 0x7f3c44072700 2024-03-18T16:33:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.480+0000 7f3c4c200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3c4415c4e0 con 0x7f3c4415b8e0 2024-03-18T16:33:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.481+0000 7f3c4a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.481+0000 7f3c4a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:40082/0 (socket says 172.21.15.102:40082) 2024-03-18T16:33:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.481+0000 7f3c4a200640 1 -- 172.21.15.102:0/3505432338 learned_addr learned my addr 172.21.15.102:0/3505432338 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.484+0000 7f3c4a200640 1 -- 172.21.15.102:0/3505432338 >> v2:172.21.15.154:3300/0 conn(0x7f3c44072700 msgr2=0x7f3c44158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.484+0000 7f3c4a200640 1 --2- 172.21.15.102:0/3505432338 >> v2:172.21.15.154:3300/0 conn(0x7f3c44072700 0x7f3c44158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.484+0000 7f3c4a200640 1 -- 172.21.15.102:0/3505432338 >> v1:172.21.15.154:6789/0 conn(0x7f3c441524c0 legacy=0x7f3c4406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:17.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.484+0000 7f3c4a200640 1 -- 172.21.15.102:0/3505432338 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4415c7c0 con 0x7f3c4415b8e0 2024-03-18T16:33:17.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.486+0000 7f3c4a200640 1 --2- 172.21.15.102:0/3505432338 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440c51e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009e50 tx=0x7f3c3c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6540dc0575581a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.490+0000 7f3c43e00640 1 -- 172.21.15.102:0/3505432338 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c02ca70 con 0x7f3c4415b8e0 2024-03-18T16:33:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.491+0000 7f3c43e00640 1 -- 172.21.15.102:0/3505432338 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c3c02cc10 con 0x7f3c4415b8e0 2024-03-18T16:33:17.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.491+0000 7f3c43e00640 1 -- 172.21.15.102:0/3505432338 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0343f0 con 0x7f3c4415b8e0 2024-03-18T16:33:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.491+0000 7f3c4c200640 1 -- 172.21.15.102:0/3505432338 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 msgr2=0x7f3c440c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.491+0000 7f3c4c200640 1 --2- 172.21.15.102:0/3505432338 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440c51e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009e50 tx=0x7f3c3c02bed0 comp rx=0 tx=0).stop 2024-03-18T16:33:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.491+0000 7f3c4c200640 1 -- 172.21.15.102:0/3505432338 shutdown_connections 2024-03-18T16:33:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.491+0000 7f3c4c200640 1 --2- 172.21.15.102:0/3505432338 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440c51e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.491+0000 7f3c4c200640 1 --2- 172.21.15.102:0/3505432338 >> v2:172.21.15.154:3300/0 conn(0x7f3c44072700 0x7f3c44158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.491+0000 7f3c4c200640 1 -- 172.21.15.102:0/3505432338 >> 172.21.15.102:0/3505432338 conn(0x7f3c440700d0 msgr2=0x7f3c4406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:17.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.491+0000 7f3c4c200640 1 -- 172.21.15.102:0/3505432338 shutdown_connections 2024-03-18T16:33:17.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.492+0000 7f3c4c200640 1 -- 172.21.15.102:0/3505432338 wait complete. 2024-03-18T16:33:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.493+0000 7f3c4c200640 1 Processor -- start 2024-03-18T16:33:17.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.493+0000 7f3c4c200640 1 -- start start 2024-03-18T16:33:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.494+0000 7f3c4c200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c44072700 0x7f3c44146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.494+0000 7f3c4c200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c441524c0 0x7f3c440c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.494+0000 7f3c4c200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.494+0000 7f3c4c200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3c441589c0 con 0x7f3c44072700 2024-03-18T16:33:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.494+0000 7f3c4c200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3c441586e0 con 0x7f3c441524c0 2024-03-18T16:33:17.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.494+0000 7f3c4c200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3c441528a0 con 0x7f3c4415b8e0 2024-03-18T16:33:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.495+0000 7f3c4a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.496+0000 7f3c4a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:40084/0 (socket says 172.21.15.102:40084) 2024-03-18T16:33:17.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.496+0000 7f3c4a200640 1 -- 172.21.15.102:0/2357278775 learned_addr learned my addr 172.21.15.102:0/2357278775 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.499+0000 7f3c4a200640 1 -- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c441524c0 msgr2=0x7f3c440c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.499+0000 7f3c4a200640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c441524c0 0x7f3c440c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.499+0000 7f3c4a200640 1 -- 172.21.15.102:0/2357278775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c44072700 msgr2=0x7f3c44146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.499+0000 7f3c4a200640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c44072700 0x7f3c44146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:17.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.499+0000 7f3c4a200640 1 -- 172.21.15.102:0/2357278775 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c441f3fe0 con 0x7f3c4415b8e0 2024-03-18T16:33:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.502+0000 7f3c4a200640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440ca320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c036040 tx=0x7f3c3c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.506+0000 7f3c41600640 1 -- 172.21.15.102:0/2357278775 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c044020 con 0x7f3c4415b8e0 2024-03-18T16:33:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.506+0000 7f3c41600640 1 -- 172.21.15.102:0/2357278775 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c3c040670 con 0x7f3c4415b8e0 2024-03-18T16:33:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.506+0000 7f3c41600640 1 -- 172.21.15.102:0/2357278775 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c040a60 con 0x7f3c4415b8e0 2024-03-18T16:33:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.506+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c441efd10 con 0x7f3c4415b8e0 2024-03-18T16:33:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.506+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c441effb0 con 0x7f3c4415b8e0 2024-03-18T16:33:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.508+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c44074fd0 con 0x7f3c4415b8e0 2024-03-18T16:33:17.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.530+0000 7f3c41600640 1 -- 172.21.15.102:0/2357278775 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3c3c047020 con 0x7f3c4415b8e0 2024-03-18T16:33:17.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.531+0000 7f3c41600640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c2406bc50 0x7f3c2406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:17.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.531+0000 7f3c49800640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c2406bc50 0x7f3c2406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:17.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.532+0000 7f3c49800640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c2406bc50 0x7f3c2406e110 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3c38001560 tx=0x7f3c38007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:17.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.769+0000 7f3c41600640 1 -- 172.21.15.102:0/2357278775 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3c3c07a080 con 0x7f3c4415b8e0 2024-03-18T16:33:17.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:17.780+0000 7f3c41600640 1 -- 172.21.15.102:0/2357278775 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3c3c04e070 con 0x7f3c4415b8e0 2024-03-18T16:33:18.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.020+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c44072f90 con 0x7f3c4415b8e0 2024-03-18T16:33:18.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.036+0000 7f3c41600640 1 -- 172.21.15.102:0/2357278775 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3c3c07c6e0 con 0x7f3c4415b8e0 2024-03-18T16:33:18.038 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:18.038 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:18.038 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c2406bc50 msgr2=0x7f3c2406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c2406bc50 0x7f3c2406e110 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3c38001560 tx=0x7f3c38007990 comp rx=0 tx=0).stop 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 msgr2=0x7f3c440ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440ca320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c036040 tx=0x7f3c3c004a40 comp rx=0 tx=0).stop 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 shutdown_connections 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c2406bc50 0x7f3c2406e110 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c4415b8e0 0x7f3c440ca320 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c441524c0 0x7f3c440c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 --2- 172.21.15.102:0/2357278775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c44072700 0x7f3c44146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 >> 172.21.15.102:0/2357278775 conn(0x7f3c440700d0 msgr2=0x7f3c44071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 shutdown_connections 2024-03-18T16:33:18.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:18.040+0000 7f3c4c200640 1 -- 172.21.15.102:0/2357278775 wait complete. 2024-03-18T16:33:18.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:18.670 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:19.671 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:19.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.849+0000 7ff311c00640 1 Processor -- start 2024-03-18T16:33:19.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.849+0000 7ff311c00640 1 -- start start 2024-03-18T16:33:19.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.850+0000 7ff311c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff30c141650 0x7ff30c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:19.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.850+0000 7ff311c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c0c4fa0 0x7ff30c0c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:19.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.850+0000 7ff311c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff30c0c72d0 con 0x7ff30c15c3a0 2024-03-18T16:33:19.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.850+0000 7ff311c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff30c159370 con 0x7ff30c141650 2024-03-18T16:33:19.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.850+0000 7ff311c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff30c158ad0 con 0x7ff30c0c4fa0 2024-03-18T16:33:19.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.851+0000 7ff311200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c0c4fa0 0x7ff30c0c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:19.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.851+0000 7ff30be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff30c141650 0x7ff30c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:19.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.852+0000 7ff311200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c0c4fa0 0x7ff30c0c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:40096/0 (socket says 172.21.15.102:40096) 2024-03-18T16:33:19.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.852+0000 7ff311200640 1 -- 172.21.15.102:0/2640807354 learned_addr learned my addr 172.21.15.102:0/2640807354 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.854+0000 7ff311200640 1 -- 172.21.15.102:0/2640807354 >> v2:172.21.15.102:3300/0 conn(0x7ff30c141650 msgr2=0x7ff30c158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.854+0000 7ff311200640 1 --2- 172.21.15.102:0/2640807354 >> v2:172.21.15.102:3300/0 conn(0x7ff30c141650 0x7ff30c158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.854+0000 7ff311200640 1 -- 172.21.15.102:0/2640807354 >> v1:172.21.15.102:6789/0 conn(0x7ff30c15c3a0 legacy=0x7ff30c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:19.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.854+0000 7ff311200640 1 -- 172.21.15.102:0/2640807354 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30c158db0 con 0x7ff30c0c4fa0 2024-03-18T16:33:19.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.857+0000 7ff311200640 1 --2- 172.21.15.102:0/2640807354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c0c4fa0 0x7ff30c0c5430 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff304009e50 tx=0x7ff30402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c3e55fd7006e895 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:19.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.859+0000 7ff30aa00640 1 -- 172.21.15.102:0/2640807354 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff30402cb00 con 0x7ff30c0c4fa0 2024-03-18T16:33:19.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.860+0000 7ff30aa00640 1 -- 172.21.15.102:0/2640807354 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff30402cca0 con 0x7ff30c0c4fa0 2024-03-18T16:33:19.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.861+0000 7ff30aa00640 1 -- 172.21.15.102:0/2640807354 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304034610 con 0x7ff30c0c4fa0 2024-03-18T16:33:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.861+0000 7ff311c00640 1 -- 172.21.15.102:0/2640807354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c0c4fa0 msgr2=0x7ff30c0c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.861+0000 7ff311c00640 1 --2- 172.21.15.102:0/2640807354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c0c4fa0 0x7ff30c0c5430 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff304009e50 tx=0x7ff30402bf60 comp rx=0 tx=0).stop 2024-03-18T16:33:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.861+0000 7ff311c00640 1 -- 172.21.15.102:0/2640807354 shutdown_connections 2024-03-18T16:33:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.861+0000 7ff311c00640 1 --2- 172.21.15.102:0/2640807354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c0c4fa0 0x7ff30c0c5430 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.861+0000 7ff311c00640 1 --2- 172.21.15.102:0/2640807354 >> v2:172.21.15.102:3300/0 conn(0x7ff30c141650 0x7ff30c158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.861+0000 7ff311c00640 1 -- 172.21.15.102:0/2640807354 >> 172.21.15.102:0/2640807354 conn(0x7ff30c0cb890 msgr2=0x7ff30c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.861+0000 7ff311c00640 1 -- 172.21.15.102:0/2640807354 shutdown_connections 2024-03-18T16:33:19.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.862+0000 7ff311c00640 1 -- 172.21.15.102:0/2640807354 wait complete. 2024-03-18T16:33:19.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.864+0000 7ff311c00640 1 Processor -- start 2024-03-18T16:33:19.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.864+0000 7ff311c00640 1 -- start start 2024-03-18T16:33:19.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.866+0000 7ff311c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff30c0c4fa0 0x7ff30c13e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:19.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.866+0000 7ff311c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c141650 0x7ff30c1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:19.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.866+0000 7ff311c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c15c3a0 0x7ff30c154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:19.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.866+0000 7ff311c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c0c66a0 con 0x7ff30c0c4fa0 2024-03-18T16:33:19.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.866+0000 7ff311c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff30c0c63c0 con 0x7ff30c15c3a0 2024-03-18T16:33:19.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.866+0000 7ff311c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff30c15c7c0 con 0x7ff30c141650 2024-03-18T16:33:19.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.867+0000 7ff30b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c141650 0x7ff30c1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:19.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.867+0000 7ff30be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff30c0c4fa0 0x7ff30c13e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:19.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.867+0000 7ff311200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c15c3a0 0x7ff30c154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:19.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.867+0000 7ff30b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c141650 0x7ff30c1541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:40112/0 (socket says 172.21.15.102:40112) 2024-03-18T16:33:19.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.867+0000 7ff30b400640 1 -- 172.21.15.102:0/2338535978 learned_addr learned my addr 172.21.15.102:0/2338535978 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:19.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.871+0000 7ff311200640 1 -- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c141650 msgr2=0x7ff30c1541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:19.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.871+0000 7ff311200640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c141650 0x7ff30c1541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:19.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.871+0000 7ff311200640 1 -- 172.21.15.102:0/2338535978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff30c0c4fa0 msgr2=0x7ff30c13e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:19.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.871+0000 7ff311200640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff30c0c4fa0 0x7ff30c13e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:19.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.871+0000 7ff311200640 1 -- 172.21.15.102:0/2338535978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30c0caf50 con 0x7ff30c15c3a0 2024-03-18T16:33:19.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.871+0000 7ff30b400640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c141650 0x7ff30c1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:33:19.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.873+0000 7ff311200640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c15c3a0 0x7ff30c154720 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff304032000 tx=0x7ff304004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:19.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.877+0000 7ff303e00640 1 -- 172.21.15.102:0/2338535978 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304043020 con 0x7ff30c15c3a0 2024-03-18T16:33:19.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.877+0000 7ff303e00640 1 -- 172.21.15.102:0/2338535978 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff30402ce50 con 0x7ff30c15c3a0 2024-03-18T16:33:19.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.877+0000 7ff303e00640 1 -- 172.21.15.102:0/2338535978 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff304009510 con 0x7ff30c15c3a0 2024-03-18T16:33:19.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.877+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff30c0cb230 con 0x7ff30c15c3a0 2024-03-18T16:33:19.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.877+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff30c153cc0 con 0x7ff30c15c3a0 2024-03-18T16:33:19.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.878+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff30c074fd0 con 0x7ff30c15c3a0 2024-03-18T16:33:19.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.897+0000 7ff303e00640 1 -- 172.21.15.102:0/2338535978 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff304046020 con 0x7ff30c15c3a0 2024-03-18T16:33:19.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.898+0000 7ff303e00640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ec06bbd0 0x7ff2ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:19.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.899+0000 7ff30be00640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ec06bbd0 0x7ff2ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:19.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.899+0000 7ff303e00640 1 -- 172.21.15.102:0/2338535978 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff304079770 con 0x7ff30c15c3a0 2024-03-18T16:33:19.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:19.899+0000 7ff30be00640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ec06bbd0 0x7ff2ec06e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009aa0 tx=0x7ff2fc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:20.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.146+0000 7ff303e00640 1 -- 172.21.15.102:0/2338535978 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff30404d030 con 0x7ff30c15c3a0 2024-03-18T16:33:20.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.388+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff30c153e90 con 0x7ff30c15c3a0 2024-03-18T16:33:20.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.400+0000 7ff303e00640 1 -- 172.21.15.102:0/2338535978 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff30407b6d0 con 0x7ff30c15c3a0 2024-03-18T16:33:20.402 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:20.403 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ec06bbd0 msgr2=0x7ff2ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ec06bbd0 0x7ff2ec06e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc009aa0 tx=0x7ff2fc009190 comp rx=0 tx=0).stop 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c15c3a0 msgr2=0x7ff30c154720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c15c3a0 0x7ff30c154720 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff304032000 tx=0x7ff304004a40 comp rx=0 tx=0).stop 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 shutdown_connections 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ec06bbd0 0x7ff2ec06e090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff30c15c3a0 0x7ff30c154720 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff30c141650 0x7ff30c1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 --2- 172.21.15.102:0/2338535978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff30c0c4fa0 0x7ff30c13e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 >> 172.21.15.102:0/2338535978 conn(0x7ff30c0cb890 msgr2=0x7ff30c0c1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:20.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 shutdown_connections 2024-03-18T16:33:20.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:20.403+0000 7ff311c00640 1 -- 172.21.15.102:0/2338535978 wait complete. 2024-03-18T16:33:21.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:21.024 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:22.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:22.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.198+0000 7fdf1a600640 1 Processor -- start 2024-03-18T16:33:22.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.198+0000 7fdf1a600640 1 -- start start 2024-03-18T16:33:22.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.200+0000 7fdf1a600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdf140d0250 0x7fdf1415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:22.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.200+0000 7fdf1a600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdf140cc2f0 0x7fdf140cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:22.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.200+0000 7fdf1a600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf140ccc80 con 0x7fdf140cfe70 2024-03-18T16:33:22.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.200+0000 7fdf1a600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fdf140cabd0 con 0x7fdf140d0250 2024-03-18T16:33:22.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.200+0000 7fdf1a600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdf140caea0 con 0x7fdf140cc2f0 2024-03-18T16:33:22.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.200+0000 7fdf19c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdf140cc2f0 0x7fdf140cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:22.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.201+0000 7fdf19c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdf140cc2f0 0x7fdf140cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54156/0 (socket says 172.21.15.102:54156) 2024-03-18T16:33:22.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.201+0000 7fdf19c00640 1 -- 172.21.15.102:0/1679013002 learned_addr learned my addr 172.21.15.102:0/1679013002 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:22.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.201+0000 7fdf13e00640 1 --2- 172.21.15.102:0/1679013002 >> v2:172.21.15.102:3300/0 conn(0x7fdf140d0250 0x7fdf1415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:22.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.203+0000 7fdf19c00640 1 -- 172.21.15.102:0/1679013002 >> v2:172.21.15.102:3300/0 conn(0x7fdf140d0250 msgr2=0x7fdf1415ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:22.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.203+0000 7fdf19c00640 1 --2- 172.21.15.102:0/1679013002 >> v2:172.21.15.102:3300/0 conn(0x7fdf140d0250 0x7fdf1415ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:22.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.203+0000 7fdf19c00640 1 -- 172.21.15.102:0/1679013002 >> v1:172.21.15.154:6789/0 conn(0x7fdf140cfe70 legacy=0x7fdf140c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:22.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.203+0000 7fdf19c00640 1 -- 172.21.15.102:0/1679013002 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf140cb180 con 0x7fdf140cc2f0 2024-03-18T16:33:22.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.207+0000 7fdf19c00640 1 --2- 172.21.15.102:0/1679013002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140cc2f0 0x7fdf140cc740 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdf140ca160 tx=0x7fdf0c02bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8d49f863b1bc1c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:22.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.211+0000 7fdf13400640 1 -- 172.21.15.102:0/1679013002 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c02cb20 con 0x7fdf140cc2f0 2024-03-18T16:33:22.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.211+0000 7fdf13400640 1 -- 172.21.15.102:0/1679013002 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf0c02ccc0 con 0x7fdf140cc2f0 2024-03-18T16:33:22.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.211+0000 7fdf13400640 1 -- 172.21.15.102:0/1679013002 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c034610 con 0x7fdf140cc2f0 2024-03-18T16:33:22.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.212+0000 7fdf1a600640 1 -- 172.21.15.102:0/1679013002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140cc2f0 msgr2=0x7fdf140cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:22.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.212+0000 7fdf1a600640 1 --2- 172.21.15.102:0/1679013002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140cc2f0 0x7fdf140cc740 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdf140ca160 tx=0x7fdf0c02bf80 comp rx=0 tx=0).stop 2024-03-18T16:33:22.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.212+0000 7fdf1a600640 1 -- 172.21.15.102:0/1679013002 shutdown_connections 2024-03-18T16:33:22.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.212+0000 7fdf1a600640 1 --2- 172.21.15.102:0/1679013002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140cc2f0 0x7fdf140cc740 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:22.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.212+0000 7fdf1a600640 1 --2- 172.21.15.102:0/1679013002 >> v2:172.21.15.102:3300/0 conn(0x7fdf140d0250 0x7fdf1415ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:22.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.212+0000 7fdf1a600640 1 -- 172.21.15.102:0/1679013002 >> 172.21.15.102:0/1679013002 conn(0x7fdf1408c3e0 msgr2=0x7fdf1406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:22.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.212+0000 7fdf1a600640 1 -- 172.21.15.102:0/1679013002 shutdown_connections 2024-03-18T16:33:22.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.212+0000 7fdf1a600640 1 -- 172.21.15.102:0/1679013002 wait complete. 2024-03-18T16:33:22.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.214+0000 7fdf1a600640 1 Processor -- start 2024-03-18T16:33:22.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.214+0000 7fdf1a600640 1 -- start start 2024-03-18T16:33:22.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.215+0000 7fdf1a600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140cc2f0 0x7fdf141f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:22.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.215+0000 7fdf1a600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf140cfe70 0x7fdf141efc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:22.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.215+0000 7fdf1a600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140d0250 0x7fdf14157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:22.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.215+0000 7fdf1a600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdf140cba20 con 0x7fdf140cfe70 2024-03-18T16:33:22.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.215+0000 7fdf1a600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf140cb740 con 0x7fdf140d0250 2024-03-18T16:33:22.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.215+0000 7fdf1a600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdf140cbd00 con 0x7fdf140cc2f0 2024-03-18T16:33:22.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.216+0000 7fdf19200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140cc2f0 0x7fdf141f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:22.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.216+0000 7fdf19c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140d0250 0x7fdf14157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:22.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.216+0000 7fdf13e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf140cfe70 0x7fdf141efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:22.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.216+0000 7fdf19c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140d0250 0x7fdf14157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54158/0 (socket says 172.21.15.102:54158) 2024-03-18T16:33:22.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.216+0000 7fdf19200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140cc2f0 0x7fdf141f3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:45776/0 (socket says 172.21.15.102:45776) 2024-03-18T16:33:22.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.216+0000 7fdf19c00640 1 -- 172.21.15.102:0/3939879429 learned_addr learned my addr 172.21.15.102:0/3939879429 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:22.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.219+0000 7fdf19200640 1 -- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140d0250 msgr2=0x7fdf14157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:22.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.219+0000 7fdf19200640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140d0250 0x7fdf14157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:22.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.220+0000 7fdf19200640 1 -- 172.21.15.102:0/3939879429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf140cfe70 msgr2=0x7fdf141efc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:22.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.220+0000 7fdf19200640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf140cfe70 0x7fdf141efc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:22.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.220+0000 7fdf19200640 1 -- 172.21.15.102:0/3939879429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf14155cc0 con 0x7fdf140cc2f0 2024-03-18T16:33:22.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.222+0000 7fdf19200640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140cc2f0 0x7fdf141f3cd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800c440 tx=0x7fdf0800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:22.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.226+0000 7fdf10c00640 1 -- 172.21.15.102:0/3939879429 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf08011790 con 0x7fdf140cc2f0 2024-03-18T16:33:22.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.226+0000 7fdf10c00640 1 -- 172.21.15.102:0/3939879429 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf08017070 con 0x7fdf140cc2f0 2024-03-18T16:33:22.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.226+0000 7fdf10c00640 1 -- 172.21.15.102:0/3939879429 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf08011b70 con 0x7fdf140cc2f0 2024-03-18T16:33:22.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.226+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf14155fa0 con 0x7fdf140cc2f0 2024-03-18T16:33:22.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.226+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf141f34d0 con 0x7fdf140cc2f0 2024-03-18T16:33:22.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.228+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf14074fd0 con 0x7fdf140cc2f0 2024-03-18T16:33:22.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.483+0000 7fdf10c00640 1 -- 172.21.15.102:0/3939879429 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdf08011d10 con 0x7fdf140cc2f0 2024-03-18T16:33:22.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.485+0000 7fdf10c00640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdef406bc20 0x7fdef406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:22.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.485+0000 7fdf10c00640 1 -- 172.21.15.102:0/3939879429 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fdf0808b570 con 0x7fdf140cc2f0 2024-03-18T16:33:22.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.485+0000 7fdf13e00640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdef406bc20 0x7fdef406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:22.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.486+0000 7fdf13e00640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdef406bc20 0x7fdef406e0e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdf141e73f0 tx=0x7fdf00009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:22.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.495+0000 7fdf10c00640 1 -- 172.21.15.102:0/3939879429 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdf08059690 con 0x7fdf140cc2f0 2024-03-18T16:33:22.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.755+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf140c6830 con 0x7fdf140cc2f0 2024-03-18T16:33:22.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.768+0000 7fdf10c00640 1 -- 172.21.15.102:0/3939879429 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fdf0805d730 con 0x7fdf140cc2f0 2024-03-18T16:33:22.770 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:22.770 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:22.770 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:22.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdef406bc20 msgr2=0x7fdef406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:22.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdef406bc20 0x7fdef406e0e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdf141e73f0 tx=0x7fdf00009190 comp rx=0 tx=0).stop 2024-03-18T16:33:22.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140cc2f0 msgr2=0x7fdf141f3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:22.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140cc2f0 0x7fdf141f3cd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800c440 tx=0x7fdf0800c900 comp rx=0 tx=0).stop 2024-03-18T16:33:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 shutdown_connections 2024-03-18T16:33:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdef406bc20 0x7fdef406e0e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf140d0250 0x7fdf14157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf140cfe70 0x7fdf141efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 --2- 172.21.15.102:0/3939879429 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf140cc2f0 0x7fdf141f3cd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.770+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 >> 172.21.15.102:0/3939879429 conn(0x7fdf1408c3e0 msgr2=0x7fdf14159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.771+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 shutdown_connections 2024-03-18T16:33:22.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:22.771+0000 7fdf1a600640 1 -- 172.21.15.102:0/3939879429 wait complete. 2024-03-18T16:33:23.399 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:23.399 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:23.667 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:33:23.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:33:24.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:24.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.580+0000 7fb8be200640 1 Processor -- start 2024-03-18T16:33:24.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.580+0000 7fb8be200640 1 -- start start 2024-03-18T16:33:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.581+0000 7fb8be200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb8b80cc290 0x7fb8b815ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.581+0000 7fb8be200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc670 0x7fb8b80c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.581+0000 7fb8be200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b806b3f0 con 0x7fb8b806c750 2024-03-18T16:33:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.581+0000 7fb8be200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b80c8880 con 0x7fb8b80cc290 2024-03-18T16:33:24.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.581+0000 7fb8be200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b806b6b0 con 0x7fb8b80cc670 2024-03-18T16:33:24.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.582+0000 7fb8b7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc670 0x7fb8b80c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:24.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.582+0000 7fb8bd800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb8b806c750 0x7fb8b8153fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58056/0 (socket says 172.21.15.102:58056) 2024-03-18T16:33:24.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.582+0000 7fb8bd800640 1 -- 172.21.15.102:0/761561921 learned_addr learned my addr 172.21.15.102:0/761561921 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:33:24.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.582+0000 7fb8bce00640 1 --2- 172.21.15.102:0/761561921 >> v2:172.21.15.102:3300/0 conn(0x7fb8b80cc290 0x7fb8b815ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:24.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.585+0000 7fb8b7e00640 1 -- 172.21.15.102:0/761561921 >> v2:172.21.15.102:3300/0 conn(0x7fb8b80cc290 msgr2=0x7fb8b815ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:24.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.585+0000 7fb8b7e00640 1 --2- 172.21.15.102:0/761561921 >> v2:172.21.15.102:3300/0 conn(0x7fb8b80cc290 0x7fb8b815ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:24.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.585+0000 7fb8b7e00640 1 -- 172.21.15.102:0/761561921 >> v1:172.21.15.154:6789/0 conn(0x7fb8b806c750 legacy=0x7fb8b8153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:24.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.585+0000 7fb8b7e00640 1 -- 172.21.15.102:0/761561921 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b80c8aa0 con 0x7fb8b80cc670 2024-03-18T16:33:24.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.587+0000 7fb8b7e00640 1 --2- 172.21.15.102:0/761561921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc670 0x7fb8b80c92a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb8a40097a0 tx=0x7fb8a402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f974fd42f96e5062 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:24.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.589+0000 7fb8b7400640 1 -- 172.21.15.102:0/761561921 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4035020 con 0x7fb8b80cc670 2024-03-18T16:33:24.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.591+0000 7fb8b7400640 1 -- 172.21.15.102:0/761561921 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a4031810 con 0x7fb8b80cc670 2024-03-18T16:33:24.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.591+0000 7fb8b7400640 1 -- 172.21.15.102:0/761561921 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4031be0 con 0x7fb8b80cc670 2024-03-18T16:33:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.592+0000 7fb8be200640 1 -- 172.21.15.102:0/761561921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc670 msgr2=0x7fb8b80c92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.592+0000 7fb8be200640 1 --2- 172.21.15.102:0/761561921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc670 0x7fb8b80c92a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb8a40097a0 tx=0x7fb8a402b8b0 comp rx=0 tx=0).stop 2024-03-18T16:33:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.592+0000 7fb8be200640 1 -- 172.21.15.102:0/761561921 shutdown_connections 2024-03-18T16:33:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.592+0000 7fb8be200640 1 --2- 172.21.15.102:0/761561921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc670 0x7fb8b80c92a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.592+0000 7fb8be200640 1 --2- 172.21.15.102:0/761561921 >> v2:172.21.15.102:3300/0 conn(0x7fb8b80cc290 0x7fb8b815ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.592+0000 7fb8be200640 1 -- 172.21.15.102:0/761561921 >> 172.21.15.102:0/761561921 conn(0x7fb8b806e260 msgr2=0x7fb8b8071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.592+0000 7fb8be200640 1 -- 172.21.15.102:0/761561921 shutdown_connections 2024-03-18T16:33:24.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.592+0000 7fb8be200640 1 -- 172.21.15.102:0/761561921 wait complete. 2024-03-18T16:33:24.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.595+0000 7fb8be200640 1 Processor -- start 2024-03-18T16:33:24.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.595+0000 7fb8be200640 1 -- start start 2024-03-18T16:33:24.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.596+0000 7fb8be200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b806c750 0x7fb8b81f04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:24.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.597+0000 7fb8be200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc290 0x7fb8b8144340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:24.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.597+0000 7fb8be200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b80cc670 0x7fb8b80ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:24.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.597+0000 7fb8be200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b80c97e0 con 0x7fb8b806c750 2024-03-18T16:33:24.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.597+0000 7fb8be200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b80c7490 con 0x7fb8b80cc670 2024-03-18T16:33:24.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.597+0000 7fb8be200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b80c9960 con 0x7fb8b80cc290 2024-03-18T16:33:24.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.597+0000 7fb8b7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc290 0x7fb8b8144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:24.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.597+0000 7fb8bd800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b80cc670 0x7fb8b80ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:24.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.598+0000 7fb8bce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b806c750 0x7fb8b81f04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:24.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.598+0000 7fb8b7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc290 0x7fb8b8144340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:45798/0 (socket says 172.21.15.102:45798) 2024-03-18T16:33:24.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.598+0000 7fb8b7e00640 1 -- 172.21.15.102:0/3958887352 learned_addr learned my addr 172.21.15.102:0/3958887352 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:24.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.601+0000 7fb8b7e00640 1 -- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b80cc670 msgr2=0x7fb8b80ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:24.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.601+0000 7fb8b7e00640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b80cc670 0x7fb8b80ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:24.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.601+0000 7fb8b7e00640 1 -- 172.21.15.102:0/3958887352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b806c750 msgr2=0x7fb8b81f04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:24.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.601+0000 7fb8b7e00640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b806c750 0x7fb8b81f04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:24.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.601+0000 7fb8b7e00640 1 -- 172.21.15.102:0/3958887352 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b81e71a0 con 0x7fb8b80cc290 2024-03-18T16:33:24.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.604+0000 7fb8b7e00640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc290 0x7fb8b8144340 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb8a402bec0 tx=0x7fb8a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:24.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.608+0000 7fb8b4c00640 1 -- 172.21.15.102:0/3958887352 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4041070 con 0x7fb8b80cc290 2024-03-18T16:33:24.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.608+0000 7fb8b4c00640 1 -- 172.21.15.102:0/3958887352 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a403ad00 con 0x7fb8b80cc290 2024-03-18T16:33:24.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.608+0000 7fb8b4c00640 1 -- 172.21.15.102:0/3958887352 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4035040 con 0x7fb8b80cc290 2024-03-18T16:33:24.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.608+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b8144880 con 0x7fb8b80cc290 2024-03-18T16:33:24.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.608+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b81f0d60 con 0x7fb8b80cc290 2024-03-18T16:33:24.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.609+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b8074fd0 con 0x7fb8b80cc290 2024-03-18T16:33:24.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.665+0000 7fb8b4c00640 1 -- 172.21.15.102:0/3958887352 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb8a404c060 con 0x7fb8b80cc290 2024-03-18T16:33:24.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.667+0000 7fb8b4c00640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb89806bc20 0x7fb89806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:24.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.668+0000 7fb8bce00640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb89806bc20 0x7fb89806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:24.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.668+0000 7fb8bce00640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb89806bc20 0x7fb89806e0e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009aa0 tx=0x7fb8ac009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:24.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.918+0000 7fb8b4c00640 1 -- 172.21.15.102:0/3958887352 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb8a4078ee0 con 0x7fb8b80cc290 2024-03-18T16:33:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:24.937+0000 7fb8b4c00640 1 -- 172.21.15.102:0/3958887352 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb8a407ae40 con 0x7fb8b80cc290 2024-03-18T16:33:25.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.179+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8b806ba90 con 0x7fb8b80cc290 2024-03-18T16:33:25.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.199+0000 7fb8b4c00640 1 -- 172.21.15.102:0/3958887352 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb8a407eee0 con 0x7fb8b80cc290 2024-03-18T16:33:25.201 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:25.201 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:25.201 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb89806bc20 msgr2=0x7fb89806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb89806bc20 0x7fb89806e0e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009aa0 tx=0x7fb8ac009190 comp rx=0 tx=0).stop 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc290 msgr2=0x7fb8b8144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc290 0x7fb8b8144340 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb8a402bec0 tx=0x7fb8a40046c0 comp rx=0 tx=0).stop 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 shutdown_connections 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb89806bc20 0x7fb89806e0e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8b80cc670 0x7fb8b80ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb8b80cc290 0x7fb8b8144340 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 --2- 172.21.15.102:0/3958887352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b806c750 0x7fb8b81f04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 >> 172.21.15.102:0/3958887352 conn(0x7fb8b806e260 msgr2=0x7fb8b8152400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 shutdown_connections 2024-03-18T16:33:25.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:25.202+0000 7fb8be200640 1 -- 172.21.15.102:0/3958887352 wait complete. 2024-03-18T16:33:25.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:25.826 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:26.827 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:27.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.009+0000 7f5de8200640 1 Processor -- start 2024-03-18T16:33:27.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.009+0000 7f5de8200640 1 -- start start 2024-03-18T16:33:27.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.011+0000 7f5de8200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5de00689a0 0x7f5de00c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:27.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.011+0000 7f5de8200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5de0150b00 con 0x7f5de00cca70 2024-03-18T16:33:27.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.011+0000 7f5de8200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5de0159ad0 con 0x7f5de0071d80 2024-03-18T16:33:27.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.011+0000 7f5de8200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5de015a6e0 con 0x7f5de00689a0 2024-03-18T16:33:27.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.012+0000 7f5de5800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5de00689a0 0x7f5de00c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:27.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.012+0000 7f5de6200640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f5de0071d80 0x7f5de006dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58072/0 (socket says 172.21.15.102:58072) 2024-03-18T16:33:27.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.012+0000 7f5de6200640 1 -- 172.21.15.102:0/1306353309 learned_addr learned my addr 172.21.15.102:0/1306353309 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:33:27.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.015+0000 7f5ddfe00640 1 -- 172.21.15.102:0/1306353309 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3965749077 0 0) 0x7f5de0159ad0 con 0x7f5de0071d80 2024-03-18T16:33:27.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.015+0000 7f5ddfe00640 1 -- 172.21.15.102:0/1306353309 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5dc8003520 con 0x7f5de0071d80 2024-03-18T16:33:27.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.017+0000 7f5de5800640 1 -- 172.21.15.102:0/1306353309 >> v1:172.21.15.154:6789/0 conn(0x7f5de0071d80 legacy=0x7f5de006dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:27.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.017+0000 7f5de5800640 1 -- 172.21.15.102:0/1306353309 >> v1:172.21.15.102:6789/0 conn(0x7f5de00cca70 legacy=0x7f5de00c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:27.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.017+0000 7f5de5800640 1 -- 172.21.15.102:0/1306353309 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de0159db0 con 0x7f5de00689a0 2024-03-18T16:33:27.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.024+0000 7f5de5800640 1 --2- 172.21.15.102:0/1306353309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de00689a0 0x7f5de00c69d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40097a0 tx=0x7f5dd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31231341a6fecb22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:27.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.029+0000 7f5ddfe00640 1 -- 172.21.15.102:0/1306353309 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4035020 con 0x7f5de00689a0 2024-03-18T16:33:27.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.029+0000 7f5ddfe00640 1 -- 172.21.15.102:0/1306353309 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd4031830 con 0x7f5de00689a0 2024-03-18T16:33:27.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.029+0000 7f5ddfe00640 1 -- 172.21.15.102:0/1306353309 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4031c00 con 0x7f5de00689a0 2024-03-18T16:33:27.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.030+0000 7f5de8200640 1 -- 172.21.15.102:0/1306353309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de00689a0 msgr2=0x7f5de00c69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:27.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.030+0000 7f5de8200640 1 --2- 172.21.15.102:0/1306353309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de00689a0 0x7f5de00c69d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40097a0 tx=0x7f5dd402b8d0 comp rx=0 tx=0).stop 2024-03-18T16:33:27.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.030+0000 7f5de8200640 1 -- 172.21.15.102:0/1306353309 shutdown_connections 2024-03-18T16:33:27.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.030+0000 7f5de8200640 1 --2- 172.21.15.102:0/1306353309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de00689a0 0x7f5de00c69d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:27.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.030+0000 7f5de8200640 1 -- 172.21.15.102:0/1306353309 >> 172.21.15.102:0/1306353309 conn(0x7f5de006fec0 msgr2=0x7f5de00cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:27.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.031+0000 7f5de8200640 1 -- 172.21.15.102:0/1306353309 shutdown_connections 2024-03-18T16:33:27.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.031+0000 7f5de8200640 1 -- 172.21.15.102:0/1306353309 wait complete. 2024-03-18T16:33:27.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.033+0000 7f5de8200640 1 Processor -- start 2024-03-18T16:33:27.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.034+0000 7f5de8200640 1 -- start start 2024-03-18T16:33:27.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.035+0000 7f5de8200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de00689a0 0x7f5de01532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:27.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.035+0000 7f5de8200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de0071d80 0x7f5de0150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:27.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.035+0000 7f5de8200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de00cca70 0x7f5de01f04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:27.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.035+0000 7f5de8200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5de006e2c0 con 0x7f5de0071d80 2024-03-18T16:33:27.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.035+0000 7f5de8200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5de015a370 con 0x7f5de00689a0 2024-03-18T16:33:27.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.035+0000 7f5de8200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5de006e440 con 0x7f5de00cca70 2024-03-18T16:33:27.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.036+0000 7f5de6200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de00cca70 0x7f5de01f04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:27.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.036+0000 7f5de5800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de00689a0 0x7f5de01532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:27.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.036+0000 7f5de4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de0071d80 0x7f5de0150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:27.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.036+0000 7f5de5800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de00689a0 0x7f5de01532c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54168/0 (socket says 172.21.15.102:54168) 2024-03-18T16:33:27.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.036+0000 7f5de5800640 1 -- 172.21.15.102:0/840610094 learned_addr learned my addr 172.21.15.102:0/840610094 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:27.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.040+0000 7f5de6200640 1 -- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de00689a0 msgr2=0x7f5de01532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:27.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.040+0000 7f5de6200640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de00689a0 0x7f5de01532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:27.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.040+0000 7f5de6200640 1 -- 172.21.15.102:0/840610094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de0071d80 msgr2=0x7f5de0150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:27.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.040+0000 7f5de6200640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de0071d80 0x7f5de0150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:27.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.040+0000 7f5de6200640 1 -- 172.21.15.102:0/840610094 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de0157000 con 0x7f5de00cca70 2024-03-18T16:33:27.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.043+0000 7f5de6200640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de00cca70 0x7f5de01f04e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800cab0 tx=0x7f5dd800cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:27.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.047+0000 7f5ddd600640 1 -- 172.21.15.102:0/840610094 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8017070 con 0x7f5de00cca70 2024-03-18T16:33:27.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.047+0000 7f5ddd600640 1 -- 172.21.15.102:0/840610094 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd800ac30 con 0x7f5de00cca70 2024-03-18T16:33:27.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.047+0000 7f5ddd600640 1 -- 172.21.15.102:0/840610094 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8012600 con 0x7f5de00cca70 2024-03-18T16:33:27.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.047+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de0158920 con 0x7f5de00cca70 2024-03-18T16:33:27.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.047+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de0158ee0 con 0x7f5de00cca70 2024-03-18T16:33:27.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.049+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de01590e0 con 0x7f5de00cca70 2024-03-18T16:33:27.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.072+0000 7f5ddd600640 1 -- 172.21.15.102:0/840610094 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5dd801a070 con 0x7f5de00cca70 2024-03-18T16:33:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.073+0000 7f5ddd600640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5dc006bc50 0x7f5dc006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:27.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.073+0000 7f5de5800640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5dc006bc50 0x7f5dc006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:27.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.074+0000 7f5de5800640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5dc006bc50 0x7f5dc006e110 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009560 tx=0x7f5dd4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:27.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.315+0000 7f5ddd600640 1 -- 172.21.15.102:0/840610094 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5dd80570a0 con 0x7f5de00cca70 2024-03-18T16:33:27.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.326+0000 7f5ddd600640 1 -- 172.21.15.102:0/840610094 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5de01590e0 con 0x7f5de00cca70 2024-03-18T16:33:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.626+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5de0074fd0 con 0x7f5de00cca70 2024-03-18T16:33:27.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.638+0000 7f5ddd600640 1 -- 172.21.15.102:0/840610094 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f5dd800ae20 con 0x7f5de00cca70 2024-03-18T16:33:27.640 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:27.640 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:27.641 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.641+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5dc006bc50 msgr2=0x7f5dc006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.641+0000 7f5de8200640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5dc006bc50 0x7f5dc006e110 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009560 tx=0x7f5dd4008030 comp rx=0 tx=0).stop 2024-03-18T16:33:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.641+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de00cca70 msgr2=0x7f5de01f04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.641+0000 7f5de8200640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de00cca70 0x7f5de01f04e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800cab0 tx=0x7f5dd800cf70 comp rx=0 tx=0).stop 2024-03-18T16:33:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.642+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 shutdown_connections 2024-03-18T16:33:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.642+0000 7f5de8200640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5dc006bc50 0x7f5dc006e110 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.642+0000 7f5de8200640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5de00cca70 0x7f5de01f04e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:27.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.642+0000 7f5de8200640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5de0071d80 0x7f5de0150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:27.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.642+0000 7f5de8200640 1 --2- 172.21.15.102:0/840610094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de00689a0 0x7f5de01532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:27.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.642+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 >> 172.21.15.102:0/840610094 conn(0x7f5de006fec0 msgr2=0x7f5de0072590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:27.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.642+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 shutdown_connections 2024-03-18T16:33:27.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:27.642+0000 7f5de8200640 1 -- 172.21.15.102:0/840610094 wait complete. 2024-03-18T16:33:28.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:28.263 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:29.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:29.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.438+0000 7faeb9000640 1 Processor -- start 2024-03-18T16:33:29.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.438+0000 7faeb9000640 1 -- start start 2024-03-18T16:33:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.440+0000 7faeb9000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faeb40c98e0 0x7faeb40c9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.440+0000 7faeb9000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb40cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.440+0000 7faeb9000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faeb40c87f0 con 0x7faeb4152390 2024-03-18T16:33:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.440+0000 7faeb9000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faeb40c85a0 con 0x7faeb40c98e0 2024-03-18T16:33:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.440+0000 7faeb9000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faeb40c8ad0 con 0x7faeb415bf10 2024-03-18T16:33:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.441+0000 7faeb3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb40cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.441+0000 7faeb2a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faeb40c98e0 0x7faeb40c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.441+0000 7faeb3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb40cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:45822/0 (socket says 172.21.15.102:45822) 2024-03-18T16:33:29.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.441+0000 7faeb3e00640 1 -- 172.21.15.102:0/3444207382 learned_addr learned my addr 172.21.15.102:0/3444207382 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:29.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.444+0000 7faeb3e00640 1 -- 172.21.15.102:0/3444207382 >> v2:172.21.15.154:3300/0 conn(0x7faeb40c98e0 msgr2=0x7faeb40c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:29.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.444+0000 7faeb3e00640 1 --2- 172.21.15.102:0/3444207382 >> v2:172.21.15.154:3300/0 conn(0x7faeb40c98e0 0x7faeb40c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:29.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.444+0000 7faeb3e00640 1 -- 172.21.15.102:0/3444207382 >> v1:172.21.15.154:6789/0 conn(0x7faeb4152390 legacy=0x7faeb40ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:29.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.444+0000 7faeb3e00640 1 -- 172.21.15.102:0/3444207382 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb40c9eb0 con 0x7faeb415bf10 2024-03-18T16:33:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.446+0000 7faeb3e00640 1 --2- 172.21.15.102:0/3444207382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb40cd880 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7faeac009e50 tx=0x7faeac02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c4ff27df833e74c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:29.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.450+0000 7faeb2000640 1 -- 172.21.15.102:0/3444207382 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac02cb90 con 0x7faeb415bf10 2024-03-18T16:33:29.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.450+0000 7faeb2000640 1 -- 172.21.15.102:0/3444207382 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeac02cd30 con 0x7faeb415bf10 2024-03-18T16:33:29.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.450+0000 7faeb2000640 1 -- 172.21.15.102:0/3444207382 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac034610 con 0x7faeb415bf10 2024-03-18T16:33:29.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.451+0000 7faeb9000640 1 -- 172.21.15.102:0/3444207382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 msgr2=0x7faeb40cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:29.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.451+0000 7faeb9000640 1 --2- 172.21.15.102:0/3444207382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb40cd880 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7faeac009e50 tx=0x7faeac02bff0 comp rx=0 tx=0).stop 2024-03-18T16:33:29.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.451+0000 7faeb9000640 1 -- 172.21.15.102:0/3444207382 shutdown_connections 2024-03-18T16:33:29.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.451+0000 7faeb9000640 1 --2- 172.21.15.102:0/3444207382 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb40cd880 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:29.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.451+0000 7faeb9000640 1 --2- 172.21.15.102:0/3444207382 >> v2:172.21.15.154:3300/0 conn(0x7faeb40c98e0 0x7faeb40c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:29.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.451+0000 7faeb9000640 1 -- 172.21.15.102:0/3444207382 >> 172.21.15.102:0/3444207382 conn(0x7faeb4066f80 msgr2=0x7faeb4067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:29.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.451+0000 7faeb9000640 1 -- 172.21.15.102:0/3444207382 shutdown_connections 2024-03-18T16:33:29.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.451+0000 7faeb9000640 1 -- 172.21.15.102:0/3444207382 wait complete. 2024-03-18T16:33:29.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.452+0000 7faeb9000640 1 Processor -- start 2024-03-18T16:33:29.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.453+0000 7faeb9000640 1 -- start start 2024-03-18T16:33:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.453+0000 7faeb9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb40c98e0 0x7faeb41552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.453+0000 7faeb9000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb4152390 0x7faeb41f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.453+0000 7faeb9000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb4155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.454+0000 7faeb9000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7faeb40cb0a0 con 0x7faeb40c98e0 2024-03-18T16:33:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.454+0000 7faeb9000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faeb40ca470 con 0x7faeb4152390 2024-03-18T16:33:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.454+0000 7faeb9000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7faeb40cb220 con 0x7faeb415bf10 2024-03-18T16:33:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.454+0000 7faeb2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb4152390 0x7faeb41f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.454+0000 7faeb3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb40c98e0 0x7faeb41552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.454+0000 7faeb3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb4155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.455+0000 7faeb2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb4152390 0x7faeb41f0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:54184/0 (socket says 172.21.15.102:54184) 2024-03-18T16:33:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.455+0000 7faeb2a00640 1 -- 172.21.15.102:0/2681223641 learned_addr learned my addr 172.21.15.102:0/2681223641 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.459+0000 7faeb3400640 1 -- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 msgr2=0x7faeb4155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.459+0000 7faeb3400640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb4155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.459+0000 7faeb3400640 1 -- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb4152390 msgr2=0x7faeb41f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.459+0000 7faeb3400640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb4152390 0x7faeb41f0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.459+0000 7faeb3400640 1 -- 172.21.15.102:0/2681223641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb40cd2e0 con 0x7faeb40c98e0 2024-03-18T16:33:29.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.462+0000 7faeb3400640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb40c98e0 0x7faeb41552c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faea400c3f0 tx=0x7faea400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:29.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.466+0000 7faeab400640 1 -- 172.21.15.102:0/2681223641 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea40052d0 con 0x7faeb40c98e0 2024-03-18T16:33:29.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.467+0000 7faeab400640 1 -- 172.21.15.102:0/2681223641 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faea4005490 con 0x7faeb40c98e0 2024-03-18T16:33:29.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.467+0000 7faeab400640 1 -- 172.21.15.102:0/2681223641 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea4007260 con 0x7faeb40c98e0 2024-03-18T16:33:29.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.467+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb40cda70 con 0x7faeb40c98e0 2024-03-18T16:33:29.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.467+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb40ce430 con 0x7faeb40c98e0 2024-03-18T16:33:29.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.468+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb4074fd0 con 0x7faeb40c98e0 2024-03-18T16:33:29.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.751+0000 7faeab400640 1 -- 172.21.15.102:0/2681223641 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faea4007400 con 0x7faeb40c98e0 2024-03-18T16:33:29.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.753+0000 7faeab400640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fae9406bc20 0x7fae9406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:29.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.753+0000 7faeb2a00640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fae9406bc20 0x7fae9406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:29.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.754+0000 7faeb2a00640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fae9406bc20 0x7fae9406e0e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faeb40c45d0 tx=0x7faea0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.777+0000 7faeab400640 1 -- 172.21.15.102:0/2681223641 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7faea4056bb0 con 0x7faeb40c98e0 2024-03-18T16:33:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:29.778+0000 7faeab400640 1 -- 172.21.15.102:0/2681223641 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faea4058b10 con 0x7faeb40c98e0 2024-03-18T16:33:30.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.039+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faeb40c9150 con 0x7faeb40c98e0 2024-03-18T16:33:30.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.060+0000 7faeab400640 1 -- 172.21.15.102:0/2681223641 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7faea405cbb0 con 0x7faeb40c98e0 2024-03-18T16:33:30.062 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:30.062 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:30.062 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:30.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.063+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fae9406bc20 msgr2=0x7fae9406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.063+0000 7faeb9000640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fae9406bc20 0x7fae9406e0e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7faeb40c45d0 tx=0x7faea0009190 comp rx=0 tx=0).stop 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.063+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb40c98e0 msgr2=0x7faeb41552c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.063+0000 7faeb9000640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb40c98e0 0x7faeb41552c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faea400c3f0 tx=0x7faea400c8b0 comp rx=0 tx=0).stop 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.064+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 shutdown_connections 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.064+0000 7faeb9000640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fae9406bc20 0x7fae9406e0e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.064+0000 7faeb9000640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7faeb415bf10 0x7faeb4155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.064+0000 7faeb9000640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb4152390 0x7faeb41f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.064+0000 7faeb9000640 1 --2- 172.21.15.102:0/2681223641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb40c98e0 0x7faeb41552c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.064+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 >> 172.21.15.102:0/2681223641 conn(0x7faeb4066f80 msgr2=0x7faeb40c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.064+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 shutdown_connections 2024-03-18T16:33:30.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:30.064+0000 7faeb9000640 1 -- 172.21.15.102:0/2681223641 wait complete. 2024-03-18T16:33:30.686 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:30.686 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:31.687 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.862+0000 7f6933c00640 1 Processor -- start 2024-03-18T16:33:31.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.862+0000 7f6933c00640 1 -- start start 2024-03-18T16:33:31.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.864+0000 7f6933c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f692c072700 0x7f692c158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:31.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.864+0000 7f6933c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c15b8e0 0x7f692c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:31.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.865+0000 7f6933c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f692c15c200 con 0x7f692c1524c0 2024-03-18T16:33:31.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.865+0000 7f6933c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f692c072b20 con 0x7f692c072700 2024-03-18T16:33:31.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.865+0000 7f6933c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f692c15c4e0 con 0x7f692c15b8e0 2024-03-18T16:33:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.865+0000 7f6931c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c15b8e0 0x7f692c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.866+0000 7f6931c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c15b8e0 0x7f692c0c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38258/0 (socket says 172.21.15.102:38258) 2024-03-18T16:33:31.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.866+0000 7f6931c00640 1 -- 172.21.15.102:0/1093367592 learned_addr learned my addr 172.21.15.102:0/1093367592 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.868+0000 7f6931c00640 1 -- 172.21.15.102:0/1093367592 >> v2:172.21.15.102:3300/0 conn(0x7f692c072700 msgr2=0x7f692c158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.868+0000 7f6931c00640 1 --2- 172.21.15.102:0/1093367592 >> v2:172.21.15.102:3300/0 conn(0x7f692c072700 0x7f692c158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.868+0000 7f6931c00640 1 -- 172.21.15.102:0/1093367592 >> v1:172.21.15.102:6789/0 conn(0x7f692c1524c0 legacy=0x7f692c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:31.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.868+0000 7f6931c00640 1 -- 172.21.15.102:0/1093367592 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692c15c7c0 con 0x7f692c15b8e0 2024-03-18T16:33:31.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.871+0000 7f6931c00640 1 --2- 172.21.15.102:0/1093367592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c15b8e0 0x7f692c0c51e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6924009e50 tx=0x7f692402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c47fa9b6214033 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:31.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.876+0000 7f692b400640 1 -- 172.21.15.102:0/1093367592 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692402ca70 con 0x7f692c15b8e0 2024-03-18T16:33:31.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.876+0000 7f692b400640 1 -- 172.21.15.102:0/1093367592 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f692402cc10 con 0x7f692c15b8e0 2024-03-18T16:33:31.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.876+0000 7f692b400640 1 -- 172.21.15.102:0/1093367592 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69240343f0 con 0x7f692c15b8e0 2024-03-18T16:33:31.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.877+0000 7f6933c00640 1 -- 172.21.15.102:0/1093367592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c15b8e0 msgr2=0x7f692c0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:31.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.877+0000 7f6933c00640 1 --2- 172.21.15.102:0/1093367592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c15b8e0 0x7f692c0c51e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6924009e50 tx=0x7f692402bed0 comp rx=0 tx=0).stop 2024-03-18T16:33:31.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.877+0000 7f6933c00640 1 -- 172.21.15.102:0/1093367592 shutdown_connections 2024-03-18T16:33:31.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.877+0000 7f6933c00640 1 --2- 172.21.15.102:0/1093367592 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c15b8e0 0x7f692c0c51e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:31.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.877+0000 7f6933c00640 1 --2- 172.21.15.102:0/1093367592 >> v2:172.21.15.102:3300/0 conn(0x7f692c072700 0x7f692c158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:31.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.877+0000 7f6933c00640 1 -- 172.21.15.102:0/1093367592 >> 172.21.15.102:0/1093367592 conn(0x7f692c0700d0 msgr2=0x7f692c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:31.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.877+0000 7f6933c00640 1 -- 172.21.15.102:0/1093367592 shutdown_connections 2024-03-18T16:33:31.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.877+0000 7f6933c00640 1 -- 172.21.15.102:0/1093367592 wait complete. 2024-03-18T16:33:31.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.880+0000 7f6933c00640 1 Processor -- start 2024-03-18T16:33:31.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.880+0000 7f6933c00640 1 -- start start 2024-03-18T16:33:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.881+0000 7f6933c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f692c072700 0x7f692c1ec950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.881+0000 7f6933c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c1524c0 0x7f692c0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.882+0000 7f6933c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f692c15b8e0 0x7f692c14ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.882+0000 7f6933c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f692c1589c0 con 0x7f692c072700 2024-03-18T16:33:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.882+0000 7f6933c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f692c1586e0 con 0x7f692c15b8e0 2024-03-18T16:33:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.882+0000 7f6933c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f692c1528a0 con 0x7f692c1524c0 2024-03-18T16:33:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.882+0000 7f692be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c1524c0 0x7f692c0ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.882+0000 7f6931c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f692c15b8e0 0x7f692c14ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.883+0000 7f692be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c1524c0 0x7f692c0ca050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38274/0 (socket says 172.21.15.102:38274) 2024-03-18T16:33:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.883+0000 7f692be00640 1 -- 172.21.15.102:0/112298629 learned_addr learned my addr 172.21.15.102:0/112298629 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.888+0000 7f692be00640 1 -- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f692c15b8e0 msgr2=0x7f692c14ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.888+0000 7f692be00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f692c15b8e0 0x7f692c14ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.888+0000 7f692be00640 1 -- 172.21.15.102:0/112298629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f692c072700 msgr2=0x7f692c1ec950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.888+0000 7f692be00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f692c072700 0x7f692c1ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.888+0000 7f692be00640 1 -- 172.21.15.102:0/112298629 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692c0c7780 con 0x7f692c1524c0 2024-03-18T16:33:31.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.888+0000 7f6931c00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f692c15b8e0 0x7f692c14ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:33:31.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.891+0000 7f692be00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c1524c0 0x7f692c0ca050 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f691800b440 tx=0x7f691800b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.894+0000 7f6928c00640 1 -- 172.21.15.102:0/112298629 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918015020 con 0x7f692c1524c0 2024-03-18T16:33:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.894+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692c0c8700 con 0x7f692c1524c0 2024-03-18T16:33:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.894+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f692c0c6ca0 con 0x7f692c1524c0 2024-03-18T16:33:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.895+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f692c074fd0 con 0x7f692c1524c0 2024-03-18T16:33:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.897+0000 7f6928c00640 1 -- 172.21.15.102:0/112298629 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6918011890 con 0x7f692c1524c0 2024-03-18T16:33:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:31.897+0000 7f6928c00640 1 -- 172.21.15.102:0/112298629 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918009210 con 0x7f692c1524c0 2024-03-18T16:33:32.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.168+0000 7f6928c00640 1 -- 172.21.15.102:0/112298629 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6918018020 con 0x7f692c1524c0 2024-03-18T16:33:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.170+0000 7f6928c00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f690c06bbd0 0x7f690c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:32.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.170+0000 7f6931200640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f690c06bbd0 0x7f690c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:32.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.171+0000 7f6928c00640 1 -- 172.21.15.102:0/112298629 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f69180574b0 con 0x7f692c1524c0 2024-03-18T16:33:32.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.171+0000 7f6931200640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f690c06bbd0 0x7f690c06e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6920001560 tx=0x7f6920007990 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:32.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.183+0000 7f6928c00640 1 -- 172.21.15.102:0/112298629 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f691801f020 con 0x7f692c1524c0 2024-03-18T16:33:32.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.446+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f692c157f00 con 0x7f692c1524c0 2024-03-18T16:33:32.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.459+0000 7f6928c00640 1 -- 172.21.15.102:0/112298629 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f691800d040 con 0x7f692c1524c0 2024-03-18T16:33:32.461 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:32.461 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:32.462 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f690c06bbd0 msgr2=0x7f690c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f690c06bbd0 0x7f690c06e090 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6920001560 tx=0x7f6920007990 comp rx=0 tx=0).stop 2024-03-18T16:33:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c1524c0 msgr2=0x7f692c0ca050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c1524c0 0x7f692c0ca050 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f691800b440 tx=0x7f691800b900 comp rx=0 tx=0).stop 2024-03-18T16:33:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 shutdown_connections 2024-03-18T16:33:32.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f690c06bbd0 0x7f690c06e090 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f692c15b8e0 0x7f692c14ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f692c1524c0 0x7f692c0ca050 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 --2- 172.21.15.102:0/112298629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f692c072700 0x7f692c1ec950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 >> 172.21.15.102:0/112298629 conn(0x7f692c0700d0 msgr2=0x7f692c06dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 shutdown_connections 2024-03-18T16:33:32.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:32.463+0000 7f6933c00640 1 -- 172.21.15.102:0/112298629 wait complete. 2024-03-18T16:33:33.088 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:33.088 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:34.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:34.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.264+0000 7fa725a00640 1 Processor -- start 2024-03-18T16:33:34.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.265+0000 7fa725a00640 1 -- start start 2024-03-18T16:33:34.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.266+0000 7fa725a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa720072700 0x7fa720158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:34.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.266+0000 7fa725a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa72015c200 con 0x7fa7201524c0 2024-03-18T16:33:34.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.266+0000 7fa725a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa72015c4e0 con 0x7fa72015b8e0 2024-03-18T16:33:34.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.266+0000 7fa725a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa720072b20 con 0x7fa720072700 2024-03-18T16:33:34.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.267+0000 7fa71e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa720072700 0x7fa720158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:34.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.267+0000 7fa71f400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa72015b8e0 0x7fa7200c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:40636/0 (socket says 172.21.15.102:40636) 2024-03-18T16:33:34.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.267+0000 7fa71f400640 1 -- 172.21.15.102:0/3994681495 learned_addr learned my addr 172.21.15.102:0/3994681495 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:33:34.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.269+0000 7fa71d600640 1 -- 172.21.15.102:0/3994681495 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932375523 0 0) 0x7fa72015c4e0 con 0x7fa72015b8e0 2024-03-18T16:33:34.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.270+0000 7fa71d600640 1 -- 172.21.15.102:0/3994681495 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa704003520 con 0x7fa72015b8e0 2024-03-18T16:33:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.271+0000 7fa71e000640 1 -- 172.21.15.102:0/3994681495 >> v1:172.21.15.154:6789/0 conn(0x7fa72015b8e0 legacy=0x7fa7200c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.271+0000 7fa71e000640 1 -- 172.21.15.102:0/3994681495 >> v1:172.21.15.102:6789/0 conn(0x7fa7201524c0 legacy=0x7fa72006e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:34.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.271+0000 7fa71e000640 1 -- 172.21.15.102:0/3994681495 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa72015c7c0 con 0x7fa720072700 2024-03-18T16:33:34.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.275+0000 7fa71e000640 1 --2- 172.21.15.102:0/3994681495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa720072700 0x7fa720158ee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0097a0 tx=0x7fa70c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd925e7bf2b0d55 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:34.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.281+0000 7fa71d600640 1 -- 172.21.15.102:0/3994681495 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c034070 con 0x7fa720072700 2024-03-18T16:33:34.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.281+0000 7fa71d600640 1 -- 172.21.15.102:0/3994681495 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa70c030610 con 0x7fa720072700 2024-03-18T16:33:34.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.282+0000 7fa71d600640 1 -- 172.21.15.102:0/3994681495 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70c0309e0 con 0x7fa720072700 2024-03-18T16:33:34.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.282+0000 7fa725a00640 1 -- 172.21.15.102:0/3994681495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa720072700 msgr2=0x7fa720158ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:34.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.282+0000 7fa725a00640 1 --2- 172.21.15.102:0/3994681495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa720072700 0x7fa720158ee0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0097a0 tx=0x7fa70c02b960 comp rx=0 tx=0).stop 2024-03-18T16:33:34.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.282+0000 7fa725a00640 1 -- 172.21.15.102:0/3994681495 shutdown_connections 2024-03-18T16:33:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.282+0000 7fa725a00640 1 --2- 172.21.15.102:0/3994681495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa720072700 0x7fa720158ee0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.282+0000 7fa725a00640 1 -- 172.21.15.102:0/3994681495 >> 172.21.15.102:0/3994681495 conn(0x7fa7200700d0 msgr2=0x7fa72006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.283+0000 7fa725a00640 1 -- 172.21.15.102:0/3994681495 shutdown_connections 2024-03-18T16:33:34.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.283+0000 7fa725a00640 1 -- 172.21.15.102:0/3994681495 wait complete. 2024-03-18T16:33:34.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.284+0000 7fa725a00640 1 Processor -- start 2024-03-18T16:33:34.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.285+0000 7fa725a00640 1 -- start start 2024-03-18T16:33:34.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.285+0000 7fa725a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa720072700 0x7fa72014ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:34.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.285+0000 7fa725a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7201524c0 0x7fa72015a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:34.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.286+0000 7fa725a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa72015b8e0 0x7fa7200c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:34.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.286+0000 7fa725a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa7201589c0 con 0x7fa7201524c0 2024-03-18T16:33:34.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.286+0000 7fa725a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa7201586e0 con 0x7fa72015b8e0 2024-03-18T16:33:34.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.286+0000 7fa725a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa7201528a0 con 0x7fa720072700 2024-03-18T16:33:34.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.286+0000 7fa71e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7201524c0 0x7fa72015a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:34.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.286+0000 7fa71f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa72015b8e0 0x7fa7200c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:34.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.286+0000 7fa71ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa720072700 0x7fa72014ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:34.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.287+0000 7fa71f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa72015b8e0 0x7fa7200c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:44412/0 (socket says 172.21.15.102:44412) 2024-03-18T16:33:34.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.287+0000 7fa71f400640 1 -- 172.21.15.102:0/1684386260 learned_addr learned my addr 172.21.15.102:0/1684386260 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:34.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.292+0000 7fa71f400640 1 -- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa720072700 msgr2=0x7fa72014ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:34.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.292+0000 7fa71f400640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa720072700 0x7fa72014ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:34.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.292+0000 7fa71f400640 1 -- 172.21.15.102:0/1684386260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7201524c0 msgr2=0x7fa72015a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:34.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.292+0000 7fa71f400640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7201524c0 0x7fa72015a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:34.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.292+0000 7fa71f400640 1 -- 172.21.15.102:0/1684386260 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7200c2d80 con 0x7fa72015b8e0 2024-03-18T16:33:34.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.295+0000 7fa71f400640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa72015b8e0 0x7fa7200c6740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa71800cad0 tx=0x7fa71800cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:34.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.299+0000 7fa716a00640 1 -- 172.21.15.102:0/1684386260 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718017070 con 0x7fa72015b8e0 2024-03-18T16:33:34.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.299+0000 7fa716a00640 1 -- 172.21.15.102:0/1684386260 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa71800ac30 con 0x7fa72015b8e0 2024-03-18T16:33:34.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.299+0000 7fa716a00640 1 -- 172.21.15.102:0/1684386260 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718012400 con 0x7fa72015b8e0 2024-03-18T16:33:34.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.299+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa72014ba30 con 0x7fa72015b8e0 2024-03-18T16:33:34.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.299+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa72014cd20 con 0x7fa72015b8e0 2024-03-18T16:33:34.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.301+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa720074fd0 con 0x7fa72015b8e0 2024-03-18T16:33:34.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.330+0000 7fa716a00640 1 -- 172.21.15.102:0/1684386260 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa71801a070 con 0x7fa72015b8e0 2024-03-18T16:33:34.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.332+0000 7fa716a00640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa70006bc20 0x7fa70006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:34.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.332+0000 7fa71ea00640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa70006bc20 0x7fa70006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:34.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.333+0000 7fa71ea00640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa70006bc20 0x7fa70006e0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa710001780 tx=0x7fa710009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:34.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.335+0000 7fa716a00640 1 -- 172.21.15.102:0/1684386260 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa718057130 con 0x7fa72015b8e0 2024-03-18T16:33:34.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.592+0000 7fa716a00640 1 -- 172.21.15.102:0/1684386260 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa718021020 con 0x7fa72015b8e0 2024-03-18T16:33:34.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.839+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7200cad10 con 0x7fa72015b8e0 2024-03-18T16:33:34.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.854+0000 7fa716a00640 1 -- 172.21.15.102:0/1684386260 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa718059090 con 0x7fa72015b8e0 2024-03-18T16:33:34.856 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:34.856 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:34.856 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:34.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa70006bc20 msgr2=0x7fa70006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:34.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa70006bc20 0x7fa70006e0e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa710001780 tx=0x7fa710009040 comp rx=0 tx=0).stop 2024-03-18T16:33:34.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa72015b8e0 msgr2=0x7fa7200c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:34.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa72015b8e0 0x7fa7200c6740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa71800cad0 tx=0x7fa71800cf90 comp rx=0 tx=0).stop 2024-03-18T16:33:34.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 shutdown_connections 2024-03-18T16:33:34.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa70006bc20 0x7fa70006e0e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:34.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa72015b8e0 0x7fa7200c6740 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:34.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa7201524c0 0x7fa72015a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:34.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 --2- 172.21.15.102:0/1684386260 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa720072700 0x7fa72014ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:34.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 >> 172.21.15.102:0/1684386260 conn(0x7fa7200700d0 msgr2=0x7fa72006dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:34.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 shutdown_connections 2024-03-18T16:33:34.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:34.857+0000 7fa725a00640 1 -- 172.21.15.102:0/1684386260 wait complete. 2024-03-18T16:33:35.482 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:35.482 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:36.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:36.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.655+0000 7efe7e600640 1 Processor -- start 2024-03-18T16:33:36.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.656+0000 7efe7e600640 1 -- start start 2024-03-18T16:33:36.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.657+0000 7efe7e600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe78158c00 0x7efe7806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:36.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.657+0000 7efe7e600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efe7806fd80 0x7efe78072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:36.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.657+0000 7efe7e600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe780c4190 con 0x7efe78071700 2024-03-18T16:33:36.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.657+0000 7efe7e600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7efe78158fe0 con 0x7efe7806fd80 2024-03-18T16:33:36.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.657+0000 7efe7e600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efe7806f3b0 con 0x7efe78158c00 2024-03-18T16:33:36.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.658+0000 7efe7d200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe78158c00 0x7efe7806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:36.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.658+0000 7efe77e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efe7806fd80 0x7efe78072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:36.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.658+0000 7efe7d200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe78158c00 0x7efe7806f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38296/0 (socket says 172.21.15.102:38296) 2024-03-18T16:33:36.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.658+0000 7efe7d200640 1 -- 172.21.15.102:0/15792833 learned_addr learned my addr 172.21.15.102:0/15792833 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:36.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.661+0000 7efe7d200640 1 -- 172.21.15.102:0/15792833 >> v2:172.21.15.102:3300/0 conn(0x7efe7806fd80 msgr2=0x7efe78072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:36.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.661+0000 7efe7d200640 1 --2- 172.21.15.102:0/15792833 >> v2:172.21.15.102:3300/0 conn(0x7efe7806fd80 0x7efe78072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:36.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.661+0000 7efe7d200640 1 -- 172.21.15.102:0/15792833 >> v1:172.21.15.102:6789/0 conn(0x7efe78071700 legacy=0x7efe7815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:36.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.661+0000 7efe7d200640 1 -- 172.21.15.102:0/15792833 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7806f060 con 0x7efe78158c00 2024-03-18T16:33:36.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.664+0000 7efe7d200640 1 --2- 172.21.15.102:0/15792833 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe78158c00 0x7efe7806f840 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009870 tx=0x7efe6c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a9c9eb6967e4552 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:36.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.668+0000 7efe77400640 1 -- 172.21.15.102:0/15792833 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6c034070 con 0x7efe78158c00 2024-03-18T16:33:36.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.668+0000 7efe77400640 1 -- 172.21.15.102:0/15792833 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe6c030840 con 0x7efe78158c00 2024-03-18T16:33:36.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.668+0000 7efe77400640 1 -- 172.21.15.102:0/15792833 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6c030c10 con 0x7efe78158c00 2024-03-18T16:33:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.669+0000 7efe7e600640 1 -- 172.21.15.102:0/15792833 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe78158c00 msgr2=0x7efe7806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.669+0000 7efe7e600640 1 --2- 172.21.15.102:0/15792833 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe78158c00 0x7efe7806f840 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7efe6c009870 tx=0x7efe6c02b980 comp rx=0 tx=0).stop 2024-03-18T16:33:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.669+0000 7efe7e600640 1 -- 172.21.15.102:0/15792833 shutdown_connections 2024-03-18T16:33:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.669+0000 7efe7e600640 1 --2- 172.21.15.102:0/15792833 >> v2:172.21.15.102:3300/0 conn(0x7efe7806fd80 0x7efe78072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.669+0000 7efe7e600640 1 --2- 172.21.15.102:0/15792833 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe78158c00 0x7efe7806f840 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.669+0000 7efe7e600640 1 -- 172.21.15.102:0/15792833 >> 172.21.15.102:0/15792833 conn(0x7efe780cb890 msgr2=0x7efe780cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.669+0000 7efe7e600640 1 -- 172.21.15.102:0/15792833 shutdown_connections 2024-03-18T16:33:36.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.669+0000 7efe7e600640 1 -- 172.21.15.102:0/15792833 wait complete. 2024-03-18T16:33:36.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.671+0000 7efe7e600640 1 Processor -- start 2024-03-18T16:33:36.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.671+0000 7efe7e600640 1 -- start start 2024-03-18T16:33:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.672+0000 7efe7e600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe7806fd80 0x7efe7815e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.672+0000 7efe7e600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe78071700 0x7efe780caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.672+0000 7efe7e600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe78158c00 0x7efe781488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.672+0000 7efe7e600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efe78072270 con 0x7efe78071700 2024-03-18T16:33:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.672+0000 7efe7e600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efe78071f90 con 0x7efe78158c00 2024-03-18T16:33:36.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.672+0000 7efe7e600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efe78072550 con 0x7efe7806fd80 2024-03-18T16:33:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.673+0000 7efe7d200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe7806fd80 0x7efe7815e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.673+0000 7efe77e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe78071700 0x7efe780caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.673+0000 7efe7dc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe78158c00 0x7efe781488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.673+0000 7efe7d200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe7806fd80 0x7efe7815e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38308/0 (socket says 172.21.15.102:38308) 2024-03-18T16:33:36.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.673+0000 7efe7d200640 1 -- 172.21.15.102:0/2338315155 learned_addr learned my addr 172.21.15.102:0/2338315155 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.677+0000 7efe7dc00640 1 -- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe7806fd80 msgr2=0x7efe7815e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.677+0000 7efe7dc00640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe7806fd80 0x7efe7815e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.678+0000 7efe7dc00640 1 -- 172.21.15.102:0/2338315155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe78071700 msgr2=0x7efe780caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.678+0000 7efe7dc00640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe78071700 0x7efe780caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:36.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.678+0000 7efe7dc00640 1 -- 172.21.15.102:0/2338315155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe7815f330 con 0x7efe78158c00 2024-03-18T16:33:36.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.680+0000 7efe7dc00640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe78158c00 0x7efe781488b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7efe7000c6c0 tx=0x7efe7000cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:36.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.684+0000 7efe74c00640 1 -- 172.21.15.102:0/2338315155 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe70003b70 con 0x7efe78158c00 2024-03-18T16:33:36.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.684+0000 7efe74c00640 1 -- 172.21.15.102:0/2338315155 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe70003d10 con 0x7efe78158c00 2024-03-18T16:33:36.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.684+0000 7efe74c00640 1 -- 172.21.15.102:0/2338315155 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe700052f0 con 0x7efe78158c00 2024-03-18T16:33:36.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.684+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe7815c700 con 0x7efe78158c00 2024-03-18T16:33:36.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.684+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe7815e360 con 0x7efe78158c00 2024-03-18T16:33:36.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.686+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe7815cc20 con 0x7efe78158c00 2024-03-18T16:33:36.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.706+0000 7efe74c00640 1 -- 172.21.15.102:0/2338315155 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efe70005490 con 0x7efe78158c00 2024-03-18T16:33:36.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.707+0000 7efe74c00640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe5806bc20 0x7efe5806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:36.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.707+0000 7efe7d200640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe5806bc20 0x7efe5806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:36.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.708+0000 7efe74c00640 1 -- 172.21.15.102:0/2338315155 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7efe700577b0 con 0x7efe78158c00 2024-03-18T16:33:36.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.708+0000 7efe7d200640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe5806bc20 0x7efe5806e0e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7efe6c02b9c0 tx=0x7efe6c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:36.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:36.958+0000 7efe74c00640 1 -- 172.21.15.102:0/2338315155 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efe7815cc20 con 0x7efe78158c00 2024-03-18T16:33:37.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.219+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe7815e560 con 0x7efe78158c00 2024-03-18T16:33:37.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.234+0000 7efe74c00640 1 -- 172.21.15.102:0/2338315155 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7efe70059710 con 0x7efe78158c00 2024-03-18T16:33:37.237 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:37.237 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:37.237 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe5806bc20 msgr2=0x7efe5806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe5806bc20 0x7efe5806e0e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7efe6c02b9c0 tx=0x7efe6c008030 comp rx=0 tx=0).stop 2024-03-18T16:33:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe78158c00 msgr2=0x7efe781488b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe78158c00 0x7efe781488b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7efe7000c6c0 tx=0x7efe7000cb80 comp rx=0 tx=0).stop 2024-03-18T16:33:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 shutdown_connections 2024-03-18T16:33:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efe5806bc20 0x7efe5806e0e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe78158c00 0x7efe781488b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efe78071700 0x7efe780caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:37.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 --2- 172.21.15.102:0/2338315155 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efe7806fd80 0x7efe7815e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:37.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 >> 172.21.15.102:0/2338315155 conn(0x7efe780cb890 msgr2=0x7efe78157ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:37.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 shutdown_connections 2024-03-18T16:33:37.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:37.238+0000 7efe7e600640 1 -- 172.21.15.102:0/2338315155 wait complete. 2024-03-18T16:33:37.869 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:37.869 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:38.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:39.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.054+0000 7f6cfa200640 1 Processor -- start 2024-03-18T16:33:39.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.054+0000 7f6cfa200640 1 -- start start 2024-03-18T16:33:39.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.055+0000 7f6cfa200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6cf415a6f0 0x7f6cf415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:39.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.055+0000 7f6cfa200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf40ca500 con 0x7f6cf415a310 2024-03-18T16:33:39.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.056+0000 7f6cfa200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf415b080 con 0x7f6cf415baa0 2024-03-18T16:33:39.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.056+0000 7f6cfa200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf40cd410 con 0x7f6cf415a6f0 2024-03-18T16:33:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.056+0000 7f6cf3e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6cf415a6f0 0x7f6cf415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.057+0000 7f6cf8e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6cf415a310 0x7f6cf406d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41282/0 (socket says 172.21.15.102:41282) 2024-03-18T16:33:39.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.057+0000 7f6cf8e00640 1 -- 172.21.15.102:0/826468068 learned_addr learned my addr 172.21.15.102:0/826468068 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:33:39.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.060+0000 7f6cf3400640 1 -- 172.21.15.102:0/826468068 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1320535371 0 0) 0x7f6cf40ca500 con 0x7f6cf415a310 2024-03-18T16:33:39.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.060+0000 7f6cf3400640 1 -- 172.21.15.102:0/826468068 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cdc003520 con 0x7f6cf415a310 2024-03-18T16:33:39.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.061+0000 7f6cf3e00640 1 -- 172.21.15.102:0/826468068 >> v1:172.21.15.154:6789/0 conn(0x7f6cf415baa0 legacy=0x7f6cf415bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:39.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.061+0000 7f6cf3e00640 1 -- 172.21.15.102:0/826468068 >> v1:172.21.15.102:6789/0 conn(0x7f6cf415a310 legacy=0x7f6cf406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:39.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.061+0000 7f6cf3e00640 1 -- 172.21.15.102:0/826468068 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf40ca7a0 con 0x7f6cf415a6f0 2024-03-18T16:33:39.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.066+0000 7f6cf3e00640 1 --2- 172.21.15.102:0/826468068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a6f0 0x7f6cf415ab40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0c3f6371fa6b787 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:39.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.070+0000 7f6cf3400640 1 -- 172.21.15.102:0/826468068 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0035020 con 0x7f6cf415a6f0 2024-03-18T16:33:39.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.070+0000 7f6cf3400640 1 -- 172.21.15.102:0/826468068 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce0031830 con 0x7f6cf415a6f0 2024-03-18T16:33:39.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.070+0000 7f6cf3400640 1 -- 172.21.15.102:0/826468068 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0031c00 con 0x7f6cf415a6f0 2024-03-18T16:33:39.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.071+0000 7f6cfa200640 1 -- 172.21.15.102:0/826468068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a6f0 msgr2=0x7f6cf415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:39.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.071+0000 7f6cfa200640 1 --2- 172.21.15.102:0/826468068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a6f0 0x7f6cf415ab40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00097a0 tx=0x7f6ce002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:33:39.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.071+0000 7f6cfa200640 1 -- 172.21.15.102:0/826468068 shutdown_connections 2024-03-18T16:33:39.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.071+0000 7f6cfa200640 1 --2- 172.21.15.102:0/826468068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a6f0 0x7f6cf415ab40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:39.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.071+0000 7f6cfa200640 1 -- 172.21.15.102:0/826468068 >> 172.21.15.102:0/826468068 conn(0x7f6cf408c1a0 msgr2=0x7f6cf4066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:39.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.071+0000 7f6cfa200640 1 -- 172.21.15.102:0/826468068 shutdown_connections 2024-03-18T16:33:39.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.072+0000 7f6cfa200640 1 -- 172.21.15.102:0/826468068 wait complete. 2024-03-18T16:33:39.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.073+0000 7f6cfa200640 1 Processor -- start 2024-03-18T16:33:39.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.073+0000 7f6cfa200640 1 -- start start 2024-03-18T16:33:39.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.074+0000 7f6cfa200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a310 0x7f6cf41e5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:39.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.074+0000 7f6cfa200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cf415a6f0 0x7f6cf4158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:39.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.074+0000 7f6cfa200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf415baa0 0x7f6cf414dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:39.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.074+0000 7f6cfa200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf40cd130 con 0x7f6cf415a6f0 2024-03-18T16:33:39.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.074+0000 7f6cfa200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf40cce50 con 0x7f6cf415a310 2024-03-18T16:33:39.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.074+0000 7f6cfa200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf41548c0 con 0x7f6cf415baa0 2024-03-18T16:33:39.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.075+0000 7f6cf8e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a310 0x7f6cf41e5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:39.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.075+0000 7f6cf9800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf415baa0 0x7f6cf414dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:39.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.075+0000 7f6cf3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cf415a6f0 0x7f6cf4158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:39.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.075+0000 7f6cf8e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a310 0x7f6cf41e5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:44452/0 (socket says 172.21.15.102:44452) 2024-03-18T16:33:39.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.075+0000 7f6cf8e00640 1 -- 172.21.15.102:0/2016468879 learned_addr learned my addr 172.21.15.102:0/2016468879 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:39.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.079+0000 7f6cf3e00640 1 -- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf415baa0 msgr2=0x7f6cf414dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:39.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.079+0000 7f6cf3e00640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf415baa0 0x7f6cf414dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:39.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.079+0000 7f6cf3e00640 1 -- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a310 msgr2=0x7f6cf41e5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:39.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.079+0000 7f6cf3e00640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a310 0x7f6cf41e5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:39.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.079+0000 7f6cf3e00640 1 -- 172.21.15.102:0/2016468879 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf41e78b0 con 0x7f6cf415a6f0 2024-03-18T16:33:39.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.082+0000 7f6cf3e00640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cf415a6f0 0x7f6cf4158730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40ca6d0 tx=0x7f6ce00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:39.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.087+0000 7f6cf0c00640 1 -- 172.21.15.102:0/2016468879 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0041070 con 0x7f6cf415a6f0 2024-03-18T16:33:39.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.087+0000 7f6cf0c00640 1 -- 172.21.15.102:0/2016468879 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ce003f070 con 0x7f6cf415a6f0 2024-03-18T16:33:39.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.087+0000 7f6cf0c00640 1 -- 172.21.15.102:0/2016468879 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0035040 con 0x7f6cf415a6f0 2024-03-18T16:33:39.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.087+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf41f0600 con 0x7f6cf415a6f0 2024-03-18T16:33:39.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.087+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf41e6dd0 con 0x7f6cf415a6f0 2024-03-18T16:33:39.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.088+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf4074fd0 con 0x7f6cf415a6f0 2024-03-18T16:33:39.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.108+0000 7f6cf0c00640 1 -- 172.21.15.102:0/2016468879 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6ce0038020 con 0x7f6cf415a6f0 2024-03-18T16:33:39.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.109+0000 7f6cf0c00640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6cd406bbd0 0x7f6cd406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:39.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.110+0000 7f6cf8e00640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6cd406bbd0 0x7f6cd406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:39.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.110+0000 7f6cf8e00640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6cd406bbd0 0x7f6cd406e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8005880 tx=0x7f6ce8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:39.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.382+0000 7f6cf0c00640 1 -- 172.21.15.102:0/2016468879 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6ce0078ce0 con 0x7f6cf415a6f0 2024-03-18T16:33:39.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.383+0000 7f6cf0c00640 1 -- 172.21.15.102:0/2016468879 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6ce0055080 con 0x7f6cf415a6f0 2024-03-18T16:33:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.687+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cf40caed0 con 0x7f6cf415a6f0 2024-03-18T16:33:39.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.708+0000 7f6cf0c00640 1 -- 172.21.15.102:0/2016468879 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f6ce0031800 con 0x7f6cf415a6f0 2024-03-18T16:33:39.712 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:39.712 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:39.712 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:39.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.711+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6cd406bbd0 msgr2=0x7f6cd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:39.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.711+0000 7f6cfa200640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6cd406bbd0 0x7f6cd406e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8005880 tx=0x7f6ce8009040 comp rx=0 tx=0).stop 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.711+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cf415a6f0 msgr2=0x7f6cf4158730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.711+0000 7f6cfa200640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cf415a6f0 0x7f6cf4158730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40ca6d0 tx=0x7f6ce00046c0 comp rx=0 tx=0).stop 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.711+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 shutdown_connections 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.711+0000 7f6cfa200640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6cd406bbd0 0x7f6cd406e090 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.712+0000 7f6cfa200640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6cf415baa0 0x7f6cf414dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.712+0000 7f6cfa200640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6cf415a6f0 0x7f6cf4158730 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6cf40ca6d0 tx=0x7f6ce00046c0 comp rx=0 tx=0).stop 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.712+0000 7f6cfa200640 1 --2- 172.21.15.102:0/2016468879 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6cf415a310 0x7f6cf41e5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.712+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 >> 172.21.15.102:0/2016468879 conn(0x7f6cf408c1a0 msgr2=0x7f6cf41529d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.712+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 shutdown_connections 2024-03-18T16:33:39.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:39.712+0000 7f6cfa200640 1 -- 172.21.15.102:0/2016468879 wait complete. 2024-03-18T16:33:40.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:40.335 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:41.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:41.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.556+0000 7f2958000640 1 Processor -- start 2024-03-18T16:33:41.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.556+0000 7f2958000640 1 -- start start 2024-03-18T16:33:41.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.558+0000 7f2958000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f29500c8b20 0x7f2950072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:41.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.558+0000 7f2958000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2950158760 0x7f295015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:41.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.558+0000 7f2958000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2950157ac0 con 0x7f29500c8740 2024-03-18T16:33:41.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.558+0000 7f2958000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f295015a860 con 0x7f29500c8b20 2024-03-18T16:33:41.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.558+0000 7f2958000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f2950072840 con 0x7f2950158760 2024-03-18T16:33:41.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.559+0000 7f2955600640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f29500c8740 0x7f29501572f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:47728/0 (socket says 172.21.15.102:47728) 2024-03-18T16:33:41.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.559+0000 7f2955600640 1 -- 172.21.15.102:0/681573632 learned_addr learned my addr 172.21.15.102:0/681573632 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:33:41.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.559+0000 7f2956000640 1 --2- 172.21.15.102:0/681573632 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2950158760 0x7f295015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:41.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.561+0000 7f2954c00640 1 --2- 172.21.15.102:0/681573632 >> v2:172.21.15.102:3300/0 conn(0x7f29500c8b20 0x7f2950072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:41.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.562+0000 7f2956000640 1 -- 172.21.15.102:0/681573632 >> v2:172.21.15.102:3300/0 conn(0x7f29500c8b20 msgr2=0x7f2950072300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:41.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.562+0000 7f2956000640 1 --2- 172.21.15.102:0/681573632 >> v2:172.21.15.102:3300/0 conn(0x7f29500c8b20 0x7f2950072300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:41.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.562+0000 7f2956000640 1 -- 172.21.15.102:0/681573632 >> v1:172.21.15.154:6789/0 conn(0x7f29500c8740 legacy=0x7f29501572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:41.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.562+0000 7f2956000640 1 -- 172.21.15.102:0/681573632 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2950157da0 con 0x7f2950158760 2024-03-18T16:33:41.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.564+0000 7f2956000640 1 --2- 172.21.15.102:0/681573632 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2950158760 0x7f295015b2b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2948009d90 tx=0x7f294802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3569dcac1c8687ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:41.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.568+0000 7f294fe00640 1 -- 172.21.15.102:0/681573632 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948039070 con 0x7f2950158760 2024-03-18T16:33:41.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.568+0000 7f294fe00640 1 -- 172.21.15.102:0/681573632 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f294802cc60 con 0x7f2950158760 2024-03-18T16:33:41.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.568+0000 7f294fe00640 1 -- 172.21.15.102:0/681573632 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948034660 con 0x7f2950158760 2024-03-18T16:33:41.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.569+0000 7f2958000640 1 -- 172.21.15.102:0/681573632 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2950158760 msgr2=0x7f295015b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:41.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.569+0000 7f2958000640 1 --2- 172.21.15.102:0/681573632 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2950158760 0x7f295015b2b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2948009d90 tx=0x7f294802beb0 comp rx=0 tx=0).stop 2024-03-18T16:33:41.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.569+0000 7f2958000640 1 -- 172.21.15.102:0/681573632 shutdown_connections 2024-03-18T16:33:41.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.569+0000 7f2958000640 1 --2- 172.21.15.102:0/681573632 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f2950158760 0x7f295015b2b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:41.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.569+0000 7f2958000640 1 --2- 172.21.15.102:0/681573632 >> v2:172.21.15.102:3300/0 conn(0x7f29500c8b20 0x7f2950072300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:41.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.569+0000 7f2958000640 1 -- 172.21.15.102:0/681573632 >> 172.21.15.102:0/681573632 conn(0x7f29500700d0 msgr2=0x7f295006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:41.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.569+0000 7f2958000640 1 -- 172.21.15.102:0/681573632 shutdown_connections 2024-03-18T16:33:41.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.569+0000 7f2958000640 1 -- 172.21.15.102:0/681573632 wait complete. 2024-03-18T16:33:41.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.571+0000 7f2958000640 1 Processor -- start 2024-03-18T16:33:41.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.572+0000 7f2958000640 1 -- start start 2024-03-18T16:33:41.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.573+0000 7f2958000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29500c8740 0x7f2950142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:41.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.573+0000 7f2958000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29500c8b20 0x7f2950148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:41.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.573+0000 7f2958000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2950158760 0x7f29500cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:41.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.573+0000 7f2958000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f295015aab0 con 0x7f2950158760 2024-03-18T16:33:41.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.573+0000 7f2958000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f295015b7f0 con 0x7f29500c8740 2024-03-18T16:33:41.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.573+0000 7f2958000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f295015ad90 con 0x7f29500c8b20 2024-03-18T16:33:41.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.574+0000 7f2954c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29500c8b20 0x7f2950148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:41.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.574+0000 7f2955600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29500c8740 0x7f2950142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:41.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.574+0000 7f2954c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29500c8b20 0x7f2950148ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41814/0 (socket says 172.21.15.102:41814) 2024-03-18T16:33:41.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.574+0000 7f2954c00640 1 -- 172.21.15.102:0/587051339 learned_addr learned my addr 172.21.15.102:0/587051339 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:41.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.574+0000 7f2956000640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2950158760 0x7f29500cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.577+0000 7f2954c00640 1 -- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29500c8740 msgr2=0x7f2950142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.577+0000 7f2954c00640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29500c8740 0x7f2950142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.577+0000 7f2954c00640 1 -- 172.21.15.102:0/587051339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2950158760 msgr2=0x7f29500cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.577+0000 7f2954c00640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2950158760 0x7f29500cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.577+0000 7f2954c00640 1 -- 172.21.15.102:0/587051339 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29500cb100 con 0x7f29500c8b20 2024-03-18T16:33:41.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.580+0000 7f2954c00640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29500c8b20 0x7f2950148ef0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f293c00d5f0 tx=0x7f293c00dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:41.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.585+0000 7f294d600640 1 -- 172.21.15.102:0/587051339 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c0125f0 con 0x7f29500c8b20 2024-03-18T16:33:41.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.585+0000 7f294d600640 1 -- 172.21.15.102:0/587051339 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293c019070 con 0x7f29500c8b20 2024-03-18T16:33:41.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.585+0000 7f294d600640 1 -- 172.21.15.102:0/587051339 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c0129d0 con 0x7f29500c8b20 2024-03-18T16:33:41.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.585+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29501456f0 con 0x7f29500c8b20 2024-03-18T16:33:41.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.585+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2950145c90 con 0x7f29500c8b20 2024-03-18T16:33:41.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.587+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29500c8160 con 0x7f29500c8b20 2024-03-18T16:33:41.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.607+0000 7f294d600640 1 -- 172.21.15.102:0/587051339 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f293c012b70 con 0x7f29500c8b20 2024-03-18T16:33:41.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.608+0000 7f294d600640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f293006bc20 0x7f293006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:41.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.609+0000 7f2955600640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f293006bc20 0x7f293006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:41.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.609+0000 7f2955600640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f293006bc20 0x7f293006e0e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2944007fd0 tx=0x7f2944007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:41.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.843+0000 7f294d600640 1 -- 172.21.15.102:0/587051339 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f293c012e60 con 0x7f29500c8b20 2024-03-18T16:33:41.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:41.854+0000 7f294d600640 1 -- 172.21.15.102:0/587051339 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f29500c8160 con 0x7f29500c8b20 2024-03-18T16:33:42.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.157+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29501639f0 con 0x7f29500c8b20 2024-03-18T16:33:42.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.170+0000 7f294d600640 1 -- 172.21.15.102:0/587051339 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f293c059cd0 con 0x7f29500c8b20 2024-03-18T16:33:42.173 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:42.174 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:42.174 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:42.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f293006bc20 msgr2=0x7f293006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:42.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f293006bc20 0x7f293006e0e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2944007fd0 tx=0x7f2944007e80 comp rx=0 tx=0).stop 2024-03-18T16:33:42.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29500c8b20 msgr2=0x7f2950148ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:42.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29500c8b20 0x7f2950148ef0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f293c00d5f0 tx=0x7f293c00dab0 comp rx=0 tx=0).stop 2024-03-18T16:33:42.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 shutdown_connections 2024-03-18T16:33:42.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f293006bc20 0x7f293006e0e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:42.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2950158760 0x7f29500cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:42.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f29500c8b20 0x7f2950148ef0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:42.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 --2- 172.21.15.102:0/587051339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29500c8740 0x7f2950142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:42.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 >> 172.21.15.102:0/587051339 conn(0x7f29500700d0 msgr2=0x7f2950159580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:42.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 shutdown_connections 2024-03-18T16:33:42.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:42.173+0000 7f2958000640 1 -- 172.21.15.102:0/587051339 wait complete. 2024-03-18T16:33:42.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:42.807 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:43.808 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:43.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.980+0000 7f19d5600640 1 Processor -- start 2024-03-18T16:33:43.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.980+0000 7f19d5600640 1 -- start start 2024-03-18T16:33:43.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.981+0000 7f19d5600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f19d00d0250 0x7f19d015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:43.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.981+0000 7f19d5600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d00cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:43.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.981+0000 7f19d5600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d00cabd0 con 0x7f19d00cfe70 2024-03-18T16:33:43.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.981+0000 7f19d5600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f19d00caea0 con 0x7f19d00d0250 2024-03-18T16:33:43.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.981+0000 7f19d5600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f19d00cb180 con 0x7f19d00cc2f0 2024-03-18T16:33:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.982+0000 7f19d4c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d00cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.982+0000 7f19cfe00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f19d00cfe70 0x7f19d00c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:47738/0 (socket says 172.21.15.102:47738) 2024-03-18T16:33:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.982+0000 7f19cfe00640 1 -- 172.21.15.102:0/2620670220 learned_addr learned my addr 172.21.15.102:0/2620670220 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:33:43.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.982+0000 7f19cf400640 1 --2- 172.21.15.102:0/2620670220 >> v2:172.21.15.102:3300/0 conn(0x7f19d00d0250 0x7f19d015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.985+0000 7f19d4c00640 1 -- 172.21.15.102:0/2620670220 >> v2:172.21.15.102:3300/0 conn(0x7f19d00d0250 msgr2=0x7f19d015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.985+0000 7f19d4c00640 1 --2- 172.21.15.102:0/2620670220 >> v2:172.21.15.102:3300/0 conn(0x7f19d00d0250 0x7f19d015ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.985+0000 7f19d4c00640 1 -- 172.21.15.102:0/2620670220 >> v1:172.21.15.154:6789/0 conn(0x7f19d00cfe70 legacy=0x7f19d00c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:43.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.985+0000 7f19d4c00640 1 -- 172.21.15.102:0/2620670220 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d00cb460 con 0x7f19d00cc2f0 2024-03-18T16:33:43.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.987+0000 7f19d4c00640 1 --2- 172.21.15.102:0/2620670220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d00cc710 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009d90 tx=0x7f19c802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe7ad009e7f9180f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:43.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.991+0000 7f19cea00640 1 -- 172.21.15.102:0/2620670220 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8039070 con 0x7f19d00cc2f0 2024-03-18T16:33:43.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.991+0000 7f19cea00640 1 -- 172.21.15.102:0/2620670220 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c802cce0 con 0x7f19d00cc2f0 2024-03-18T16:33:43.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.991+0000 7f19cea00640 1 -- 172.21.15.102:0/2620670220 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8034660 con 0x7f19d00cc2f0 2024-03-18T16:33:43.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.992+0000 7f19d5600640 1 -- 172.21.15.102:0/2620670220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 msgr2=0x7f19d00cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:43.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.992+0000 7f19d5600640 1 --2- 172.21.15.102:0/2620670220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d00cc710 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f19c8009d90 tx=0x7f19c802bf30 comp rx=0 tx=0).stop 2024-03-18T16:33:43.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.992+0000 7f19d5600640 1 -- 172.21.15.102:0/2620670220 shutdown_connections 2024-03-18T16:33:43.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.992+0000 7f19d5600640 1 --2- 172.21.15.102:0/2620670220 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d00cc710 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:43.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.992+0000 7f19d5600640 1 --2- 172.21.15.102:0/2620670220 >> v2:172.21.15.102:3300/0 conn(0x7f19d00d0250 0x7f19d015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:43.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.992+0000 7f19d5600640 1 -- 172.21.15.102:0/2620670220 >> 172.21.15.102:0/2620670220 conn(0x7f19d008c3e0 msgr2=0x7f19d006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:43.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.992+0000 7f19d5600640 1 -- 172.21.15.102:0/2620670220 shutdown_connections 2024-03-18T16:33:43.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.992+0000 7f19d5600640 1 -- 172.21.15.102:0/2620670220 wait complete. 2024-03-18T16:33:43.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.994+0000 7f19d5600640 1 Processor -- start 2024-03-18T16:33:43.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.994+0000 7f19d5600640 1 -- start start 2024-03-18T16:33:43.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.995+0000 7f19d5600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d0159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:43.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.995+0000 7f19d5600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19d00cfe70 0x7f19d01e0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:43.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.995+0000 7f19d5600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d00d0250 0x7f19d01e14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:43.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.995+0000 7f19d5600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f19d00cbd00 con 0x7f19d00d0250 2024-03-18T16:33:43.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.995+0000 7f19d5600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f19d00cba20 con 0x7f19d00cfe70 2024-03-18T16:33:43.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.995+0000 7f19d5600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f19d00ccc50 con 0x7f19d00cc2f0 2024-03-18T16:33:43.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.996+0000 7f19cf400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19d00cfe70 0x7f19d01e0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:43.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.996+0000 7f19cfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d0159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:43.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.996+0000 7f19d4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d00d0250 0x7f19d01e14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:43.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.996+0000 7f19cf400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19d00cfe70 0x7f19d01e0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53388/0 (socket says 172.21.15.102:53388) 2024-03-18T16:33:43.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:43.997+0000 7f19cf400640 1 -- 172.21.15.102:0/2762164729 learned_addr learned my addr 172.21.15.102:0/2762164729 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:44.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.000+0000 7f19cfe00640 1 -- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19d00cfe70 msgr2=0x7f19d01e0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:44.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.000+0000 7f19cfe00640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19d00cfe70 0x7f19d01e0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:44.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.000+0000 7f19cfe00640 1 -- 172.21.15.102:0/2762164729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d00d0250 msgr2=0x7f19d01e14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:44.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.000+0000 7f19cfe00640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d00d0250 0x7f19d01e14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:44.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.000+0000 7f19cfe00640 1 -- 172.21.15.102:0/2762164729 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d0159710 con 0x7f19d00cc2f0 2024-03-18T16:33:44.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.003+0000 7f19cfe00640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d0159060 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f19c000c370 tx=0x7f19c000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.007+0000 7f19c7e00640 1 -- 172.21.15.102:0/2762164729 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c0017070 con 0x7f19d00cc2f0 2024-03-18T16:33:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.007+0000 7f19c7e00640 1 -- 172.21.15.102:0/2762164729 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c0005ce0 con 0x7f19d00cc2f0 2024-03-18T16:33:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.007+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d01599f0 con 0x7f19d00cc2f0 2024-03-18T16:33:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.007+0000 7f19c7e00640 1 -- 172.21.15.102:0/2762164729 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c0007310 con 0x7f19d00cc2f0 2024-03-18T16:33:44.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.007+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d0159fb0 con 0x7f19d00cc2f0 2024-03-18T16:33:44.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.008+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d00c5f40 con 0x7f19d00cc2f0 2024-03-18T16:33:44.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.266+0000 7f19c7e00640 1 -- 172.21.15.102:0/2762164729 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f19c001a070 con 0x7f19d00cc2f0 2024-03-18T16:33:44.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.267+0000 7f19c7e00640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19b006bc50 0x7f19b006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:44.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.267+0000 7f19cf400640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19b006bc50 0x7f19b006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:44.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.268+0000 7f19cf400640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19b006bc50 0x7f19b006e110 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f19d006ce30 tx=0x7f19bc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:44.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.270+0000 7f19c7e00640 1 -- 172.21.15.102:0/2762164729 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f19c0056750 con 0x7f19d00cc2f0 2024-03-18T16:33:44.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.280+0000 7f19c7e00640 1 -- 172.21.15.102:0/2762164729 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f19d00c5f40 con 0x7f19d00cc2f0 2024-03-18T16:33:44.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.544+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19d0158900 con 0x7f19d00cc2f0 2024-03-18T16:33:44.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.616+0000 7f19c7e00640 1 -- 172.21.15.102:0/2762164729 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f19c0021020 con 0x7f19d00cc2f0 2024-03-18T16:33:44.619 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:44.619 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:44.619 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19b006bc50 msgr2=0x7f19b006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19b006bc50 0x7f19b006e110 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f19d006ce30 tx=0x7f19bc009190 comp rx=0 tx=0).stop 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 msgr2=0x7f19d0159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d0159060 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f19c000c370 tx=0x7f19c000c830 comp rx=0 tx=0).stop 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 shutdown_connections 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f19b006bc50 0x7f19b006e110 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d00d0250 0x7f19d01e14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f19d00cfe70 0x7f19d01e0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 --2- 172.21.15.102:0/2762164729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f19d00cc2f0 0x7f19d0159060 secure :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f19c000c370 tx=0x7f19c000c830 comp rx=0 tx=0).stop 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 >> 172.21.15.102:0/2762164729 conn(0x7f19d008c3e0 msgr2=0x7f19d00c8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 shutdown_connections 2024-03-18T16:33:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:44.620+0000 7f19d5600640 1 -- 172.21.15.102:0/2762164729 wait complete. 2024-03-18T16:33:45.252 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:45.252 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:46.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:46.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.425+0000 7efd8f200640 1 Processor -- start 2024-03-18T16:33:46.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.426+0000 7efd8f200640 1 -- start start 2024-03-18T16:33:46.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.427+0000 7efd8f200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd900ca960 0x7efd9006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:46.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.427+0000 7efd8f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efd900cad40 0x7efd9015e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:46.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.427+0000 7efd8f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd9015a1a0 0x7efd900705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:46.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.427+0000 7efd8f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7efd900c8680 con 0x7efd900cad40 2024-03-18T16:33:46.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.427+0000 7efd8f200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efd9015e670 con 0x7efd900ca960 2024-03-18T16:33:46.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.427+0000 7efd8f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd9006c9c0 con 0x7efd9015a1a0 2024-03-18T16:33:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.428+0000 7efd8e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd9015a1a0 0x7efd900705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.428+0000 7efd8de00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd900ca960 0x7efd9006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.428+0000 7efd8d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efd900cad40 0x7efd9015e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.428+0000 7efd8e800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd9015a1a0 0x7efd900705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41836/0 (socket says 172.21.15.102:41836) 2024-03-18T16:33:46.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.428+0000 7efd8e800640 1 -- 172.21.15.102:0/1415667211 learned_addr learned my addr 172.21.15.102:0/1415667211 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:46.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.431+0000 7efd8e800640 1 -- 172.21.15.102:0/1415667211 >> v2:172.21.15.154:3300/0 conn(0x7efd900ca960 msgr2=0x7efd9006c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:46.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.431+0000 7efd8e800640 1 --2- 172.21.15.102:0/1415667211 >> v2:172.21.15.154:3300/0 conn(0x7efd900ca960 0x7efd9006c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:46.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.431+0000 7efd8e800640 1 -- 172.21.15.102:0/1415667211 >> v2:172.21.15.102:3300/0 conn(0x7efd900cad40 msgr2=0x7efd9015e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:46.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.431+0000 7efd8e800640 1 --2- 172.21.15.102:0/1415667211 >> v2:172.21.15.102:3300/0 conn(0x7efd900cad40 0x7efd9015e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:46.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.431+0000 7efd8e800640 1 -- 172.21.15.102:0/1415667211 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9006cca0 con 0x7efd9015a1a0 2024-03-18T16:33:46.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.435+0000 7efd8e800640 1 --2- 172.21.15.102:0/1415667211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd9015a1a0 0x7efd900705a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efd900cb250 tx=0x7efd8802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a3d141af6f35606 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:46.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.438+0000 7efd8ca00640 1 -- 172.21.15.102:0/1415667211 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8802cb00 con 0x7efd9015a1a0 2024-03-18T16:33:46.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.440+0000 7efd8ca00640 1 -- 172.21.15.102:0/1415667211 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd8802cca0 con 0x7efd9015a1a0 2024-03-18T16:33:46.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.440+0000 7efd8ca00640 1 -- 172.21.15.102:0/1415667211 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88034610 con 0x7efd9015a1a0 2024-03-18T16:33:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.441+0000 7efd8f200640 1 -- 172.21.15.102:0/1415667211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd9015a1a0 msgr2=0x7efd900705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.441+0000 7efd8f200640 1 --2- 172.21.15.102:0/1415667211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd9015a1a0 0x7efd900705a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efd900cb250 tx=0x7efd8802bf60 comp rx=0 tx=0).stop 2024-03-18T16:33:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.441+0000 7efd8f200640 1 -- 172.21.15.102:0/1415667211 shutdown_connections 2024-03-18T16:33:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.441+0000 7efd8f200640 1 --2- 172.21.15.102:0/1415667211 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd9015a1a0 0x7efd900705a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.441+0000 7efd8f200640 1 --2- 172.21.15.102:0/1415667211 >> v2:172.21.15.102:3300/0 conn(0x7efd900cad40 0x7efd9015e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.441+0000 7efd8f200640 1 --2- 172.21.15.102:0/1415667211 >> v2:172.21.15.154:3300/0 conn(0x7efd900ca960 0x7efd9006c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.441+0000 7efd8f200640 1 -- 172.21.15.102:0/1415667211 >> 172.21.15.102:0/1415667211 conn(0x7efd90067190 msgr2=0x7efd900675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.441+0000 7efd8f200640 1 -- 172.21.15.102:0/1415667211 shutdown_connections 2024-03-18T16:33:46.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.441+0000 7efd8f200640 1 -- 172.21.15.102:0/1415667211 wait complete. 2024-03-18T16:33:46.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.444+0000 7efd8f200640 1 Processor -- start 2024-03-18T16:33:46.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.444+0000 7efd8f200640 1 -- start start 2024-03-18T16:33:46.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.446+0000 7efd8f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd900ca960 0x7efd9013beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.446+0000 7efd8f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd900cad40 0x7efd901f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.446+0000 7efd8f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd9015a1a0 0x7efd90156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.446+0000 7efd8f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7efd9015dc10 con 0x7efd900ca960 2024-03-18T16:33:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.446+0000 7efd8f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd9015d930 con 0x7efd9015a1a0 2024-03-18T16:33:46.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.446+0000 7efd8f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7efd90070ae0 con 0x7efd900cad40 2024-03-18T16:33:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.446+0000 7efd8d400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd900cad40 0x7efd901f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.447+0000 7efd8de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd900ca960 0x7efd9013beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.447+0000 7efd8d400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd900cad40 0x7efd901f03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41840/0 (socket says 172.21.15.102:41840) 2024-03-18T16:33:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.447+0000 7efd8d400640 1 -- 172.21.15.102:0/1239932240 learned_addr learned my addr 172.21.15.102:0/1239932240 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:46.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.447+0000 7efd8e800640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd9015a1a0 0x7efd90156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.450+0000 7efd8d400640 1 -- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd9015a1a0 msgr2=0x7efd90156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.450+0000 7efd8d400640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd9015a1a0 0x7efd90156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.450+0000 7efd8d400640 1 -- 172.21.15.102:0/1239932240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd900ca960 msgr2=0x7efd9013beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.450+0000 7efd8d400640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd900ca960 0x7efd9013beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.450+0000 7efd8d400640 1 -- 172.21.15.102:0/1239932240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd901591b0 con 0x7efd900cad40 2024-03-18T16:33:46.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.453+0000 7efd8d400640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd900cad40 0x7efd901f03f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00e870 tx=0x7efd7c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.457+0000 7efd86000640 1 -- 172.21.15.102:0/1239932240 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c017070 con 0x7efd900cad40 2024-03-18T16:33:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.457+0000 7efd86000640 1 -- 172.21.15.102:0/1239932240 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd7c00c940 con 0x7efd900cad40 2024-03-18T16:33:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.457+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd901e38f0 con 0x7efd900cad40 2024-03-18T16:33:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.457+0000 7efd86000640 1 -- 172.21.15.102:0/1239932240 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c00cd10 con 0x7efd900cad40 2024-03-18T16:33:46.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.457+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd901e61b0 con 0x7efd900cad40 2024-03-18T16:33:46.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.459+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd90074fd0 con 0x7efd900cad40 2024-03-18T16:33:46.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.478+0000 7efd86000640 1 -- 172.21.15.102:0/1239932240 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efd7c01a070 con 0x7efd900cad40 2024-03-18T16:33:46.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.479+0000 7efd86000640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd7006bc20 0x7efd7006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:46.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.479+0000 7efd8de00640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd7006bc20 0x7efd7006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:46.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.480+0000 7efd8de00640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd7006bc20 0x7efd7006e0e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efd80009a20 tx=0x7efd80009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:46.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.480+0000 7efd86000640 1 -- 172.21.15.102:0/1239932240 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7efd7c057570 con 0x7efd900cad40 2024-03-18T16:33:46.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:46.729+0000 7efd86000640 1 -- 172.21.15.102:0/1239932240 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efd7c0594d0 con 0x7efd900cad40 2024-03-18T16:33:47.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.060+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd900ca210 con 0x7efd900cad40 2024-03-18T16:33:47.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.076+0000 7efd86000640 1 -- 172.21.15.102:0/1239932240 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7efd7c05d570 con 0x7efd900cad40 2024-03-18T16:33:47.078 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:47.078 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:47.078 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:47.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.079+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd7006bc20 msgr2=0x7efd7006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.079+0000 7efd8f200640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd7006bc20 0x7efd7006e0e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efd80009a20 tx=0x7efd80009190 comp rx=0 tx=0).stop 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd900cad40 msgr2=0x7efd901f03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd900cad40 0x7efd901f03f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efd7c00e870 tx=0x7efd7c00ed30 comp rx=0 tx=0).stop 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 shutdown_connections 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7efd7006bc20 0x7efd7006e0e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd9015a1a0 0x7efd90156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7efd900cad40 0x7efd901f03f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 --2- 172.21.15.102:0/1239932240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd900ca960 0x7efd9013beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 >> 172.21.15.102:0/1239932240 conn(0x7efd90067190 msgr2=0x7efd90158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 shutdown_connections 2024-03-18T16:33:47.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:47.080+0000 7efd8f200640 1 -- 172.21.15.102:0/1239932240 wait complete. 2024-03-18T16:33:47.703 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:47.703 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:48.705 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:48.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.890+0000 7fe959200640 1 Processor -- start 2024-03-18T16:33:48.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.891+0000 7fe959200640 1 -- start start 2024-03-18T16:33:48.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.892+0000 7fe959200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe95406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.892+0000 7fe959200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe95415a6f0 0x7fe95415ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.893+0000 7fe959200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9540ca500 con 0x7fe95415baa0 2024-03-18T16:33:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.893+0000 7fe959200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe9540cd410 con 0x7fe95415a6f0 2024-03-18T16:33:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.893+0000 7fe959200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe95415b050 con 0x7fe95415a310 2024-03-18T16:33:48.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.893+0000 7fe953400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe95406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:48.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.894+0000 7fe953400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe95406d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41856/0 (socket says 172.21.15.102:41856) 2024-03-18T16:33:48.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.894+0000 7fe953400640 1 -- 172.21.15.102:0/2387459146 learned_addr learned my addr 172.21.15.102:0/2387459146 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:48.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.896+0000 7fe952a00640 1 --2- 172.21.15.102:0/2387459146 >> v2:172.21.15.102:3300/0 conn(0x7fe95415a6f0 0x7fe95415ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:48.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.896+0000 7fe953400640 1 -- 172.21.15.102:0/2387459146 >> v2:172.21.15.102:3300/0 conn(0x7fe95415a6f0 msgr2=0x7fe95415ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:48.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.896+0000 7fe953400640 1 --2- 172.21.15.102:0/2387459146 >> v2:172.21.15.102:3300/0 conn(0x7fe95415a6f0 0x7fe95415ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:48.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.896+0000 7fe953400640 1 -- 172.21.15.102:0/2387459146 >> v1:172.21.15.102:6789/0 conn(0x7fe95415baa0 legacy=0x7fe95415bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:48.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.896+0000 7fe953400640 1 -- 172.21.15.102:0/2387459146 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9540ca7a0 con 0x7fe95415a310 2024-03-18T16:33:48.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.899+0000 7fe953400640 1 --2- 172.21.15.102:0/2387459146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe95406d3e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe944009870 tx=0x7fe94402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98fe26f6476ce4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.903+0000 7fe952000640 1 -- 172.21.15.102:0/2387459146 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944035020 con 0x7fe95415a310 2024-03-18T16:33:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.903+0000 7fe952000640 1 -- 172.21.15.102:0/2387459146 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe944031640 con 0x7fe95415a310 2024-03-18T16:33:48.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.903+0000 7fe952000640 1 -- 172.21.15.102:0/2387459146 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94402d040 con 0x7fe95415a310 2024-03-18T16:33:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.904+0000 7fe959200640 1 -- 172.21.15.102:0/2387459146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 msgr2=0x7fe95406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.904+0000 7fe959200640 1 --2- 172.21.15.102:0/2387459146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe95406d3e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe944009870 tx=0x7fe94402b8f0 comp rx=0 tx=0).stop 2024-03-18T16:33:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.904+0000 7fe959200640 1 -- 172.21.15.102:0/2387459146 shutdown_connections 2024-03-18T16:33:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.904+0000 7fe959200640 1 --2- 172.21.15.102:0/2387459146 >> v2:172.21.15.102:3300/0 conn(0x7fe95415a6f0 0x7fe95415ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.904+0000 7fe959200640 1 --2- 172.21.15.102:0/2387459146 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe95406d3e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.904+0000 7fe959200640 1 -- 172.21.15.102:0/2387459146 >> 172.21.15.102:0/2387459146 conn(0x7fe95408c1a0 msgr2=0x7fe954066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.904+0000 7fe959200640 1 -- 172.21.15.102:0/2387459146 shutdown_connections 2024-03-18T16:33:48.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.904+0000 7fe959200640 1 -- 172.21.15.102:0/2387459146 wait complete. 2024-03-18T16:33:48.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.907+0000 7fe959200640 1 Processor -- start 2024-03-18T16:33:48.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.907+0000 7fe959200640 1 -- start start 2024-03-18T16:33:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.909+0000 7fe959200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe9540c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.909+0000 7fe959200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95415a6f0 0x7fe9540c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.909+0000 7fe959200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe95415baa0 0x7fe9541591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.909+0000 7fe959200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe9540cd130 con 0x7fe95415baa0 2024-03-18T16:33:48.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.909+0000 7fe959200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9540cce50 con 0x7fe95415a6f0 2024-03-18T16:33:48.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.909+0000 7fe959200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe95406d920 con 0x7fe95415a310 2024-03-18T16:33:48.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.909+0000 7fe953400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe9540c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:48.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.909+0000 7fe952a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95415a6f0 0x7fe9540c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:48.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.910+0000 7fe953400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe9540c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:41860/0 (socket says 172.21.15.102:41860) 2024-03-18T16:33:48.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.910+0000 7fe953400640 1 -- 172.21.15.102:0/2043484503 learned_addr learned my addr 172.21.15.102:0/2043484503 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:48.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.910+0000 7fe953e00640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe95415baa0 0x7fe9541591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:48.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.913+0000 7fe953400640 1 -- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95415a6f0 msgr2=0x7fe9540c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:48.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.913+0000 7fe953400640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95415a6f0 0x7fe9540c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:48.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.913+0000 7fe953400640 1 -- 172.21.15.102:0/2043484503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe95415baa0 msgr2=0x7fe9541591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:48.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.913+0000 7fe953400640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe95415baa0 0x7fe9541591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:48.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.913+0000 7fe953400640 1 -- 172.21.15.102:0/2043484503 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9540c6f80 con 0x7fe95415a310 2024-03-18T16:33:48.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.916+0000 7fe953400640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe9540c9840 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe94402b930 tx=0x7fe9440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:48.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.918+0000 7fe94b400640 1 -- 172.21.15.102:0/2043484503 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944040070 con 0x7fe95415a310 2024-03-18T16:33:48.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.918+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9541f0b90 con 0x7fe95415a310 2024-03-18T16:33:48.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.919+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9541f1150 con 0x7fe95415a310 2024-03-18T16:33:48.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.920+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9541548c0 con 0x7fe95415a310 2024-03-18T16:33:48.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.922+0000 7fe94b400640 1 -- 172.21.15.102:0/2043484503 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9440315f0 con 0x7fe95415a310 2024-03-18T16:33:48.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:48.922+0000 7fe94b400640 1 -- 172.21.15.102:0/2043484503 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944035040 con 0x7fe95415a310 2024-03-18T16:33:49.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.242+0000 7fe94b400640 1 -- 172.21.15.102:0/2043484503 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe94404b060 con 0x7fe95415a310 2024-03-18T16:33:49.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.244+0000 7fe94b400640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe93406bbd0 0x7fe93406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:49.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.244+0000 7fe952a00640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe93406bbd0 0x7fe93406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:49.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.245+0000 7fe952a00640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe93406bbd0 0x7fe93406e090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe954159f20 tx=0x7fe940008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:49.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.250+0000 7fe94b400640 1 -- 172.21.15.102:0/2043484503 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe944078e60 con 0x7fe95415a310 2024-03-18T16:33:49.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.263+0000 7fe94b400640 1 -- 172.21.15.102:0/2043484503 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe9541548c0 con 0x7fe95415a310 2024-03-18T16:33:49.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.518+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe954074fd0 con 0x7fe95415a310 2024-03-18T16:33:49.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.531+0000 7fe94b400640 1 -- 172.21.15.102:0/2043484503 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe94407b4c0 con 0x7fe95415a310 2024-03-18T16:33:49.533 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:49.533 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:49.533 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:49.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe93406bbd0 msgr2=0x7fe93406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:49.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe93406bbd0 0x7fe93406e090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe954159f20 tx=0x7fe940008040 comp rx=0 tx=0).stop 2024-03-18T16:33:49.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 msgr2=0x7fe9540c9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe9540c9840 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe94402b930 tx=0x7fe9440046c0 comp rx=0 tx=0).stop 2024-03-18T16:33:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 shutdown_connections 2024-03-18T16:33:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe93406bbd0 0x7fe93406e090 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe95415baa0 0x7fe9541591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95415a6f0 0x7fe9540c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 --2- 172.21.15.102:0/2043484503 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe95415a310 0x7fe9540c9840 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 >> 172.21.15.102:0/2043484503 conn(0x7fe95408c1a0 msgr2=0x7fe954151e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 shutdown_connections 2024-03-18T16:33:49.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:49.534+0000 7fe959200640 1 -- 172.21.15.102:0/2043484503 wait complete. 2024-03-18T16:33:50.155 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:50.155 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:51.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:51.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.329+0000 7f13cbe00640 1 Processor -- start 2024-03-18T16:33:51.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.330+0000 7f13cbe00640 1 -- start start 2024-03-18T16:33:51.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.331+0000 7f13cbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13c4158ee0 0x7f13c40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:51.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.331+0000 7f13cbe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c415c200 con 0x7f13c41524c0 2024-03-18T16:33:51.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.331+0000 7f13cbe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13c415c4e0 con 0x7f13c4072700 2024-03-18T16:33:51.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.331+0000 7f13cbe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f13c4072b20 con 0x7f13c4158ee0 2024-03-18T16:33:51.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.332+0000 7f13c8a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f13c4072700 0x7f13c415b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:56232/0 (socket says 172.21.15.102:56232) 2024-03-18T16:33:51.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.332+0000 7f13c8a00640 1 -- 172.21.15.102:0/4032980299 learned_addr learned my addr 172.21.15.102:0/4032980299 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:33:51.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.333+0000 7f13c9e00640 1 --2- 172.21.15.102:0/4032980299 >> v2:172.21.15.102:3300/0 conn(0x7f13c4158ee0 0x7f13c40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:51.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.335+0000 7f13c3e00640 1 -- 172.21.15.102:0/4032980299 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 780659470 0 0) 0x7f13c415c4e0 con 0x7f13c4072700 2024-03-18T16:33:51.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.335+0000 7f13c3e00640 1 -- 172.21.15.102:0/4032980299 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13ac003520 con 0x7f13c4072700 2024-03-18T16:33:51.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.338+0000 7f13c9e00640 1 -- 172.21.15.102:0/4032980299 >> v1:172.21.15.154:6789/0 conn(0x7f13c4072700 legacy=0x7f13c415b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:51.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.338+0000 7f13c9e00640 1 -- 172.21.15.102:0/4032980299 >> v1:172.21.15.102:6789/0 conn(0x7f13c41524c0 legacy=0x7f13c406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:51.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.338+0000 7f13c9e00640 1 -- 172.21.15.102:0/4032980299 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c415c7c0 con 0x7f13c4158ee0 2024-03-18T16:33:51.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.349+0000 7f13c9e00640 1 --2- 172.21.15.102:0/4032980299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c4158ee0 0x7f13c40c51e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009d90 tx=0x7f13bc02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08462bca4ceb86c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:51.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.355+0000 7f13c3e00640 1 -- 172.21.15.102:0/4032980299 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc039070 con 0x7f13c4158ee0 2024-03-18T16:33:51.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.355+0000 7f13c3e00640 1 -- 172.21.15.102:0/4032980299 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13bc02cd80 con 0x7f13c4158ee0 2024-03-18T16:33:51.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.355+0000 7f13c3e00640 1 -- 172.21.15.102:0/4032980299 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13bc034660 con 0x7f13c4158ee0 2024-03-18T16:33:51.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.356+0000 7f13cbe00640 1 -- 172.21.15.102:0/4032980299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c4158ee0 msgr2=0x7f13c40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:51.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.356+0000 7f13cbe00640 1 --2- 172.21.15.102:0/4032980299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c4158ee0 0x7f13c40c51e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009d90 tx=0x7f13bc02bfd0 comp rx=0 tx=0).stop 2024-03-18T16:33:51.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.356+0000 7f13cbe00640 1 -- 172.21.15.102:0/4032980299 shutdown_connections 2024-03-18T16:33:51.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.356+0000 7f13cbe00640 1 --2- 172.21.15.102:0/4032980299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c4158ee0 0x7f13c40c51e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:51.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.356+0000 7f13cbe00640 1 -- 172.21.15.102:0/4032980299 >> 172.21.15.102:0/4032980299 conn(0x7f13c40700d0 msgr2=0x7f13c406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:51.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.356+0000 7f13cbe00640 1 -- 172.21.15.102:0/4032980299 shutdown_connections 2024-03-18T16:33:51.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.356+0000 7f13cbe00640 1 -- 172.21.15.102:0/4032980299 wait complete. 2024-03-18T16:33:51.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.359+0000 7f13cbe00640 1 Processor -- start 2024-03-18T16:33:51.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.359+0000 7f13cbe00640 1 -- start start 2024-03-18T16:33:51.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.359+0000 7f13cbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13c4072700 0x7f13c41f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:51.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.360+0000 7f13cbe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13c41524c0 0x7f13c41f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:51.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.360+0000 7f13cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c4158ee0 0x7f13c413c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:51.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.360+0000 7f13cbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41589c0 con 0x7f13c4158ee0 2024-03-18T16:33:51.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.360+0000 7f13cbe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13c41586e0 con 0x7f13c4072700 2024-03-18T16:33:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.360+0000 7f13cbe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f13c41528a0 con 0x7f13c41524c0 2024-03-18T16:33:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.360+0000 7f13c8a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13c41524c0 0x7f13c41f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.360+0000 7f13c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c4158ee0 0x7f13c413c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.361+0000 7f13c9400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13c4072700 0x7f13c41f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:51.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.361+0000 7f13c8a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13c41524c0 0x7f13c41f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53214/0 (socket says 172.21.15.102:53214) 2024-03-18T16:33:51.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.361+0000 7f13c8a00640 1 -- 172.21.15.102:0/3599318366 learned_addr learned my addr 172.21.15.102:0/3599318366 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.366+0000 7f13c8a00640 1 -- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13c4072700 msgr2=0x7f13c41f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.366+0000 7f13c8a00640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13c4072700 0x7f13c41f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.366+0000 7f13c8a00640 1 -- 172.21.15.102:0/3599318366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c4158ee0 msgr2=0x7f13c413c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.366+0000 7f13c8a00640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c4158ee0 0x7f13c413c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:51.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.366+0000 7f13c8a00640 1 -- 172.21.15.102:0/3599318366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c4141630 con 0x7f13c41524c0 2024-03-18T16:33:51.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.368+0000 7f13c8a00640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13c41524c0 0x7f13c41f3e40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c430 tx=0x7f13b000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:51.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.387+0000 7f13c1600640 1 -- 172.21.15.102:0/3599318366 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b0017070 con 0x7f13c41524c0 2024-03-18T16:33:51.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.387+0000 7f13c1600640 1 -- 172.21.15.102:0/3599318366 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b0005ce0 con 0x7f13c41524c0 2024-03-18T16:33:51.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.387+0000 7f13c1600640 1 -- 172.21.15.102:0/3599318366 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b00072f0 con 0x7f13c41524c0 2024-03-18T16:33:51.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.387+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c4146f50 con 0x7f13c41524c0 2024-03-18T16:33:51.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.387+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c4146990 con 0x7f13c41524c0 2024-03-18T16:33:51.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.388+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13c40cb1f0 con 0x7f13c41524c0 2024-03-18T16:33:51.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.654+0000 7f13c1600640 1 -- 172.21.15.102:0/3599318366 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f13b001a070 con 0x7f13c41524c0 2024-03-18T16:33:51.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.656+0000 7f13c1600640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13a406bc20 0x7f13a406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:51.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.657+0000 7f13c1600640 1 -- 172.21.15.102:0/3599318366 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f13b008a610 con 0x7f13c41524c0 2024-03-18T16:33:51.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.657+0000 7f13c9400640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13a406bc20 0x7f13a406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:51.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.658+0000 7f13c9400640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13a406bc20 0x7f13a406e0e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f13b8001780 tx=0x7f13b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:51.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.667+0000 7f13c1600640 1 -- 172.21.15.102:0/3599318366 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f13c40cb1f0 con 0x7f13c41524c0 2024-03-18T16:33:51.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.899+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13c4074fd0 con 0x7f13c41524c0 2024-03-18T16:33:51.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.911+0000 7f13c1600640 1 -- 172.21.15.102:0/3599318366 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f13b0021020 con 0x7f13c41524c0 2024-03-18T16:33:51.946 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:51.946 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:51.948 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:51.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13a406bc20 msgr2=0x7f13a406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:51.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13a406bc20 0x7f13a406e0e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f13b8001780 tx=0x7f13b8009040 comp rx=0 tx=0).stop 2024-03-18T16:33:51.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13c41524c0 msgr2=0x7f13c41f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:51.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13c41524c0 0x7f13c41f3e40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c430 tx=0x7f13b000c8f0 comp rx=0 tx=0).stop 2024-03-18T16:33:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 shutdown_connections 2024-03-18T16:33:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13a406bc20 0x7f13a406e0e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13c4158ee0 0x7f13c413c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13c41524c0 0x7f13c41f3e40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 --2- 172.21.15.102:0/3599318366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13c4072700 0x7f13c41f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 >> 172.21.15.102:0/3599318366 conn(0x7f13c40700d0 msgr2=0x7f13c40c6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 shutdown_connections 2024-03-18T16:33:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:51.914+0000 7f13cbe00640 1 -- 172.21.15.102:0/3599318366 wait complete. 2024-03-18T16:33:52.555 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:52.555 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:53.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:53.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.731+0000 7fe300e00640 1 Processor -- start 2024-03-18T16:33:53.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.731+0000 7fe300e00640 1 -- start start 2024-03-18T16:33:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.733+0000 7fe300e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cfe70 0x7fe2fc15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.733+0000 7fe300e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc0cabd0 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.733+0000 7fe300e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2fc0caea0 con 0x7fe2fc0d0250 2024-03-18T16:33:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.733+0000 7fe300e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe2fc0cb180 con 0x7fe2fc0cfe70 2024-03-18T16:33:53.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.734+0000 7fe2fb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cfe70 0x7fe2fc15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:53.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.734+0000 7fe2faa00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe2fc0d0250 0x7fe2fc0c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:56238/0 (socket says 172.21.15.102:56238) 2024-03-18T16:33:53.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.734+0000 7fe2faa00640 1 -- 172.21.15.102:0/942047032 learned_addr learned my addr 172.21.15.102:0/942047032 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:33:53.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.736+0000 7fe2fa000640 1 -- 172.21.15.102:0/942047032 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2043894255 0 0) 0x7fe2fc0caea0 con 0x7fe2fc0d0250 2024-03-18T16:33:53.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.736+0000 7fe2fa000640 1 -- 172.21.15.102:0/942047032 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2e0003520 con 0x7fe2fc0d0250 2024-03-18T16:33:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.737+0000 7fe2fa000640 1 -- 172.21.15.102:0/942047032 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1482547250 0 0) 0x7fe2fc0cabd0 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.737+0000 7fe2fa000640 1 -- 172.21.15.102:0/942047032 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2fc0caea0 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.737+0000 7fe2fb400640 1 -- 172.21.15.102:0/942047032 >> v1:172.21.15.154:6789/0 conn(0x7fe2fc0d0250 legacy=0x7fe2fc0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.737+0000 7fe2fb400640 1 -- 172.21.15.102:0/942047032 >> v1:172.21.15.102:6789/0 conn(0x7fe2fc0cc2f0 legacy=0x7fe2fc0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.738+0000 7fe2fb400640 1 -- 172.21.15.102:0/942047032 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2fc0cb460 con 0x7fe2fc0cfe70 2024-03-18T16:33:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.741+0000 7fe2fb400640 1 --2- 172.21.15.102:0/942047032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cfe70 0x7fe2fc15ae40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec0097d0 tx=0x7fe2ec02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=658919b081c86eea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.745+0000 7fe2fa000640 1 -- 172.21.15.102:0/942047032 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec034070 con 0x7fe2fc0cfe70 2024-03-18T16:33:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.745+0000 7fe2fa000640 1 -- 172.21.15.102:0/942047032 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2ec030880 con 0x7fe2fc0cfe70 2024-03-18T16:33:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.745+0000 7fe2fa000640 1 -- 172.21.15.102:0/942047032 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec030c50 con 0x7fe2fc0cfe70 2024-03-18T16:33:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.746+0000 7fe300e00640 1 -- 172.21.15.102:0/942047032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cfe70 msgr2=0x7fe2fc15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.746+0000 7fe300e00640 1 --2- 172.21.15.102:0/942047032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cfe70 0x7fe2fc15ae40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec0097d0 tx=0x7fe2ec02b9c0 comp rx=0 tx=0).stop 2024-03-18T16:33:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.746+0000 7fe300e00640 1 -- 172.21.15.102:0/942047032 shutdown_connections 2024-03-18T16:33:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.746+0000 7fe300e00640 1 --2- 172.21.15.102:0/942047032 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cfe70 0x7fe2fc15ae40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.746+0000 7fe300e00640 1 -- 172.21.15.102:0/942047032 >> 172.21.15.102:0/942047032 conn(0x7fe2fc08c3e0 msgr2=0x7fe2fc06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.746+0000 7fe300e00640 1 -- 172.21.15.102:0/942047032 shutdown_connections 2024-03-18T16:33:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.746+0000 7fe300e00640 1 -- 172.21.15.102:0/942047032 wait complete. 2024-03-18T16:33:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.748+0000 7fe300e00640 1 Processor -- start 2024-03-18T16:33:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.748+0000 7fe300e00640 1 -- start start 2024-03-18T16:33:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.749+0000 7fe300e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cc2f0 0x7fe2fc1ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.749+0000 7fe300e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2fc0cfe70 0x7fe2fc0ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.749+0000 7fe300e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2fc0d0250 0x7fe2fc0ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.749+0000 7fe300e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe2fc0cbd00 con 0x7fe2fc0d0250 2024-03-18T16:33:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.749+0000 7fe300e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2fc0cba20 con 0x7fe2fc0cfe70 2024-03-18T16:33:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.749+0000 7fe300e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe2fc0c8330 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.749+0000 7fe2fb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cc2f0 0x7fe2fc1ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.749+0000 7fe2faa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2fc0cfe70 0x7fe2fc0ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.749+0000 7fe2fbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2fc0d0250 0x7fe2fc0ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.750+0000 7fe2fb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cc2f0 0x7fe2fc1ef940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53232/0 (socket says 172.21.15.102:53232) 2024-03-18T16:33:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.750+0000 7fe2fb400640 1 -- 172.21.15.102:0/1248077898 learned_addr learned my addr 172.21.15.102:0/1248077898 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:53.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.755+0000 7fe2fb400640 1 -- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2fc0cfe70 msgr2=0x7fe2fc0ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:53.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.755+0000 7fe2fb400640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2fc0cfe70 0x7fe2fc0ae030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:53.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.755+0000 7fe2fb400640 1 -- 172.21.15.102:0/1248077898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2fc0d0250 msgr2=0x7fe2fc0ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:53.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.755+0000 7fe2fb400640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2fc0d0250 0x7fe2fc0ad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:53.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.755+0000 7fe2fb400640 1 -- 172.21.15.102:0/1248077898 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2fc0ae790 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.758+0000 7fe2fb400640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cc2f0 0x7fe2fc1ef940 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec02bfd0 tx=0x7fe2ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.763+0000 7fe2f3400640 1 -- 172.21.15.102:0/1248077898 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec034040 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.763+0000 7fe2f3400640 1 -- 172.21.15.102:0/1248077898 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2ec030680 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.763+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2fc0aea70 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.763+0000 7fe2f3400640 1 -- 172.21.15.102:0/1248077898 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec006e10 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.763+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2fc0af030 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.764+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2fc074fd0 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.795+0000 7fe2f3400640 1 -- 172.21.15.102:0/1248077898 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe2ec037070 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.796+0000 7fe2f3400640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe2dc06bc50 0x7fe2dc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:53.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.797+0000 7fe2faa00640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe2dc06bc50 0x7fe2dc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:53.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.798+0000 7fe2faa00640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe2dc06bc50 0x7fe2dc06e110 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc1ea280 tx=0x7fe2e8007d70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:53.801 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:33:53.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:53.800+0000 7fe2f3400640 1 -- 172.21.15.102:0/1248077898 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe2ec030830 con 0x7fe2fc0cc2f0 2024-03-18T16:33:53.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:33:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.058+0000 7fe2f3400640 1 -- 172.21.15.102:0/1248077898 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe2ec07bb70 con 0x7fe2fc0cc2f0 2024-03-18T16:33:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.316+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2fc06ca60 con 0x7fe2fc0cc2f0 2024-03-18T16:33:54.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.339+0000 7fe2f3400640 1 -- 172.21.15.102:0/1248077898 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe2ec07fc10 con 0x7fe2fc0cc2f0 2024-03-18T16:33:54.341 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:54.341 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:54.342 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe2dc06bc50 msgr2=0x7fe2dc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe2dc06bc50 0x7fe2dc06e110 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc1ea280 tx=0x7fe2e8007d70 comp rx=0 tx=0).stop 2024-03-18T16:33:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cc2f0 msgr2=0x7fe2fc1ef940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cc2f0 0x7fe2fc1ef940 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec02bfd0 tx=0x7fe2ec0046c0 comp rx=0 tx=0).stop 2024-03-18T16:33:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 shutdown_connections 2024-03-18T16:33:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe2dc06bc50 0x7fe2dc06e110 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe2fc0d0250 0x7fe2fc0ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:54.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2fc0cfe70 0x7fe2fc0ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 --2- 172.21.15.102:0/1248077898 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe2fc0cc2f0 0x7fe2fc1ef940 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 >> 172.21.15.102:0/1248077898 conn(0x7fe2fc08c3e0 msgr2=0x7fe2fc0c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 shutdown_connections 2024-03-18T16:33:54.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:54.342+0000 7fe300e00640 1 -- 172.21.15.102:0/1248077898 wait complete. 2024-03-18T16:33:54.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:54.964 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:55.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.140+0000 7f9ec8c00640 1 Processor -- start 2024-03-18T16:33:56.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.140+0000 7f9ec8c00640 1 -- start start 2024-03-18T16:33:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.141+0000 7f9ec8c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ec40cfe70 0x7f9ec415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.141+0000 7f9ec8c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9ec40d0250 0x7f9ec40c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.142+0000 7f9ec8c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ec40cabd0 con 0x7f9ec40cc2f0 2024-03-18T16:33:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.142+0000 7f9ec8c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9ec40caea0 con 0x7f9ec40cfe70 2024-03-18T16:33:56.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.142+0000 7f9ec8c00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9ec40cb180 con 0x7f9ec40d0250 2024-03-18T16:33:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.142+0000 7f9ec3400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ec40cfe70 0x7f9ec415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.143+0000 7f9ec2a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9ec40d0250 0x7f9ec40c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.143+0000 7f9ec3400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9ec40cfe70 0x7f9ec415ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:60414/0 (socket says 172.21.15.102:60414) 2024-03-18T16:33:56.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.143+0000 7f9ec3400640 1 -- 172.21.15.102:0/129091640 learned_addr learned my addr 172.21.15.102:0/129091640 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.145+0000 7f9ec3400640 1 -- 172.21.15.102:0/129091640 >> v2:172.21.15.154:3300/0 conn(0x7f9ec40d0250 msgr2=0x7f9ec40c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.145+0000 7f9ec3400640 1 --2- 172.21.15.102:0/129091640 >> v2:172.21.15.154:3300/0 conn(0x7f9ec40d0250 0x7f9ec40c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.145+0000 7f9ec3400640 1 -- 172.21.15.102:0/129091640 >> v1:172.21.15.154:6789/0 conn(0x7f9ec40cc2f0 legacy=0x7f9ec40cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.145+0000 7f9ec3400640 1 -- 172.21.15.102:0/129091640 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec40cb460 con 0x7f9ec40cfe70 2024-03-18T16:33:56.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.148+0000 7f9ec3400640 1 --2- 172.21.15.102:0/129091640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ec40cfe70 0x7f9ec415ae40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9ec40c6370 tx=0x7f9eb402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde4c89136c7e2a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:56.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.152+0000 7f9ec2000640 1 -- 172.21.15.102:0/129091640 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4034070 con 0x7f9ec40cfe70 2024-03-18T16:33:56.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.152+0000 7f9ec2000640 1 -- 172.21.15.102:0/129091640 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eb4030860 con 0x7f9ec40cfe70 2024-03-18T16:33:56.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.152+0000 7f9ec2000640 1 -- 172.21.15.102:0/129091640 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb4030c30 con 0x7f9ec40cfe70 2024-03-18T16:33:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.153+0000 7f9ec8c00640 1 -- 172.21.15.102:0/129091640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ec40cfe70 msgr2=0x7f9ec415ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.153+0000 7f9ec8c00640 1 --2- 172.21.15.102:0/129091640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ec40cfe70 0x7f9ec415ae40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9ec40c6370 tx=0x7f9eb402b9a0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.154+0000 7f9ec8c00640 1 -- 172.21.15.102:0/129091640 shutdown_connections 2024-03-18T16:33:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.154+0000 7f9ec8c00640 1 --2- 172.21.15.102:0/129091640 >> v2:172.21.15.154:3300/0 conn(0x7f9ec40d0250 0x7f9ec40c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.154+0000 7f9ec8c00640 1 --2- 172.21.15.102:0/129091640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ec40cfe70 0x7f9ec415ae40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.154+0000 7f9ec8c00640 1 -- 172.21.15.102:0/129091640 >> 172.21.15.102:0/129091640 conn(0x7f9ec408c3e0 msgr2=0x7f9ec406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.154+0000 7f9ec8c00640 1 -- 172.21.15.102:0/129091640 shutdown_connections 2024-03-18T16:33:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.154+0000 7f9ec8c00640 1 -- 172.21.15.102:0/129091640 wait complete. 2024-03-18T16:33:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.156+0000 7f9ec8c00640 1 Processor -- start 2024-03-18T16:33:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.157+0000 7f9ec8c00640 1 -- start start 2024-03-18T16:33:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.158+0000 7f9ec8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ec40cc2f0 0x7f9ec4158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.158+0000 7f9ec8c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec40cfe70 0x7f9ec4159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.158+0000 7f9ec8c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec40d0250 0x7f9ec4150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.158+0000 7f9ec8c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec40cbd00 con 0x7f9ec40cc2f0 2024-03-18T16:33:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.158+0000 7f9ec8c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec40cba20 con 0x7f9ec40cfe70 2024-03-18T16:33:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.158+0000 7f9ec8c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9ec40c6b20 con 0x7f9ec40d0250 2024-03-18T16:33:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.159+0000 7f9ec3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ec40cc2f0 0x7f9ec4158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.159+0000 7f9ec3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec40d0250 0x7f9ec4150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.159+0000 7f9ec2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec40cfe70 0x7f9ec4159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.159+0000 7f9ec3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec40d0250 0x7f9ec4150d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53238/0 (socket says 172.21.15.102:53238) 2024-03-18T16:33:56.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.159+0000 7f9ec3e00640 1 -- 172.21.15.102:0/1409631312 learned_addr learned my addr 172.21.15.102:0/1409631312 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.163+0000 7f9ec3e00640 1 -- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec40cfe70 msgr2=0x7f9ec4159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.163+0000 7f9ec3e00640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec40cfe70 0x7f9ec4159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.163+0000 7f9ec3e00640 1 -- 172.21.15.102:0/1409631312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ec40cc2f0 msgr2=0x7f9ec4158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.163+0000 7f9ec3e00640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ec40cc2f0 0x7f9ec4158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.163+0000 7f9ec3e00640 1 -- 172.21.15.102:0/1409631312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec41e16e0 con 0x7f9ec40d0250 2024-03-18T16:33:56.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.165+0000 7f9ec3e00640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec40d0250 0x7f9ec4150d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00b9a0 tx=0x7f9ebc00be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.169+0000 7f9ebb400640 1 -- 172.21.15.102:0/1409631312 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc019070 con 0x7f9ec40d0250 2024-03-18T16:33:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.169+0000 7f9ebb400640 1 -- 172.21.15.102:0/1409631312 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ebc00cba0 con 0x7f9ec40d0250 2024-03-18T16:33:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.169+0000 7f9ebb400640 1 -- 172.21.15.102:0/1409631312 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc014440 con 0x7f9ec40d0250 2024-03-18T16:33:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.169+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ec41e1980 con 0x7f9ec40d0250 2024-03-18T16:33:56.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.169+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ec4158710 con 0x7f9ec40d0250 2024-03-18T16:33:56.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.171+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ec41e1ea0 con 0x7f9ec40d0250 2024-03-18T16:33:56.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.190+0000 7f9ebb400640 1 -- 172.21.15.102:0/1409631312 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9ebc01c070 con 0x7f9ec40d0250 2024-03-18T16:33:56.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.191+0000 7f9ebb400640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9ea406bc20 0x7f9ea406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:56.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.192+0000 7f9ec3400640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9ea406bc20 0x7f9ea406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:56.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.193+0000 7f9ec3400640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9ea406bc20 0x7f9ea406e0e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4036000 tx=0x7f9eb4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.449+0000 7f9ebb400640 1 -- 172.21.15.102:0/1409631312 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9ebc056cc0 con 0x7f9ec40d0250 2024-03-18T16:33:56.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.468+0000 7f9ebb400640 1 -- 172.21.15.102:0/1409631312 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9ebc058c20 con 0x7f9ec40d0250 2024-03-18T16:33:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.828+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ec4158500 con 0x7f9ec40d0250 2024-03-18T16:33:56.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.845+0000 7f9ebb400640 1 -- 172.21.15.102:0/1409631312 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f9ebc05ccc0 con 0x7f9ec40d0250 2024-03-18T16:33:56.847 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:56.847 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:56.847 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:56.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9ea406bc20 msgr2=0x7f9ea406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:56.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9ea406bc20 0x7f9ea406e0e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9eb4036000 tx=0x7f9eb4008030 comp rx=0 tx=0).stop 2024-03-18T16:33:56.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec40d0250 msgr2=0x7f9ec4150d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:56.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec40d0250 0x7f9ec4150d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00b9a0 tx=0x7f9ebc00be60 comp rx=0 tx=0).stop 2024-03-18T16:33:56.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 shutdown_connections 2024-03-18T16:33:56.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9ea406bc20 0x7f9ea406e0e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9ec40d0250 0x7f9ec4150d80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ec40cfe70 0x7f9ec4159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 --2- 172.21.15.102:0/1409631312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ec40cc2f0 0x7f9ec4158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:56.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.847+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 >> 172.21.15.102:0/1409631312 conn(0x7f9ec408c3e0 msgr2=0x7f9ec40c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:56.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.848+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 shutdown_connections 2024-03-18T16:33:56.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:56.848+0000 7f9ec8c00640 1 -- 172.21.15.102:0/1409631312 wait complete. 2024-03-18T16:33:57.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:57.469 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:33:58.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:33:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.644+0000 7fc9d7200640 1 Processor -- start 2024-03-18T16:33:58.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.644+0000 7fc9d7200640 1 -- start start 2024-03-18T16:33:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.645+0000 7fc9d7200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc9d815a6f0 0x7fc9d815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.645+0000 7fc9d7200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9d815baa0 0x7fc9d815bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.645+0000 7fc9d7200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9d80ca500 con 0x7fc9d815a310 2024-03-18T16:33:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.645+0000 7fc9d7200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d80cd410 con 0x7fc9d815a6f0 2024-03-18T16:33:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.645+0000 7fc9d7200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d815b080 con 0x7fc9d815baa0 2024-03-18T16:33:58.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.646+0000 7fc9d5e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc9d815a310 0x7fc9d806d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:56264/0 (socket says 172.21.15.102:56264) 2024-03-18T16:33:58.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.646+0000 7fc9d5e00640 1 -- 172.21.15.102:0/4109240579 learned_addr learned my addr 172.21.15.102:0/4109240579 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:33:58.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.647+0000 7fc9d6800640 1 --2- 172.21.15.102:0/4109240579 >> v2:172.21.15.154:3300/0 conn(0x7fc9d815baa0 0x7fc9d815bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:58.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.650+0000 7fc9d5400640 1 --2- 172.21.15.102:0/4109240579 >> v2:172.21.15.102:3300/0 conn(0x7fc9d815a6f0 0x7fc9d815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.651+0000 7fc9d6800640 1 -- 172.21.15.102:0/4109240579 >> v2:172.21.15.102:3300/0 conn(0x7fc9d815a6f0 msgr2=0x7fc9d815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:33:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.651+0000 7fc9d6800640 1 --2- 172.21.15.102:0/4109240579 >> v2:172.21.15.102:3300/0 conn(0x7fc9d815a6f0 0x7fc9d815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.651+0000 7fc9d6800640 1 -- 172.21.15.102:0/4109240579 >> v1:172.21.15.154:6789/0 conn(0x7fc9d815a310 legacy=0x7fc9d806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:58.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.651+0000 7fc9d6800640 1 -- 172.21.15.102:0/4109240579 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d815c430 con 0x7fc9d815baa0 2024-03-18T16:33:58.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.654+0000 7fc9d6800640 1 --2- 172.21.15.102:0/4109240579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d815baa0 0x7fc9d815bef0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8067f40 tx=0x7fc9d002bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=733070de95c43a37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:58.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.658+0000 7fc9d4a00640 1 -- 172.21.15.102:0/4109240579 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0039070 con 0x7fc9d815baa0 2024-03-18T16:33:58.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.714+0000 7fc9d4a00640 1 -- 172.21.15.102:0/4109240579 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9d002cc70 con 0x7fc9d815baa0 2024-03-18T16:33:58.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.715+0000 7fc9d4a00640 1 -- 172.21.15.102:0/4109240579 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0034660 con 0x7fc9d815baa0 2024-03-18T16:33:58.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.715+0000 7fc9d7200640 1 -- 172.21.15.102:0/4109240579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d815baa0 msgr2=0x7fc9d815bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:58.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.715+0000 7fc9d7200640 1 --2- 172.21.15.102:0/4109240579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d815baa0 0x7fc9d815bef0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8067f40 tx=0x7fc9d002bec0 comp rx=0 tx=0).stop 2024-03-18T16:33:58.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.715+0000 7fc9d7200640 1 -- 172.21.15.102:0/4109240579 shutdown_connections 2024-03-18T16:33:58.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.716+0000 7fc9d7200640 1 --2- 172.21.15.102:0/4109240579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d815baa0 0x7fc9d815bef0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:58.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.716+0000 7fc9d7200640 1 --2- 172.21.15.102:0/4109240579 >> v2:172.21.15.102:3300/0 conn(0x7fc9d815a6f0 0x7fc9d815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:58.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.716+0000 7fc9d7200640 1 -- 172.21.15.102:0/4109240579 >> 172.21.15.102:0/4109240579 conn(0x7fc9d808c1a0 msgr2=0x7fc9d8066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:58.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.716+0000 7fc9d7200640 1 -- 172.21.15.102:0/4109240579 shutdown_connections 2024-03-18T16:33:58.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.716+0000 7fc9d7200640 1 -- 172.21.15.102:0/4109240579 wait complete. 2024-03-18T16:33:58.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.718+0000 7fc9d7200640 1 Processor -- start 2024-03-18T16:33:58.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.718+0000 7fc9d7200640 1 -- start start 2024-03-18T16:33:58.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.720+0000 7fc9d7200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d815a310 0x7fc9d80c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:58.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.720+0000 7fc9d7200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d815a6f0 0x7fc9d8159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:58.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.720+0000 7fc9d7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9d815baa0 0x7fc9d80c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:58.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.720+0000 7fc9d7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d80cce50 con 0x7fc9d815baa0 2024-03-18T16:33:58.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.720+0000 7fc9d7200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d80caa80 con 0x7fc9d815a310 2024-03-18T16:33:58.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.720+0000 7fc9d7200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d80cd130 con 0x7fc9d815a6f0 2024-03-18T16:33:58.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.721+0000 7fc9d5e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d815a310 0x7fc9d80c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:58.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.721+0000 7fc9d5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d815a6f0 0x7fc9d8159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:58.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.721+0000 7fc9d5400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d815a6f0 0x7fc9d8159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53254/0 (socket says 172.21.15.102:53254) 2024-03-18T16:33:58.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.721+0000 7fc9d5400640 1 -- 172.21.15.102:0/822406007 learned_addr learned my addr 172.21.15.102:0/822406007 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:33:58.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.723+0000 7fc9d6800640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9d815baa0 0x7fc9d80c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:58.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.725+0000 7fc9d5400640 1 -- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d815a310 msgr2=0x7fc9d80c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:58.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.725+0000 7fc9d5400640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d815a310 0x7fc9d80c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:58.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.725+0000 7fc9d5400640 1 -- 172.21.15.102:0/822406007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9d815baa0 msgr2=0x7fc9d80c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:58.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.725+0000 7fc9d5400640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9d815baa0 0x7fc9d80c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:58.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.725+0000 7fc9d5400640 1 -- 172.21.15.102:0/822406007 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d8156ef0 con 0x7fc9d815a6f0 2024-03-18T16:33:58.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.727+0000 7fc9d5400640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d815a6f0 0x7fc9d8159230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400d830 tx=0x7fc9c400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:58.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.731+0000 7fc9ce000640 1 -- 172.21.15.102:0/822406007 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400b6c0 con 0x7fc9d815a6f0 2024-03-18T16:33:58.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.731+0000 7fc9ce000640 1 -- 172.21.15.102:0/822406007 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c4019070 con 0x7fc9d815a6f0 2024-03-18T16:33:58.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.731+0000 7fc9ce000640 1 -- 172.21.15.102:0/822406007 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400ba80 con 0x7fc9d815a6f0 2024-03-18T16:33:58.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.731+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d81f0440 con 0x7fc9d815a6f0 2024-03-18T16:33:58.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.731+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d81f09c0 con 0x7fc9d815a6f0 2024-03-18T16:33:58.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.732+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d80cac90 con 0x7fc9d815a6f0 2024-03-18T16:33:58.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.752+0000 7fc9ce000640 1 -- 172.21.15.102:0/822406007 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc9c401e460 con 0x7fc9d815a6f0 2024-03-18T16:33:58.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.753+0000 7fc9ce000640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9b806bbd0 0x7fc9b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:33:58.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.753+0000 7fc9d5e00640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9b806bbd0 0x7fc9b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:33:58.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.754+0000 7fc9d5e00640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9b806bbd0 0x7fc9b806e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000f60 tx=0x7fc9c8007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:33:58.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:58.755+0000 7fc9ce000640 1 -- 172.21.15.102:0/822406007 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fc9c4058080 con 0x7fc9d815a6f0 2024-03-18T16:33:59.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.006+0000 7fc9ce000640 1 -- 172.21.15.102:0/822406007 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc9d80cac90 con 0x7fc9d815a6f0 2024-03-18T16:33:59.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.249+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9d8074fd0 con 0x7fc9d815a6f0 2024-03-18T16:33:59.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.262+0000 7fc9ce000640 1 -- 172.21.15.102:0/822406007 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fc9c4059fe0 con 0x7fc9d815a6f0 2024-03-18T16:33:59.264 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:33:59.264 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:33:59.265 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9b806bbd0 msgr2=0x7fc9b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9b806bbd0 0x7fc9b806e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000f60 tx=0x7fc9c8007e80 comp rx=0 tx=0).stop 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d815a6f0 msgr2=0x7fc9d8159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d815a6f0 0x7fc9d8159230 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400d830 tx=0x7fc9c400dcf0 comp rx=0 tx=0).stop 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 shutdown_connections 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc9b806bbd0 0x7fc9b806e090 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9d815baa0 0x7fc9d80c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc9d815a6f0 0x7fc9d8159230 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 --2- 172.21.15.102:0/822406007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9d815a310 0x7fc9d80c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 >> 172.21.15.102:0/822406007 conn(0x7fc9d808c1a0 msgr2=0x7fc9d8153b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 shutdown_connections 2024-03-18T16:33:59.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:33:59.266+0000 7fc9d7200640 1 -- 172.21.15.102:0/822406007 wait complete. 2024-03-18T16:33:59.895 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:33:59.896 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:00.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:01.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.097+0000 7f0598a00640 1 Processor -- start 2024-03-18T16:34:01.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.097+0000 7f0598a00640 1 -- start start 2024-03-18T16:34:01.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.098+0000 7f0598a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 0x7f0594157a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:01.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.098+0000 7f0598a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f059415b1c0 0x7f059406de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:01.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.098+0000 7f0598a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05940c75b0 con 0x7f05940733b0 2024-03-18T16:34:01.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.098+0000 7f0598a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f059414deb0 con 0x7f059415b1c0 2024-03-18T16:34:01.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.098+0000 7f0598a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f059415ad30 con 0x7f059414dad0 2024-03-18T16:34:01.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.099+0000 7f0593200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f059415b1c0 0x7f059406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:01.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.099+0000 7f058be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f05940733b0 0x7f05940c6230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:43456/0 (socket says 172.21.15.102:43456) 2024-03-18T16:34:01.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.099+0000 7f058be00640 1 -- 172.21.15.102:0/396388300 learned_addr learned my addr 172.21.15.102:0/396388300 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:01.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.099+0000 7f0592800640 1 --2- 172.21.15.102:0/396388300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 0x7f0594157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:01.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.102+0000 7f0593200640 1 -- 172.21.15.102:0/396388300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 msgr2=0x7f0594157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:01.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.102+0000 7f0593200640 1 --2- 172.21.15.102:0/396388300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 0x7f0594157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.102+0000 7f0593200640 1 -- 172.21.15.102:0/396388300 >> v1:172.21.15.102:6789/0 conn(0x7f05940733b0 legacy=0x7f05940c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:01.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.102+0000 7f0593200640 1 -- 172.21.15.102:0/396388300 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05940c7850 con 0x7f059415b1c0 2024-03-18T16:34:01.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.104+0000 7f0593200640 1 --2- 172.21.15.102:0/396388300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f059415b1c0 0x7f059406de50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f058c009d90 tx=0x7f058c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9b8a9992e33c7d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:01.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.108+0000 7f0591e00640 1 -- 172.21.15.102:0/396388300 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c039070 con 0x7f059415b1c0 2024-03-18T16:34:01.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.108+0000 7f0591e00640 1 -- 172.21.15.102:0/396388300 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f058c02cc60 con 0x7f059415b1c0 2024-03-18T16:34:01.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.108+0000 7f0591e00640 1 -- 172.21.15.102:0/396388300 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c034660 con 0x7f059415b1c0 2024-03-18T16:34:01.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.109+0000 7f0598a00640 1 -- 172.21.15.102:0/396388300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f059415b1c0 msgr2=0x7f059406de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:01.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.109+0000 7f0598a00640 1 --2- 172.21.15.102:0/396388300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f059415b1c0 0x7f059406de50 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f058c009d90 tx=0x7f058c02beb0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.109+0000 7f0598a00640 1 -- 172.21.15.102:0/396388300 shutdown_connections 2024-03-18T16:34:01.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.109+0000 7f0598a00640 1 --2- 172.21.15.102:0/396388300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f059415b1c0 0x7f059406de50 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.109+0000 7f0598a00640 1 --2- 172.21.15.102:0/396388300 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 0x7f0594157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.109+0000 7f0598a00640 1 -- 172.21.15.102:0/396388300 >> 172.21.15.102:0/396388300 conn(0x7f05940727e0 msgr2=0x7f0594072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:01.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.110+0000 7f0598a00640 1 -- 172.21.15.102:0/396388300 shutdown_connections 2024-03-18T16:34:01.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.110+0000 7f0598a00640 1 -- 172.21.15.102:0/396388300 wait complete. 2024-03-18T16:34:01.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.111+0000 7f0598a00640 1 Processor -- start 2024-03-18T16:34:01.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.112+0000 7f0598a00640 1 -- start start 2024-03-18T16:34:01.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.112+0000 7f0598a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05940733b0 0x7f05940ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:01.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.112+0000 7f0598a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 0x7f05941f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:01.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.112+0000 7f0598a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f059415b1c0 0x7f0594145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:01.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.112+0000 7f0598a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f059415a6e0 con 0x7f059415b1c0 2024-03-18T16:34:01.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.112+0000 7f0598a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f059406e390 con 0x7f05940733b0 2024-03-18T16:34:01.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.113+0000 7f0598a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f059415a9c0 con 0x7f059414dad0 2024-03-18T16:34:01.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.113+0000 7f0592800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05940733b0 0x7f05940ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:01.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.113+0000 7f058be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 0x7f05941f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:01.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.113+0000 7f0593200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f059415b1c0 0x7f0594145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:01.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.114+0000 7f0592800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05940733b0 0x7f05940ca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:52172/0 (socket says 172.21.15.102:52172) 2024-03-18T16:34:01.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.114+0000 7f0592800640 1 -- 172.21.15.102:0/1341384534 learned_addr learned my addr 172.21.15.102:0/1341384534 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:01.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.117+0000 7f0592800640 1 -- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 msgr2=0x7f05941f35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:01.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.117+0000 7f0592800640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 0x7f05941f35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.117+0000 7f0592800640 1 -- 172.21.15.102:0/1341384534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f059415b1c0 msgr2=0x7f0594145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:01.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.117+0000 7f0592800640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f059415b1c0 0x7f0594145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.117+0000 7f0592800640 1 -- 172.21.15.102:0/1341384534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0594151a30 con 0x7f05940733b0 2024-03-18T16:34:01.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.119+0000 7f0592800640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05940733b0 0x7f05940ca750 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f058400e9b0 tx=0x7f058400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:01.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.123+0000 7f058aa00640 1 -- 172.21.15.102:0/1341384534 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584017070 con 0x7f05940733b0 2024-03-18T16:34:01.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.123+0000 7f058aa00640 1 -- 172.21.15.102:0/1341384534 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f058400c9a0 con 0x7f05940733b0 2024-03-18T16:34:01.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.123+0000 7f058aa00640 1 -- 172.21.15.102:0/1341384534 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058400cd90 con 0x7f05940733b0 2024-03-18T16:34:01.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.123+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0594151d10 con 0x7f05940733b0 2024-03-18T16:34:01.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.123+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0594152d90 con 0x7f05940733b0 2024-03-18T16:34:01.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.125+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0594074fd0 con 0x7f05940733b0 2024-03-18T16:34:01.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.143+0000 7f058aa00640 1 -- 172.21.15.102:0/1341384534 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f058401a070 con 0x7f05940733b0 2024-03-18T16:34:01.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.144+0000 7f058aa00640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f057006bc50 0x7f057006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:01.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.144+0000 7f058be00640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f057006bc50 0x7f057006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:01.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.145+0000 7f058be00640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f057006bc50 0x7f057006e110 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0594068db0 tx=0x7f057c007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:01.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.147+0000 7f058aa00640 1 -- 172.21.15.102:0/1341384534 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f0584057650 con 0x7f05940733b0 2024-03-18T16:34:01.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.411+0000 7f058aa00640 1 -- 172.21.15.102:0/1341384534 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f05840595b0 con 0x7f05940733b0 2024-03-18T16:34:01.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.651+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05940c7390 con 0x7f05940733b0 2024-03-18T16:34:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.664+0000 7f058aa00640 1 -- 172.21.15.102:0/1341384534 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f058405d650 con 0x7f05940733b0 2024-03-18T16:34:01.666 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:01.666 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:01.666 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.666+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f057006bc50 msgr2=0x7f057006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.666+0000 7f0598a00640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f057006bc50 0x7f057006e110 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0594068db0 tx=0x7f057c007e80 comp rx=0 tx=0).stop 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.666+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05940733b0 msgr2=0x7f05940ca750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.666+0000 7f0598a00640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05940733b0 0x7f05940ca750 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f058400e9b0 tx=0x7f058400ee70 comp rx=0 tx=0).stop 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.667+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 shutdown_connections 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.667+0000 7f0598a00640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f057006bc50 0x7f057006e110 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.667+0000 7f0598a00640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f059415b1c0 0x7f0594145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.667+0000 7f0598a00640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f059414dad0 0x7f05941f35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.667+0000 7f0598a00640 1 --2- 172.21.15.102:0/1341384534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05940733b0 0x7f05940ca750 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.667+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 >> 172.21.15.102:0/1341384534 conn(0x7f05940727e0 msgr2=0x7f059406e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.667+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 shutdown_connections 2024-03-18T16:34:01.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:01.667+0000 7f0598a00640 1 -- 172.21.15.102:0/1341384534 wait complete. 2024-03-18T16:34:02.290 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:02.290 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:03.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:03.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.467+0000 7f4c05200640 1 Processor -- start 2024-03-18T16:34:03.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.467+0000 7f4c05200640 1 -- start start 2024-03-18T16:34:03.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.468+0000 7f4c05200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000d0250 0x7f4c0015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:03.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.468+0000 7f4c05200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c000cc2f0 0x7f4c000cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:03.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.468+0000 7f4c05200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c000ccca0 con 0x7f4c000cfe70 2024-03-18T16:34:03.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.468+0000 7f4c05200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4c000cabd0 con 0x7f4c000cc2f0 2024-03-18T16:34:03.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.468+0000 7f4c05200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4c000caea0 con 0x7f4c000d0250 2024-03-18T16:34:03.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.469+0000 7f4bfea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000d0250 0x7f4c0015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:03.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.469+0000 7f4bfea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000d0250 0x7f4c0015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50880/0 (socket says 172.21.15.102:50880) 2024-03-18T16:34:03.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.469+0000 7f4bfea00640 1 -- 172.21.15.102:0/2193700648 learned_addr learned my addr 172.21.15.102:0/2193700648 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:03.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.470+0000 7f4bffe00640 1 --2- 172.21.15.102:0/2193700648 >> v2:172.21.15.102:3300/0 conn(0x7f4c000cc2f0 0x7f4c000cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.471+0000 7f4bfea00640 1 -- 172.21.15.102:0/2193700648 >> v2:172.21.15.102:3300/0 conn(0x7f4c000cc2f0 msgr2=0x7f4c000cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:03.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.471+0000 7f4bfea00640 1 --2- 172.21.15.102:0/2193700648 >> v2:172.21.15.102:3300/0 conn(0x7f4c000cc2f0 0x7f4c000cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:03.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.471+0000 7f4bfea00640 1 -- 172.21.15.102:0/2193700648 >> v1:172.21.15.102:6789/0 conn(0x7f4c000cfe70 legacy=0x7f4c000c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:03.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.471+0000 7f4bfea00640 1 -- 172.21.15.102:0/2193700648 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c000cb180 con 0x7f4c000d0250 2024-03-18T16:34:03.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.473+0000 7f4bfea00640 1 --2- 172.21.15.102:0/2193700648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000d0250 0x7f4c0015ae40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009870 tx=0x7f4bec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad9d20b797566ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:03.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.477+0000 7f4bfe000640 1 -- 172.21.15.102:0/2193700648 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec034070 con 0x7f4c000d0250 2024-03-18T16:34:03.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.479+0000 7f4bfe000640 1 -- 172.21.15.102:0/2193700648 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bec030840 con 0x7f4c000d0250 2024-03-18T16:34:03.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.479+0000 7f4bfe000640 1 -- 172.21.15.102:0/2193700648 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bec030c10 con 0x7f4c000d0250 2024-03-18T16:34:03.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.480+0000 7f4c05200640 1 -- 172.21.15.102:0/2193700648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000d0250 msgr2=0x7f4c0015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:03.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.480+0000 7f4c05200640 1 --2- 172.21.15.102:0/2193700648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000d0250 0x7f4c0015ae40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4bec009870 tx=0x7f4bec02b980 comp rx=0 tx=0).stop 2024-03-18T16:34:03.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.480+0000 7f4c05200640 1 -- 172.21.15.102:0/2193700648 shutdown_connections 2024-03-18T16:34:03.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.480+0000 7f4c05200640 1 --2- 172.21.15.102:0/2193700648 >> v2:172.21.15.102:3300/0 conn(0x7f4c000cc2f0 0x7f4c000cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:03.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.480+0000 7f4c05200640 1 --2- 172.21.15.102:0/2193700648 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000d0250 0x7f4c0015ae40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:03.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.480+0000 7f4c05200640 1 -- 172.21.15.102:0/2193700648 >> 172.21.15.102:0/2193700648 conn(0x7f4c0008c3e0 msgr2=0x7f4c0006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:03.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.480+0000 7f4c05200640 1 -- 172.21.15.102:0/2193700648 shutdown_connections 2024-03-18T16:34:03.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.480+0000 7f4c05200640 1 -- 172.21.15.102:0/2193700648 wait complete. 2024-03-18T16:34:03.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.483+0000 7f4c05200640 1 Processor -- start 2024-03-18T16:34:03.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.483+0000 7f4c05200640 1 -- start start 2024-03-18T16:34:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.484+0000 7f4c05200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c000cc2f0 0x7f4c001e6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.484+0000 7f4c05200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000cfe70 0x7f4c00070530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.484+0000 7f4c05200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c000d0250 0x7f4c001e36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.484+0000 7f4c05200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4c000cba20 con 0x7f4c000cc2f0 2024-03-18T16:34:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.484+0000 7f4c05200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c000cb740 con 0x7f4c000d0250 2024-03-18T16:34:03.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.484+0000 7f4c05200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4c000cbd00 con 0x7f4c000cfe70 2024-03-18T16:34:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.485+0000 7f4bff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c000cc2f0 0x7f4c001e6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.485+0000 7f4bffe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c000d0250 0x7f4c001e36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.486+0000 7f4bff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c000cc2f0 0x7f4c001e6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56136/0 (socket says 172.21.15.102:56136) 2024-03-18T16:34:03.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.486+0000 7f4bff400640 1 -- 172.21.15.102:0/806251158 learned_addr learned my addr 172.21.15.102:0/806251158 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:03.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.487+0000 7f4bfea00640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000cfe70 0x7f4c00070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.491+0000 7f4bffe00640 1 -- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000cfe70 msgr2=0x7f4c00070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.491+0000 7f4bffe00640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000cfe70 0x7f4c00070530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.491+0000 7f4bffe00640 1 -- 172.21.15.102:0/806251158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c000cc2f0 msgr2=0x7f4c001e6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.491+0000 7f4bffe00640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c000cc2f0 0x7f4c001e6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:03.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.491+0000 7f4bffe00640 1 -- 172.21.15.102:0/806251158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c001e3210 con 0x7f4c000d0250 2024-03-18T16:34:03.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.493+0000 7f4bffe00640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c000d0250 0x7f4c001e36f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8007df0 tx=0x7f4bf800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:03.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.498+0000 7f4bf7400640 1 -- 172.21.15.102:0/806251158 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf800edd0 con 0x7f4c000d0250 2024-03-18T16:34:03.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.498+0000 7f4bf7400640 1 -- 172.21.15.102:0/806251158 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf8003990 con 0x7f4c000d0250 2024-03-18T16:34:03.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.498+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c0006fb10 con 0x7f4c000d0250 2024-03-18T16:34:03.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.498+0000 7f4bf7400640 1 -- 172.21.15.102:0/806251158 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8003d80 con 0x7f4c000d0250 2024-03-18T16:34:03.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.498+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c001e2a00 con 0x7f4c000d0250 2024-03-18T16:34:03.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.499+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c00074fd0 con 0x7f4c000d0250 2024-03-18T16:34:03.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.519+0000 7f4bf7400640 1 -- 172.21.15.102:0/806251158 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4bf801b020 con 0x7f4c000d0250 2024-03-18T16:34:03.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.520+0000 7f4bf7400640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4be006bc20 0x7f4be006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:03.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.521+0000 7f4bff400640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4be006bc20 0x7f4be006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:03.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.521+0000 7f4bff400640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4be006bc20 0x7f4be006e0e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0005020 tx=0x7f4bf0007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:03.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.522+0000 7f4bf7400640 1 -- 172.21.15.102:0/806251158 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f4bf8057f00 con 0x7f4c000d0250 2024-03-18T16:34:03.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:03.773+0000 7f4bf7400640 1 -- 172.21.15.102:0/806251158 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4bf8059e60 con 0x7f4c000d0250 2024-03-18T16:34:04.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.017+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c001e33e0 con 0x7f4c000d0250 2024-03-18T16:34:04.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.120+0000 7f4bf7400640 1 -- 172.21.15.102:0/806251158 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f4bf805ee20 con 0x7f4c000d0250 2024-03-18T16:34:04.122 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:04.122 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:04.122 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.123+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4be006bc20 msgr2=0x7f4be006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.123+0000 7f4c05200640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4be006bc20 0x7f4be006e0e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4bf0005020 tx=0x7f4bf0007ef0 comp rx=0 tx=0).stop 2024-03-18T16:34:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.123+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c000d0250 msgr2=0x7f4c001e36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:04.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.123+0000 7f4c05200640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c000d0250 0x7f4c001e36f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8007df0 tx=0x7f4bf800e4b0 comp rx=0 tx=0).stop 2024-03-18T16:34:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.124+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 shutdown_connections 2024-03-18T16:34:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.124+0000 7f4c05200640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4be006bc20 0x7f4be006e0e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.124+0000 7f4c05200640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c000d0250 0x7f4c001e36f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.124+0000 7f4c05200640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4c000cfe70 0x7f4c00070530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.124+0000 7f4c05200640 1 --2- 172.21.15.102:0/806251158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c000cc2f0 0x7f4c001e6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.124+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 >> 172.21.15.102:0/806251158 conn(0x7f4c0008c3e0 msgr2=0x7f4c000cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.124+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 shutdown_connections 2024-03-18T16:34:04.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:04.124+0000 7f4c05200640 1 -- 172.21.15.102:0/806251158 wait complete. 2024-03-18T16:34:04.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:04.751 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:05.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:05.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.923+0000 7feeeaa00640 1 Processor -- start 2024-03-18T16:34:05.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.924+0000 7feeeaa00640 1 -- start start 2024-03-18T16:34:05.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.925+0000 7feeeaa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feee4071700 0x7feee4072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:05.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.925+0000 7feeeaa00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feee40c4190 con 0x7feee4158c00 2024-03-18T16:34:05.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.925+0000 7feeeaa00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feee406f060 con 0x7feee4161d30 2024-03-18T16:34:05.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.925+0000 7feeeaa00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7feee4158fe0 con 0x7feee4071700 2024-03-18T16:34:05.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.926+0000 7feee8c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feee4071700 0x7feee4072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:05.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.926+0000 7feeea000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7feee4161d30 0x7feee4169110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44534/0 (socket says 172.21.15.102:44534) 2024-03-18T16:34:05.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.926+0000 7feeea000640 1 -- 172.21.15.102:0/3244266462 learned_addr learned my addr 172.21.15.102:0/3244266462 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:05.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.928+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846605688 0 0) 0x7feee406f060 con 0x7feee4161d30 2024-03-18T16:34:05.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.929+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feecc003520 con 0x7feee4161d30 2024-03-18T16:34:05.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2440736112 0 0) 0x7feecc003520 con 0x7feee4161d30 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 >> v2:172.21.15.102:3300/0 conn(0x7feee4071700 msgr2=0x7feee4072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 --2- 172.21.15.102:0/3244266462 >> v2:172.21.15.102:3300/0 conn(0x7feee4071700 0x7feee4072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 >> v1:172.21.15.102:6789/0 conn(0x7feee4158c00 legacy=0x7feee406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee4071cb0 con 0x7feee4161d30 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7feedc0019f0 con 0x7feee4161d30 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 >> v1:172.21.15.154:6789/0 conn(0x7feee4161d30 legacy=0x7feee4169110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feecc0039b0 0x7feecc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feecc004800 0x7feecc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feecc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feee4158fe0 con 0x7feecc0039b0 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feecc00c060 con 0x7feecc004800 2024-03-18T16:34:05.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.932+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feecc00c260 con 0x7feecc0051c0 2024-03-18T16:34:05.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.933+0000 7feeea000640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feecc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:05.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.935+0000 7feee8c00640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feecc004800 0x7feecc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:05.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.936+0000 7feeea000640 1 -- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feecc004800 msgr2=0x7feecc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:05.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.936+0000 7feeea000640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feecc004800 0x7feecc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:05.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.936+0000 7feeea000640 1 -- 172.21.15.102:0/3244266462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feecc0039b0 msgr2=0x7feecc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:05.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.936+0000 7feeea000640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feecc0039b0 0x7feecc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:05.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.936+0000 7feeea000640 1 -- 172.21.15.102:0/3244266462 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feecc003520 con 0x7feecc0051c0 2024-03-18T16:34:05.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.937+0000 7feeea000640 1 -- 172.21.15.102:0/3244266462 reap_dead start 2024-03-18T16:34:05.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.939+0000 7feeea000640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feecc00b9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7feedc0013d0 tx=0x7feecc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43f9d3b9fd90ae17 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:05.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.942+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc00a450 con 0x7feecc0051c0 2024-03-18T16:34:05.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.943+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feedc0072d0 con 0x7feecc0051c0 2024-03-18T16:34:05.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.944+0000 7feee3e00640 1 -- 172.21.15.102:0/3244266462 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feedc0076c0 con 0x7feecc0051c0 2024-03-18T16:34:05.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.944+0000 7feeeaa00640 1 -- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 msgr2=0x7feecc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:05.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.944+0000 7feeeaa00640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feecc00b9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7feedc0013d0 tx=0x7feecc005830 comp rx=0 tx=0).stop 2024-03-18T16:34:05.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.944+0000 7feeeaa00640 1 -- 172.21.15.102:0/3244266462 shutdown_connections 2024-03-18T16:34:05.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.944+0000 7feeeaa00640 1 --2- 172.21.15.102:0/3244266462 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feecc00b9c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:05.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.944+0000 7feeeaa00640 1 -- 172.21.15.102:0/3244266462 >> 172.21.15.102:0/3244266462 conn(0x7feee40cb890 msgr2=0x7feee40cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:05.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.945+0000 7feeeaa00640 1 -- 172.21.15.102:0/3244266462 shutdown_connections 2024-03-18T16:34:05.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.945+0000 7feeeaa00640 1 -- 172.21.15.102:0/3244266462 wait complete. 2024-03-18T16:34:05.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.947+0000 7feeeaa00640 1 Processor -- start 2024-03-18T16:34:05.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.948+0000 7feeeaa00640 1 -- start start 2024-03-18T16:34:05.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.949+0000 7feeeaa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feee41f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.949+0000 7feeeaa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feee414b9f0 0x7feee4156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.949+0000 7feeeaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee40ca180 0x7feee4156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.949+0000 7feeeaa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feee4072270 con 0x7feee40ca180 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.949+0000 7feeeaa00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feee406f3b0 con 0x7feee414b9f0 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.949+0000 7feeeaa00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feee4072550 con 0x7feecc0051c0 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.950+0000 7feee9600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feee41f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.950+0000 7feee8c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feee414b9f0 0x7feee4156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.950+0000 7feeea000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee40ca180 0x7feee4156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.950+0000 7feee9600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feee41f4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50904/0 (socket says 172.21.15.102:50904) 2024-03-18T16:34:05.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.950+0000 7feee9600640 1 -- 172.21.15.102:0/2087291163 learned_addr learned my addr 172.21.15.102:0/2087291163 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.955+0000 7feee9600640 1 -- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feee414b9f0 msgr2=0x7feee4156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.955+0000 7feee9600640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feee414b9f0 0x7feee4156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.955+0000 7feee9600640 1 -- 172.21.15.102:0/2087291163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee40ca180 msgr2=0x7feee4156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.955+0000 7feee9600640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee40ca180 0x7feee4156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.955+0000 7feee9600640 1 -- 172.21.15.102:0/2087291163 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee41f5360 con 0x7feecc0051c0 2024-03-18T16:34:05.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.955+0000 7feee8c00640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feee414b9f0 0x7feee4156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:34:05.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.958+0000 7feee9600640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feee41f4e20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7feed800c430 tx=0x7feed800c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:05.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.962+0000 7feee1600640 1 -- 172.21.15.102:0/2087291163 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed80052d0 con 0x7feecc0051c0 2024-03-18T16:34:05.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.962+0000 7feee1600640 1 -- 172.21.15.102:0/2087291163 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feed8007020 con 0x7feecc0051c0 2024-03-18T16:34:05.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.962+0000 7feee1600640 1 -- 172.21.15.102:0/2087291163 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed80054b0 con 0x7feecc0051c0 2024-03-18T16:34:05.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.962+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee414d4a0 con 0x7feecc0051c0 2024-03-18T16:34:05.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.963+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feee41f4620 con 0x7feecc0051c0 2024-03-18T16:34:05.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.964+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee414d9c0 con 0x7feecc0051c0 2024-03-18T16:34:05.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.983+0000 7feee1600640 1 -- 172.21.15.102:0/2087291163 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7feed8007410 con 0x7feecc0051c0 2024-03-18T16:34:05.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.984+0000 7feee1600640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feec406bc20 0x7feec406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:05.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.985+0000 7feee8c00640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feec406bc20 0x7feec406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:05.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.985+0000 7feee8c00640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feec406bc20 0x7feec406e0e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7feee40c2130 tx=0x7feed0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:05.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:05.986+0000 7feee1600640 1 -- 172.21.15.102:0/2087291163 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7feed80568a0 con 0x7feecc0051c0 2024-03-18T16:34:06.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.257+0000 7feee1600640 1 -- 172.21.15.102:0/2087291163 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feee414d9c0 con 0x7feecc0051c0 2024-03-18T16:34:06.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.546+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feee41f4410 con 0x7feecc0051c0 2024-03-18T16:34:06.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.570+0000 7feee1600640 1 -- 172.21.15.102:0/2087291163 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7feed8058800 con 0x7feecc0051c0 2024-03-18T16:34:06.573 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:06.573 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:06.573 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:06.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feec406bc20 msgr2=0x7feec406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:06.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feec406bc20 0x7feec406e0e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7feee40c2130 tx=0x7feed0009190 comp rx=0 tx=0).stop 2024-03-18T16:34:06.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 msgr2=0x7feee41f4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:06.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feee41f4e20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7feed800c430 tx=0x7feed800c830 comp rx=0 tx=0).stop 2024-03-18T16:34:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 shutdown_connections 2024-03-18T16:34:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7feec406bc20 0x7feec406e0e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feee40ca180 0x7feee4156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feee414b9f0 0x7feee4156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 --2- 172.21.15.102:0/2087291163 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feecc0051c0 0x7feee41f4e20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:06.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.573+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 >> 172.21.15.102:0/2087291163 conn(0x7feee40cb890 msgr2=0x7feee40c5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.575+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 shutdown_connections 2024-03-18T16:34:06.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:06.575+0000 7feeeaa00640 1 -- 172.21.15.102:0/2087291163 wait complete. 2024-03-18T16:34:07.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:07.200 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:08.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:08.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.374+0000 7f11ebe00640 1 Processor -- start 2024-03-18T16:34:08.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.374+0000 7f11ebe00640 1 -- start start 2024-03-18T16:34:08.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.375+0000 7f11ebe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11ec154630 0x7f11ec0cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:08.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.376+0000 7f11ebe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f11ec0ca510 0x7f11ec0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:08.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.376+0000 7f11ebe00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11ec0cf5c0 con 0x7f11ec0ca960 2024-03-18T16:34:08.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.376+0000 7f11ebe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f11ec06d640 con 0x7f11ec154630 2024-03-18T16:34:08.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.376+0000 7f11ebe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f11ec06d920 con 0x7f11ec0ca510 2024-03-18T16:34:08.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.376+0000 7f11ea000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f11ec0ca510 0x7f11ec0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:08.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.377+0000 7f11eaa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11ec154630 0x7f11ec0cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:08.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.377+0000 7f11eb400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f11ec0ca960 0x7f11ec15a520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44540/0 (socket says 172.21.15.102:44540) 2024-03-18T16:34:08.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.377+0000 7f11eb400640 1 -- 172.21.15.102:0/2181593728 learned_addr learned my addr 172.21.15.102:0/2181593728 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:08.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.380+0000 7f11ea000640 1 -- 172.21.15.102:0/2181593728 >> v2:172.21.15.102:3300/0 conn(0x7f11ec154630 msgr2=0x7f11ec0cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:08.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.380+0000 7f11ea000640 1 --2- 172.21.15.102:0/2181593728 >> v2:172.21.15.102:3300/0 conn(0x7f11ec154630 0x7f11ec0cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:08.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.380+0000 7f11ea000640 1 -- 172.21.15.102:0/2181593728 >> v1:172.21.15.154:6789/0 conn(0x7f11ec0ca960 legacy=0x7f11ec15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:08.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.380+0000 7f11ea000640 1 -- 172.21.15.102:0/2181593728 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec06dc00 con 0x7f11ec0ca510 2024-03-18T16:34:08.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.384+0000 7f11ea000640 1 --2- 172.21.15.102:0/2181593728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 0x7f11ec0cfe30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f11ec06d450 tx=0x7f11d802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac841cb3f8e3262e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:08.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.389+0000 7f11e9600640 1 -- 172.21.15.102:0/2181593728 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8034070 con 0x7f11ec0ca510 2024-03-18T16:34:08.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.389+0000 7f11e9600640 1 -- 172.21.15.102:0/2181593728 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d8030610 con 0x7f11ec0ca510 2024-03-18T16:34:08.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.389+0000 7f11e9600640 1 -- 172.21.15.102:0/2181593728 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d80309e0 con 0x7f11ec0ca510 2024-03-18T16:34:08.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.390+0000 7f11ebe00640 1 -- 172.21.15.102:0/2181593728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 msgr2=0x7f11ec0cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:08.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.390+0000 7f11ebe00640 1 --2- 172.21.15.102:0/2181593728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 0x7f11ec0cfe30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f11ec06d450 tx=0x7f11d802b960 comp rx=0 tx=0).stop 2024-03-18T16:34:08.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.390+0000 7f11ebe00640 1 -- 172.21.15.102:0/2181593728 shutdown_connections 2024-03-18T16:34:08.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.390+0000 7f11ebe00640 1 --2- 172.21.15.102:0/2181593728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 0x7f11ec0cfe30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:08.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.390+0000 7f11ebe00640 1 --2- 172.21.15.102:0/2181593728 >> v2:172.21.15.102:3300/0 conn(0x7f11ec154630 0x7f11ec0cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:08.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.390+0000 7f11ebe00640 1 -- 172.21.15.102:0/2181593728 >> 172.21.15.102:0/2181593728 conn(0x7f11ec067160 msgr2=0x7f11ec067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:08.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.390+0000 7f11ebe00640 1 -- 172.21.15.102:0/2181593728 shutdown_connections 2024-03-18T16:34:08.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.390+0000 7f11ebe00640 1 -- 172.21.15.102:0/2181593728 wait complete. 2024-03-18T16:34:08.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.393+0000 7f11ebe00640 1 Processor -- start 2024-03-18T16:34:08.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.393+0000 7f11ebe00640 1 -- start start 2024-03-18T16:34:08.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.395+0000 7f11ebe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 0x7f11ec1f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:08.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.395+0000 7f11ebe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec0ca960 0x7f11ec1584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:08.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.395+0000 7f11ebe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11ec154630 0x7f11ec15ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:08.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.395+0000 7f11ebe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec154b90 con 0x7f11ec0ca960 2024-03-18T16:34:08.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.395+0000 7f11ebe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec154a10 con 0x7f11ec0ca510 2024-03-18T16:34:08.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.395+0000 7f11ebe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f11ec15ac30 con 0x7f11ec154630 2024-03-18T16:34:08.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.395+0000 7f11eaa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 0x7f11ec1f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:08.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.395+0000 7f11eb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11ec154630 0x7f11ec15ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:08.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.396+0000 7f11eaa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 0x7f11ec1f0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:52220/0 (socket says 172.21.15.102:52220) 2024-03-18T16:34:08.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.396+0000 7f11eaa00640 1 -- 172.21.15.102:0/2647617642 learned_addr learned my addr 172.21.15.102:0/2647617642 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:08.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.396+0000 7f11ea000640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec0ca960 0x7f11ec1584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:08.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.399+0000 7f11eaa00640 1 -- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11ec154630 msgr2=0x7f11ec15ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:08.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.399+0000 7f11eaa00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11ec154630 0x7f11ec15ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:08.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.399+0000 7f11eaa00640 1 -- 172.21.15.102:0/2647617642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec0ca960 msgr2=0x7f11ec1584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:08.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.399+0000 7f11eaa00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec0ca960 0x7f11ec1584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:08.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.399+0000 7f11eaa00640 1 -- 172.21.15.102:0/2647617642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec1f11d0 con 0x7f11ec0ca510 2024-03-18T16:34:08.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.404+0000 7f11eaa00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 0x7f11ec1f0c90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00e8a0 tx=0x7f11dc00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:08.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.410+0000 7f11e2a00640 1 -- 172.21.15.102:0/2647617642 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc017070 con 0x7f11ec0ca510 2024-03-18T16:34:08.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.410+0000 7f11e2a00640 1 -- 172.21.15.102:0/2647617642 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11dc00c940 con 0x7f11ec0ca510 2024-03-18T16:34:08.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.410+0000 7f11e2a00640 1 -- 172.21.15.102:0/2647617642 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc00cd30 con 0x7f11ec0ca510 2024-03-18T16:34:08.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.410+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec15d860 con 0x7f11ec0ca510 2024-03-18T16:34:08.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.410+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec15e390 con 0x7f11ec0ca510 2024-03-18T16:34:08.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.412+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11ec074fd0 con 0x7f11ec0ca510 2024-03-18T16:34:08.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.444+0000 7f11e2a00640 1 -- 172.21.15.102:0/2647617642 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f11dc01a070 con 0x7f11ec0ca510 2024-03-18T16:34:08.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.446+0000 7f11e2a00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f11cc06bc20 0x7f11cc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:08.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.446+0000 7f11ea000640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f11cc06bc20 0x7f11cc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.447+0000 7f11ea000640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f11cc06bc20 0x7f11cc06e0e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f11d8036000 tx=0x7f11d803b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.448+0000 7f11e2a00640 1 -- 172.21.15.102:0/2647617642 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f11dc057590 con 0x7f11ec0ca510 2024-03-18T16:34:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:08.710+0000 7f11e2a00640 1 -- 172.21.15.102:0/2647617642 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f11dc0594f0 con 0x7f11ec0ca510 2024-03-18T16:34:09.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.071+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11ec0cb750 con 0x7f11ec0ca510 2024-03-18T16:34:09.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.086+0000 7f11e2a00640 1 -- 172.21.15.102:0/2647617642 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f11dc05d590 con 0x7f11ec0ca510 2024-03-18T16:34:09.089 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:09.089 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:09.089 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:09.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f11cc06bc20 msgr2=0x7f11cc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:09.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f11cc06bc20 0x7f11cc06e0e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f11d8036000 tx=0x7f11d803b040 comp rx=0 tx=0).stop 2024-03-18T16:34:09.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 msgr2=0x7f11ec1f0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:09.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 0x7f11ec1f0c90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00e8a0 tx=0x7f11dc00ed60 comp rx=0 tx=0).stop 2024-03-18T16:34:09.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 shutdown_connections 2024-03-18T16:34:09.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f11cc06bc20 0x7f11cc06e0e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:09.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f11ec154630 0x7f11ec15ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:09.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11ec0ca960 0x7f11ec1584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:09.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 --2- 172.21.15.102:0/2647617642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ec0ca510 0x7f11ec1f0c90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:09.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 >> 172.21.15.102:0/2647617642 conn(0x7f11ec067160 msgr2=0x7f11ec152f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:09.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 shutdown_connections 2024-03-18T16:34:09.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:09.090+0000 7f11ebe00640 1 -- 172.21.15.102:0/2647617642 wait complete. 2024-03-18T16:34:09.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:09.720 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:10.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:10.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.895+0000 7fdee8400640 1 Processor -- start 2024-03-18T16:34:10.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.895+0000 7fdee8400640 1 -- start start 2024-03-18T16:34:10.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.897+0000 7fdee8400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee0072300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:10.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.897+0000 7fdee8400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdee0158760 0x7fdee015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:10.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.898+0000 7fdee8400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee0157ac0 con 0x7fdee00c8b20 2024-03-18T16:34:10.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.898+0000 7fdee8400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdee015a860 con 0x7fdee0158760 2024-03-18T16:34:10.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.898+0000 7fdee8400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdee0072840 con 0x7fdee00c8740 2024-03-18T16:34:10.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.900+0000 7fdee6400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdee0158760 0x7fdee015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:10.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.900+0000 7fdee6400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdee0158760 0x7fdee015b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57216/0 (socket says 172.21.15.102:57216) 2024-03-18T16:34:10.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.900+0000 7fdee6400640 1 -- 172.21.15.102:0/3123522753 learned_addr learned my addr 172.21.15.102:0/3123522753 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:10.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.901+0000 7fdee5a00640 1 --2- 172.21.15.102:0/3123522753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee0072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:10.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.950+0000 7fdee6400640 1 -- 172.21.15.102:0/3123522753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 msgr2=0x7fdee0072300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:10.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.950+0000 7fdee6400640 1 --2- 172.21.15.102:0/3123522753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee0072300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:10.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.950+0000 7fdee6400640 1 -- 172.21.15.102:0/3123522753 >> v1:172.21.15.154:6789/0 conn(0x7fdee00c8b20 legacy=0x7fdee01572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:10.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.950+0000 7fdee6400640 1 -- 172.21.15.102:0/3123522753 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee0157da0 con 0x7fdee0158760 2024-03-18T16:34:10.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.968+0000 7fdee6400640 1 --2- 172.21.15.102:0/3123522753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0158760 0x7fdee015b2b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fded8009e50 tx=0x7fded802e0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51e365540896a0e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:10.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.974+0000 7fdedfe00640 1 -- 172.21.15.102:0/3123522753 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded802ec80 con 0x7fdee0158760 2024-03-18T16:34:10.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.974+0000 7fdedfe00640 1 -- 172.21.15.102:0/3123522753 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded802ee20 con 0x7fdee0158760 2024-03-18T16:34:10.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.974+0000 7fdedfe00640 1 -- 172.21.15.102:0/3123522753 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded8036610 con 0x7fdee0158760 2024-03-18T16:34:10.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.975+0000 7fdee8400640 1 -- 172.21.15.102:0/3123522753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0158760 msgr2=0x7fdee015b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:10.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.975+0000 7fdee8400640 1 --2- 172.21.15.102:0/3123522753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0158760 0x7fdee015b2b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fded8009e50 tx=0x7fded802e0e0 comp rx=0 tx=0).stop 2024-03-18T16:34:10.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.975+0000 7fdee8400640 1 -- 172.21.15.102:0/3123522753 shutdown_connections 2024-03-18T16:34:10.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.975+0000 7fdee8400640 1 --2- 172.21.15.102:0/3123522753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0158760 0x7fdee015b2b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:10.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.975+0000 7fdee8400640 1 --2- 172.21.15.102:0/3123522753 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee0072300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:10.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.975+0000 7fdee8400640 1 -- 172.21.15.102:0/3123522753 >> 172.21.15.102:0/3123522753 conn(0x7fdee00700d0 msgr2=0x7fdee006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:10.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.975+0000 7fdee8400640 1 -- 172.21.15.102:0/3123522753 shutdown_connections 2024-03-18T16:34:10.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.975+0000 7fdee8400640 1 -- 172.21.15.102:0/3123522753 wait complete. 2024-03-18T16:34:10.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.978+0000 7fdee8400640 1 Processor -- start 2024-03-18T16:34:10.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.978+0000 7fdee8400640 1 -- start start 2024-03-18T16:34:10.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.979+0000 7fdee8400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:10.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.979+0000 7fdee8400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee00c8b20 0x7fdee0144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:10.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.979+0000 7fdee8400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0158760 0x7fdee0149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:10.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.979+0000 7fdee8400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdee015aab0 con 0x7fdee00c8b20 2024-03-18T16:34:10.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.979+0000 7fdee8400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdee015b7f0 con 0x7fdee0158760 2024-03-18T16:34:10.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.979+0000 7fdee8400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdee015ad90 con 0x7fdee00c8740 2024-03-18T16:34:10.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.980+0000 7fdee5a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:10.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.980+0000 7fdee5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee00c8b20 0x7fdee0144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:10.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.981+0000 7fdee5a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee014aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:37422/0 (socket says 172.21.15.102:37422) 2024-03-18T16:34:10.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.981+0000 7fdee5a00640 1 -- 172.21.15.102:0/3978049498 learned_addr learned my addr 172.21.15.102:0/3978049498 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:10.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.984+0000 7fdee5a00640 1 -- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0158760 msgr2=0x7fdee0149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:10.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.984+0000 7fdee5a00640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0158760 0x7fdee0149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:10.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.984+0000 7fdee5a00640 1 -- 172.21.15.102:0/3978049498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee00c8b20 msgr2=0x7fdee0144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:10.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.984+0000 7fdee5a00640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee00c8b20 0x7fdee0144070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:10.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.984+0000 7fdee5a00640 1 -- 172.21.15.102:0/3978049498 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee014b1a0 con 0x7fdee00c8740 2024-03-18T16:34:10.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.987+0000 7fdee5a00640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee014aa90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fded400e8a0 tx=0x7fded400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:10.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.991+0000 7fdedd600640 1 -- 172.21.15.102:0/3978049498 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4017070 con 0x7fdee00c8740 2024-03-18T16:34:10.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.991+0000 7fdedd600640 1 -- 172.21.15.102:0/3978049498 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded400c940 con 0x7fdee00c8740 2024-03-18T16:34:10.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.991+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee014b440 con 0x7fdee00c8740 2024-03-18T16:34:10.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.991+0000 7fdedd600640 1 -- 172.21.15.102:0/3978049498 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded400cd30 con 0x7fdee00c8740 2024-03-18T16:34:10.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.991+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee01ef450 con 0x7fdee00c8740 2024-03-18T16:34:10.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:10.993+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee0074fd0 con 0x7fdee00c8740 2024-03-18T16:34:11.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.014+0000 7fdedd600640 1 -- 172.21.15.102:0/3978049498 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fded401a070 con 0x7fdee00c8740 2024-03-18T16:34:11.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.016+0000 7fdedd600640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdec006bc20 0x7fdec006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:11.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.016+0000 7fdee5000640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdec006bc20 0x7fdec006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:11.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.017+0000 7fdedd600640 1 -- 172.21.15.102:0/3978049498 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fded4057550 con 0x7fdee00c8740 2024-03-18T16:34:11.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.017+0000 7fdee5000640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdec006bc20 0x7fdec006e0e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdee00c2fd0 tx=0x7fdecc009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:11.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.267+0000 7fdedd600640 1 -- 172.21.15.102:0/3978049498 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fded40594b0 con 0x7fdee00c8740 2024-03-18T16:34:11.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.543+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdee00c5b90 con 0x7fdee00c8740 2024-03-18T16:34:11.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.555+0000 7fdedd600640 1 -- 172.21.15.102:0/3978049498 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fded405d550 con 0x7fdee00c8740 2024-03-18T16:34:11.557 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:11.557 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:11.558 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:11.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdec006bc20 msgr2=0x7fdec006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:11.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdec006bc20 0x7fdec006e0e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdee00c2fd0 tx=0x7fdecc009060 comp rx=0 tx=0).stop 2024-03-18T16:34:11.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 msgr2=0x7fdee014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:11.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee014aa90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fded400e8a0 tx=0x7fded400ed60 comp rx=0 tx=0).stop 2024-03-18T16:34:11.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 shutdown_connections 2024-03-18T16:34:11.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdec006bc20 0x7fdec006e0e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:11.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee0158760 0x7fdee0149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:11.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee00c8b20 0x7fdee0144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:11.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 --2- 172.21.15.102:0/3978049498 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdee00c8740 0x7fdee014aa90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:11.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 >> 172.21.15.102:0/3978049498 conn(0x7fdee00700d0 msgr2=0x7fdee0073170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:11.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 shutdown_connections 2024-03-18T16:34:11.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:11.558+0000 7fdee8400640 1 -- 172.21.15.102:0/3978049498 wait complete. 2024-03-18T16:34:12.187 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:12.187 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:13.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:13.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.380+0000 7fb103200640 1 Processor -- start 2024-03-18T16:34:13.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.381+0000 7fb103200640 1 -- start start 2024-03-18T16:34:13.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.382+0000 7fb103200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb1040d0250 0x7fb10415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:13.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.382+0000 7fb103200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1040cabd0 con 0x7fb1040cc2f0 2024-03-18T16:34:13.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.382+0000 7fb103200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1040caea0 con 0x7fb1040cfe70 2024-03-18T16:34:13.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.382+0000 7fb103200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb1040cb180 con 0x7fb1040d0250 2024-03-18T16:34:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.383+0000 7fb101e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb1040cfe70 0x7fb1040c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:38774/0 (socket says 172.21.15.102:38774) 2024-03-18T16:34:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.383+0000 7fb101e00640 1 -- 172.21.15.102:0/1938796701 learned_addr learned my addr 172.21.15.102:0/1938796701 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.383+0000 7fb101400640 1 --2- 172.21.15.102:0/1938796701 >> v2:172.21.15.102:3300/0 conn(0x7fb1040d0250 0x7fb10415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:13.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.386+0000 7fb100a00640 1 -- 172.21.15.102:0/1938796701 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1624893535 0 0) 0x7fb1040caea0 con 0x7fb1040cfe70 2024-03-18T16:34:13.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.386+0000 7fb100a00640 1 -- 172.21.15.102:0/1938796701 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0e8003520 con 0x7fb1040cfe70 2024-03-18T16:34:13.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.387+0000 7fb101400640 1 -- 172.21.15.102:0/1938796701 >> v1:172.21.15.154:6789/0 conn(0x7fb1040cfe70 legacy=0x7fb1040c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:13.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.387+0000 7fb101400640 1 -- 172.21.15.102:0/1938796701 >> v1:172.21.15.102:6789/0 conn(0x7fb1040cc2f0 legacy=0x7fb1040cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:13.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.387+0000 7fb101400640 1 -- 172.21.15.102:0/1938796701 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1040cb460 con 0x7fb1040d0250 2024-03-18T16:34:13.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.393+0000 7fb101400640 1 --2- 172.21.15.102:0/1938796701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1040d0250 0x7fb10415ae40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00097a0 tx=0x7fb0f002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d6f389a6c21632 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:13.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.427+0000 7fb100a00640 1 -- 172.21.15.102:0/1938796701 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f002f810 con 0x7fb1040d0250 2024-03-18T16:34:13.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.427+0000 7fb100a00640 1 -- 172.21.15.102:0/1938796701 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f002f9b0 con 0x7fb1040d0250 2024-03-18T16:34:13.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.427+0000 7fb100a00640 1 -- 172.21.15.102:0/1938796701 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f002fd80 con 0x7fb1040d0250 2024-03-18T16:34:13.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.428+0000 7fb103200640 1 -- 172.21.15.102:0/1938796701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1040d0250 msgr2=0x7fb10415ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:13.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.428+0000 7fb103200640 1 --2- 172.21.15.102:0/1938796701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1040d0250 0x7fb10415ae40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00097a0 tx=0x7fb0f002b9f0 comp rx=0 tx=0).stop 2024-03-18T16:34:13.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.428+0000 7fb103200640 1 -- 172.21.15.102:0/1938796701 shutdown_connections 2024-03-18T16:34:13.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.428+0000 7fb103200640 1 --2- 172.21.15.102:0/1938796701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1040d0250 0x7fb10415ae40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:13.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.428+0000 7fb103200640 1 -- 172.21.15.102:0/1938796701 >> 172.21.15.102:0/1938796701 conn(0x7fb10408c3e0 msgr2=0x7fb10406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:13.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.428+0000 7fb103200640 1 -- 172.21.15.102:0/1938796701 shutdown_connections 2024-03-18T16:34:13.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.428+0000 7fb103200640 1 -- 172.21.15.102:0/1938796701 wait complete. 2024-03-18T16:34:13.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.431+0000 7fb103200640 1 Processor -- start 2024-03-18T16:34:13.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.431+0000 7fb103200640 1 -- start start 2024-03-18T16:34:13.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.433+0000 7fb103200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1040cc2f0 0x7fb1041582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:13.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.433+0000 7fb103200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1040cfe70 0x7fb10414f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:13.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.433+0000 7fb103200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1040d0250 0x7fb10414f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:13.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.433+0000 7fb103200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb1040cbd00 con 0x7fb1040d0250 2024-03-18T16:34:13.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.433+0000 7fb103200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb1040cba20 con 0x7fb1040cc2f0 2024-03-18T16:34:13.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.433+0000 7fb103200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb1040c8330 con 0x7fb1040cfe70 2024-03-18T16:34:13.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.433+0000 7fb101e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1040cc2f0 0x7fb1041582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:13.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.434+0000 7fb101400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1040cfe70 0x7fb10414f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:13.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.434+0000 7fb101400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1040cfe70 0x7fb10414f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:37428/0 (socket says 172.21.15.102:37428) 2024-03-18T16:34:13.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.434+0000 7fb101e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1040cc2f0 0x7fb1041582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57248/0 (socket says 172.21.15.102:57248) 2024-03-18T16:34:13.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.434+0000 7fb101400640 1 -- 172.21.15.102:0/1957334118 learned_addr learned my addr 172.21.15.102:0/1957334118 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:13.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.437+0000 7fb102800640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1040d0250 0x7fb10414f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.437+0000 7fb101400640 1 -- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1040cc2f0 msgr2=0x7fb1041582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.437+0000 7fb101400640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1040cc2f0 0x7fb1041582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.437+0000 7fb101400640 1 -- 172.21.15.102:0/1957334118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1040d0250 msgr2=0x7fb10414f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.437+0000 7fb101400640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1040d0250 0x7fb10414f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:13.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.437+0000 7fb101400640 1 -- 172.21.15.102:0/1957334118 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb104159d50 con 0x7fb1040cfe70 2024-03-18T16:34:13.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.439+0000 7fb101400640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1040cfe70 0x7fb10414f1b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009890 tx=0x7fb0f0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.443+0000 7fb0fa000640 1 -- 172.21.15.102:0/1957334118 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0046070 con 0x7fb1040cfe70 2024-03-18T16:34:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.443+0000 7fb0fa000640 1 -- 172.21.15.102:0/1957334118 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f00378f0 con 0x7fb1040cfe70 2024-03-18T16:34:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.443+0000 7fb0fa000640 1 -- 172.21.15.102:0/1957334118 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0037ce0 con 0x7fb1040cfe70 2024-03-18T16:34:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.443+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb104159a70 con 0x7fb1040cfe70 2024-03-18T16:34:13.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.444+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb104157ab0 con 0x7fb1040cfe70 2024-03-18T16:34:13.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.445+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb104074fd0 con 0x7fb1040cfe70 2024-03-18T16:34:13.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.466+0000 7fb0fa000640 1 -- 172.21.15.102:0/1957334118 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb0f0049070 con 0x7fb1040cfe70 2024-03-18T16:34:13.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.466+0000 7fb0fa000640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0e406bc20 0x7fb0e406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:13.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.467+0000 7fb101e00640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0e406bc20 0x7fb0e406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:13.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.467+0000 7fb0fa000640 1 -- 172.21.15.102:0/1957334118 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb0f0078d70 con 0x7fb1040cfe70 2024-03-18T16:34:13.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.468+0000 7fb101e00640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0e406bc20 0x7fb0e406e0e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40011e0 tx=0x7fb0f4007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:13.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.716+0000 7fb0fa000640 1 -- 172.21.15.102:0/1957334118 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb0f0050020 con 0x7fb1040cfe70 2024-03-18T16:34:13.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.962+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb10415b120 con 0x7fb1040cfe70 2024-03-18T16:34:13.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.975+0000 7fb0fa000640 1 -- 172.21.15.102:0/1957334118 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb0f0033030 con 0x7fb1040cfe70 2024-03-18T16:34:13.977 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:13.977 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:13.977 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:13.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0e406bc20 msgr2=0x7fb0e406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:13.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0e406bc20 0x7fb0e406e0e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb0f40011e0 tx=0x7fb0f4007d30 comp rx=0 tx=0).stop 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1040cfe70 msgr2=0x7fb10414f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1040cfe70 0x7fb10414f1b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009890 tx=0x7fb0f0004730 comp rx=0 tx=0).stop 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 shutdown_connections 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb0e406bc20 0x7fb0e406e0e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1040d0250 0x7fb10414f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb1040cfe70 0x7fb10414f1b0 secure :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009890 tx=0x7fb0f0004730 comp rx=0 tx=0).stop 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 --2- 172.21.15.102:0/1957334118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1040cc2f0 0x7fb1041582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 >> 172.21.15.102:0/1957334118 conn(0x7fb10408c3e0 msgr2=0x7fb1040cf460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 shutdown_connections 2024-03-18T16:34:13.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:13.979+0000 7fb103200640 1 -- 172.21.15.102:0/1957334118 wait complete. 2024-03-18T16:34:14.608 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:14.609 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:15.609 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:15.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.784+0000 7f9073200640 1 Processor -- start 2024-03-18T16:34:15.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.784+0000 7f9073200640 1 -- start start 2024-03-18T16:34:15.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.785+0000 7f9073200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f906c141650 0x7f906c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.785+0000 7f9073200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c15c3a0 0x7f906c0c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.785+0000 7f9073200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f906c158ad0 con 0x7f906c0c4fa0 2024-03-18T16:34:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.785+0000 7f9073200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f906c159370 con 0x7f906c141650 2024-03-18T16:34:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.785+0000 7f9073200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f906c158db0 con 0x7f906c15c3a0 2024-03-18T16:34:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.786+0000 7f9071400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c15c3a0 0x7f906c0c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:15.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.786+0000 7f9072800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f906c0c4fa0 0x7f906c06f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:38788/0 (socket says 172.21.15.102:38788) 2024-03-18T16:34:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.786+0000 7f9072800640 1 -- 172.21.15.102:0/3573440516 learned_addr learned my addr 172.21.15.102:0/3573440516 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:15.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.786+0000 7f9071e00640 1 --2- 172.21.15.102:0/3573440516 >> v2:172.21.15.102:3300/0 conn(0x7f906c141650 0x7f906c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.789+0000 7f9070a00640 1 -- 172.21.15.102:0/3573440516 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2374606866 0 0) 0x7f906c158ad0 con 0x7f906c0c4fa0 2024-03-18T16:34:15.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.789+0000 7f9070a00640 1 -- 172.21.15.102:0/3573440516 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9054003520 con 0x7f906c0c4fa0 2024-03-18T16:34:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.790+0000 7f9071400640 1 -- 172.21.15.102:0/3573440516 >> v2:172.21.15.102:3300/0 conn(0x7f906c141650 msgr2=0x7f906c158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.790+0000 7f9071400640 1 --2- 172.21.15.102:0/3573440516 >> v2:172.21.15.102:3300/0 conn(0x7f906c141650 0x7f906c158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:15.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.790+0000 7f9071400640 1 -- 172.21.15.102:0/3573440516 >> v1:172.21.15.154:6789/0 conn(0x7f906c0c4fa0 legacy=0x7f906c06f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:15.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.790+0000 7f9071400640 1 -- 172.21.15.102:0/3573440516 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906c159090 con 0x7f906c15c3a0 2024-03-18T16:34:15.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.793+0000 7f9071400640 1 --2- 172.21.15.102:0/3573440516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c15c3a0 0x7f906c0c6bc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f905c0097a0 tx=0x7f905c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7144b683e7440a96 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.797+0000 7f9070a00640 1 -- 172.21.15.102:0/3573440516 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c035020 con 0x7f906c15c3a0 2024-03-18T16:34:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.797+0000 7f9070a00640 1 -- 172.21.15.102:0/3573440516 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f905c031810 con 0x7f906c15c3a0 2024-03-18T16:34:15.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.797+0000 7f9070a00640 1 -- 172.21.15.102:0/3573440516 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c031be0 con 0x7f906c15c3a0 2024-03-18T16:34:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.798+0000 7f9073200640 1 -- 172.21.15.102:0/3573440516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c15c3a0 msgr2=0x7f906c0c6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.798+0000 7f9073200640 1 --2- 172.21.15.102:0/3573440516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c15c3a0 0x7f906c0c6bc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f905c0097a0 tx=0x7f905c02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:34:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.798+0000 7f9073200640 1 -- 172.21.15.102:0/3573440516 shutdown_connections 2024-03-18T16:34:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.798+0000 7f9073200640 1 --2- 172.21.15.102:0/3573440516 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c15c3a0 0x7f906c0c6bc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.798+0000 7f9073200640 1 --2- 172.21.15.102:0/3573440516 >> v2:172.21.15.102:3300/0 conn(0x7f906c141650 0x7f906c158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.798+0000 7f9073200640 1 -- 172.21.15.102:0/3573440516 >> 172.21.15.102:0/3573440516 conn(0x7f906c0cb890 msgr2=0x7f906c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.798+0000 7f9073200640 1 -- 172.21.15.102:0/3573440516 shutdown_connections 2024-03-18T16:34:15.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.798+0000 7f9073200640 1 -- 172.21.15.102:0/3573440516 wait complete. 2024-03-18T16:34:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.800+0000 7f9073200640 1 Processor -- start 2024-03-18T16:34:15.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.800+0000 7f9073200640 1 -- start start 2024-03-18T16:34:15.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.801+0000 7f9073200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c0c4fa0 0x7f906c1e7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.801+0000 7f9073200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c141650 0x7f906c13e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.801+0000 7f9073200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f906c15c3a0 0x7f906c13e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.801+0000 7f9073200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f906c15c7c0 con 0x7f906c15c3a0 2024-03-18T16:34:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.801+0000 7f9073200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f906c0c66a0 con 0x7f906c0c4fa0 2024-03-18T16:34:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.801+0000 7f9073200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f906c141a30 con 0x7f906c141650 2024-03-18T16:34:15.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.802+0000 7f9071400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c141650 0x7f906c13e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.802+0000 7f9071e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c0c4fa0 0x7f906c1e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.802+0000 7f9072800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f906c15c3a0 0x7f906c13e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.802+0000 7f9071400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c141650 0x7f906c13e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:37436/0 (socket says 172.21.15.102:37436) 2024-03-18T16:34:15.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.802+0000 7f9071400640 1 -- 172.21.15.102:0/567275715 learned_addr learned my addr 172.21.15.102:0/567275715 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.807+0000 7f9071400640 1 -- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c0c4fa0 msgr2=0x7f906c1e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:15.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.807+0000 7f9071400640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c0c4fa0 0x7f906c1e7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.807+0000 7f9071400640 1 -- 172.21.15.102:0/567275715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f906c15c3a0 msgr2=0x7f906c13e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.807+0000 7f9071400640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f906c15c3a0 0x7f906c13e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:15.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.807+0000 7f9071400640 1 -- 172.21.15.102:0/567275715 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906c1f1240 con 0x7f906c141650 2024-03-18T16:34:15.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.810+0000 7f9071400640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c141650 0x7f906c13e080 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f905c02bec0 tx=0x7f905c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.814+0000 7f9066000640 1 -- 172.21.15.102:0/567275715 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c035040 con 0x7f906c141650 2024-03-18T16:34:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.814+0000 7f9066000640 1 -- 172.21.15.102:0/567275715 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f905c039c80 con 0x7f906c141650 2024-03-18T16:34:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.814+0000 7f9066000640 1 -- 172.21.15.102:0/567275715 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905c043480 con 0x7f906c141650 2024-03-18T16:34:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.814+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906c15b010 con 0x7f906c141650 2024-03-18T16:34:15.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.815+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f906c1f0760 con 0x7f906c141650 2024-03-18T16:34:15.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:15.816+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f906c074fd0 con 0x7f906c141650 2024-03-18T16:34:16.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.089+0000 7f9066000640 1 -- 172.21.15.102:0/567275715 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f905c038020 con 0x7f906c141650 2024-03-18T16:34:16.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.091+0000 7f9066000640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f905006bc50 0x7f905006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:16.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.092+0000 7f9071e00640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f905006bc50 0x7f905006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:16.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.092+0000 7f9071e00640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f905006bc50 0x7f905006e110 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9060009b40 tx=0x7f9060009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:16.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.093+0000 7f9066000640 1 -- 172.21.15.102:0/567275715 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f905c078e40 con 0x7f906c141650 2024-03-18T16:34:16.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.112+0000 7f9066000640 1 -- 172.21.15.102:0/567275715 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f905c07ada0 con 0x7f906c141650 2024-03-18T16:34:16.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.355+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f906c1582b0 con 0x7f906c141650 2024-03-18T16:34:16.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.368+0000 7f9066000640 1 -- 172.21.15.102:0/567275715 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f905c07ee40 con 0x7f906c141650 2024-03-18T16:34:16.369 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:16.369 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:16.370 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:16.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f905006bc50 msgr2=0x7f905006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:16.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f905006bc50 0x7f905006e110 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9060009b40 tx=0x7f9060009190 comp rx=0 tx=0).stop 2024-03-18T16:34:16.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c141650 msgr2=0x7f906c13e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:16.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c141650 0x7f906c13e080 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f905c02bec0 tx=0x7f905c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:34:16.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 shutdown_connections 2024-03-18T16:34:16.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f905006bc50 0x7f905006e110 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:16.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f906c15c3a0 0x7f906c13e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:16.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f906c141650 0x7f906c13e080 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:16.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 --2- 172.21.15.102:0/567275715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f906c0c4fa0 0x7f906c1e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:16.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 >> 172.21.15.102:0/567275715 conn(0x7f906c0cb890 msgr2=0x7f906c140230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:16.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.370+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 shutdown_connections 2024-03-18T16:34:16.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:16.371+0000 7f9073200640 1 -- 172.21.15.102:0/567275715 wait complete. 2024-03-18T16:34:16.999 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:16.999 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:18.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.174+0000 7f9150a00640 1 Processor -- start 2024-03-18T16:34:18.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.174+0000 7f9150a00640 1 -- start start 2024-03-18T16:34:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.175+0000 7f9150a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f914c0cfe70 0x7f914c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.175+0000 7f9150a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f914c0cc2f0 0x7f914c0cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.175+0000 7f9150a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914c0ccc80 con 0x7f914c0d0250 2024-03-18T16:34:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.175+0000 7f9150a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f914c0cabd0 con 0x7f914c0cfe70 2024-03-18T16:34:18.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.175+0000 7f9150a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f914c0caea0 con 0x7f914c0cc2f0 2024-03-18T16:34:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.176+0000 7f914b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f914c0cfe70 0x7f914c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.176+0000 7f914be00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f914c0cc2f0 0x7f914c0cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.177+0000 7f914be00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f914c0cc2f0 0x7f914c0cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57260/0 (socket says 172.21.15.102:57260) 2024-03-18T16:34:18.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.177+0000 7f914be00640 1 -- 172.21.15.102:0/2879951276 learned_addr learned my addr 172.21.15.102:0/2879951276 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.179+0000 7f914be00640 1 -- 172.21.15.102:0/2879951276 >> v2:172.21.15.102:3300/0 conn(0x7f914c0cfe70 msgr2=0x7f914c15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.179+0000 7f914be00640 1 --2- 172.21.15.102:0/2879951276 >> v2:172.21.15.102:3300/0 conn(0x7f914c0cfe70 0x7f914c15ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.179+0000 7f914be00640 1 -- 172.21.15.102:0/2879951276 >> v1:172.21.15.154:6789/0 conn(0x7f914c0d0250 legacy=0x7f914c0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:18.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.180+0000 7f914be00640 1 -- 172.21.15.102:0/2879951276 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c0cb180 con 0x7f914c0cc2f0 2024-03-18T16:34:18.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.183+0000 7f914be00640 1 --2- 172.21.15.102:0/2879951276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f914c0cc2f0 0x7f914c0cc740 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f914c0ca160 tx=0x7f914402c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7510b89ae54e6e21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.187+0000 7f914a000640 1 -- 172.21.15.102:0/2879951276 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914402cbb0 con 0x7f914c0cc2f0 2024-03-18T16:34:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.187+0000 7f914a000640 1 -- 172.21.15.102:0/2879951276 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f914402cd50 con 0x7f914c0cc2f0 2024-03-18T16:34:18.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.187+0000 7f914a000640 1 -- 172.21.15.102:0/2879951276 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144034610 con 0x7f914c0cc2f0 2024-03-18T16:34:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.187+0000 7f9150a00640 1 -- 172.21.15.102:0/2879951276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f914c0cc2f0 msgr2=0x7f914c0cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.187+0000 7f9150a00640 1 --2- 172.21.15.102:0/2879951276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f914c0cc2f0 0x7f914c0cc740 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f914c0ca160 tx=0x7f914402c010 comp rx=0 tx=0).stop 2024-03-18T16:34:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.187+0000 7f9150a00640 1 -- 172.21.15.102:0/2879951276 shutdown_connections 2024-03-18T16:34:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.187+0000 7f9150a00640 1 --2- 172.21.15.102:0/2879951276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f914c0cc2f0 0x7f914c0cc740 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.187+0000 7f9150a00640 1 --2- 172.21.15.102:0/2879951276 >> v2:172.21.15.102:3300/0 conn(0x7f914c0cfe70 0x7f914c15ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:18.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.187+0000 7f9150a00640 1 -- 172.21.15.102:0/2879951276 >> 172.21.15.102:0/2879951276 conn(0x7f914c08c3e0 msgr2=0x7f914c06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.188+0000 7f9150a00640 1 -- 172.21.15.102:0/2879951276 shutdown_connections 2024-03-18T16:34:18.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.188+0000 7f9150a00640 1 -- 172.21.15.102:0/2879951276 wait complete. 2024-03-18T16:34:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.189+0000 7f9150a00640 1 Processor -- start 2024-03-18T16:34:18.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.190+0000 7f9150a00640 1 -- start start 2024-03-18T16:34:18.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.190+0000 7f9150a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f914c0cc2f0 0x7f914c1e6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:18.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.190+0000 7f9150a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f914c0cfe70 0x7f914c156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.190+0000 7f9150a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f914c0d0250 0x7f914c1f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.190+0000 7f9150a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f914c0cba20 con 0x7f914c0cc2f0 2024-03-18T16:34:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.190+0000 7f9150a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f914c0cb740 con 0x7f914c0d0250 2024-03-18T16:34:18.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.190+0000 7f9150a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f914c0cbd00 con 0x7f914c0cfe70 2024-03-18T16:34:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.191+0000 7f914aa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f914c0cfe70 0x7f914c156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.191+0000 7f914b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f914c0cc2f0 0x7f914c1e6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.191+0000 7f914be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f914c0d0250 0x7f914c1f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.192+0000 7f914aa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f914c0cfe70 0x7f914c156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:37448/0 (socket says 172.21.15.102:37448) 2024-03-18T16:34:18.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.192+0000 7f914aa00640 1 -- 172.21.15.102:0/492852990 learned_addr learned my addr 172.21.15.102:0/492852990 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.195+0000 7f914aa00640 1 -- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f914c0d0250 msgr2=0x7f914c1f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.195+0000 7f914aa00640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f914c0d0250 0x7f914c1f3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.195+0000 7f914aa00640 1 -- 172.21.15.102:0/492852990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f914c0cc2f0 msgr2=0x7f914c1e6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.195+0000 7f914aa00640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f914c0cc2f0 0x7f914c1e6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:18.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.195+0000 7f914aa00640 1 -- 172.21.15.102:0/492852990 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c1e58d0 con 0x7f914c0cfe70 2024-03-18T16:34:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.198+0000 7f914aa00640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f914c0cfe70 0x7f914c156300 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f913800c470 tx=0x7f913800c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.201+0000 7f9143400640 1 -- 172.21.15.102:0/492852990 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91380052d0 con 0x7f914c0cfe70 2024-03-18T16:34:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.201+0000 7f9143400640 1 -- 172.21.15.102:0/492852990 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9138005490 con 0x7f914c0cfe70 2024-03-18T16:34:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.201+0000 7f9143400640 1 -- 172.21.15.102:0/492852990 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9138007260 con 0x7f914c0cfe70 2024-03-18T16:34:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.201+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914c1e5b40 con 0x7f914c0cfe70 2024-03-18T16:34:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.202+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f914c1e6100 con 0x7f914c0cfe70 2024-03-18T16:34:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.203+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f914c0cf860 con 0x7f914c0cfe70 2024-03-18T16:34:18.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.222+0000 7f9143400640 1 -- 172.21.15.102:0/492852990 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9138007400 con 0x7f914c0cfe70 2024-03-18T16:34:18.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.223+0000 7f9143400640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f912c06bc20 0x7f912c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:18.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.224+0000 7f914b400640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f912c06bc20 0x7f912c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:18.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.224+0000 7f914b400640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f912c06bc20 0x7f912c06e0e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f913c009b40 tx=0x7f913c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:18.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.225+0000 7f9143400640 1 -- 172.21.15.102:0/492852990 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9138056be0 con 0x7f914c0cfe70 2024-03-18T16:34:18.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.482+0000 7f9143400640 1 -- 172.21.15.102:0/492852990 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f914c0cf860 con 0x7f914c0cfe70 2024-03-18T16:34:18.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.758+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f914c15ae20 con 0x7f914c0cfe70 2024-03-18T16:34:18.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.773+0000 7f9143400640 1 -- 172.21.15.102:0/492852990 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f914c15ae20 con 0x7f914c0cfe70 2024-03-18T16:34:18.774 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:18.775 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:18.775 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:18.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f912c06bc20 msgr2=0x7f912c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:18.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f912c06bc20 0x7f912c06e0e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f913c009b40 tx=0x7f913c009190 comp rx=0 tx=0).stop 2024-03-18T16:34:18.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f914c0cfe70 msgr2=0x7f914c156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:18.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f914c0cfe70 0x7f914c156300 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f913800c470 tx=0x7f913800c930 comp rx=0 tx=0).stop 2024-03-18T16:34:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 shutdown_connections 2024-03-18T16:34:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f912c06bc20 0x7f912c06e0e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f914c0d0250 0x7f914c1f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f914c0cfe70 0x7f914c156300 secure :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f913800c470 tx=0x7f913800c930 comp rx=0 tx=0).stop 2024-03-18T16:34:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 --2- 172.21.15.102:0/492852990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f914c0cc2f0 0x7f914c1e6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.775+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 >> 172.21.15.102:0/492852990 conn(0x7f914c08c3e0 msgr2=0x7f914c0cef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.776+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 shutdown_connections 2024-03-18T16:34:18.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:18.776+0000 7f9150a00640 1 -- 172.21.15.102:0/492852990 wait complete. 2024-03-18T16:34:19.399 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:19.399 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:20.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:20.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.573+0000 7f375b200640 1 Processor -- start 2024-03-18T16:34:20.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.573+0000 7f375b200640 1 -- start start 2024-03-18T16:34:20.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.575+0000 7f375b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c141650 0x7f375c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:20.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.575+0000 7f375b200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f375c15c3a0 0x7f375c0c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:20.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.575+0000 7f375b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f375c158ad0 con 0x7f375c0c4fa0 2024-03-18T16:34:20.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.575+0000 7f375b200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f375c159370 con 0x7f375c15c3a0 2024-03-18T16:34:20.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.575+0000 7f375b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f375c158db0 con 0x7f375c141650 2024-03-18T16:34:20.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.575+0000 7f3759e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c141650 0x7f375c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:20.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.576+0000 7f3759400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f375c15c3a0 0x7f375c0c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:20.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.576+0000 7f375a800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f375c0c4fa0 0x7f375c06f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33742/0 (socket says 172.21.15.102:33742) 2024-03-18T16:34:20.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.576+0000 7f375a800640 1 -- 172.21.15.102:0/232135089 learned_addr learned my addr 172.21.15.102:0/232135089 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:20.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.578+0000 7f3759400640 1 -- 172.21.15.102:0/232135089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c141650 msgr2=0x7f375c158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:20.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.578+0000 7f3759400640 1 --2- 172.21.15.102:0/232135089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c141650 0x7f375c158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:20.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.578+0000 7f3759400640 1 -- 172.21.15.102:0/232135089 >> v1:172.21.15.102:6789/0 conn(0x7f375c0c4fa0 legacy=0x7f375c06f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:20.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.578+0000 7f3759400640 1 -- 172.21.15.102:0/232135089 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375c159090 con 0x7f375c15c3a0 2024-03-18T16:34:20.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.581+0000 7f3759400640 1 --2- 172.21.15.102:0/232135089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c15c3a0 0x7f375c0c6bc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f37480097a0 tx=0x7f374802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b6065f83e0d07a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:20.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.585+0000 7f3758a00640 1 -- 172.21.15.102:0/232135089 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748035020 con 0x7f375c15c3a0 2024-03-18T16:34:20.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.585+0000 7f3758a00640 1 -- 172.21.15.102:0/232135089 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3748031810 con 0x7f375c15c3a0 2024-03-18T16:34:20.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.585+0000 7f3758a00640 1 -- 172.21.15.102:0/232135089 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748031be0 con 0x7f375c15c3a0 2024-03-18T16:34:20.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.586+0000 7f375b200640 1 -- 172.21.15.102:0/232135089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c15c3a0 msgr2=0x7f375c0c6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:20.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.586+0000 7f375b200640 1 --2- 172.21.15.102:0/232135089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c15c3a0 0x7f375c0c6bc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f37480097a0 tx=0x7f374802b8b0 comp rx=0 tx=0).stop 2024-03-18T16:34:20.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.586+0000 7f375b200640 1 -- 172.21.15.102:0/232135089 shutdown_connections 2024-03-18T16:34:20.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.586+0000 7f375b200640 1 --2- 172.21.15.102:0/232135089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c15c3a0 0x7f375c0c6bc0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:20.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.586+0000 7f375b200640 1 --2- 172.21.15.102:0/232135089 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c141650 0x7f375c158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:20.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.586+0000 7f375b200640 1 -- 172.21.15.102:0/232135089 >> 172.21.15.102:0/232135089 conn(0x7f375c0cb890 msgr2=0x7f375c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:20.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.586+0000 7f375b200640 1 -- 172.21.15.102:0/232135089 shutdown_connections 2024-03-18T16:34:20.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.586+0000 7f375b200640 1 -- 172.21.15.102:0/232135089 wait complete. 2024-03-18T16:34:20.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.588+0000 7f375b200640 1 Processor -- start 2024-03-18T16:34:20.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.588+0000 7f375b200640 1 -- start start 2024-03-18T16:34:20.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.589+0000 7f375b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f375c0c4fa0 0x7f375c159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:20.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.589+0000 7f375b200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c141650 0x7f375c1573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:20.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.589+0000 7f375b200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c15c3a0 0x7f375c157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:20.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.589+0000 7f375b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f375c15c7c0 con 0x7f375c0c4fa0 2024-03-18T16:34:20.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.589+0000 7f375b200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f375c0c66a0 con 0x7f375c141650 2024-03-18T16:34:20.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.589+0000 7f375b200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f375c141a30 con 0x7f375c15c3a0 2024-03-18T16:34:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.590+0000 7f375a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c15c3a0 0x7f375c157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.590+0000 7f3759400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c141650 0x7f375c1573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.590+0000 7f3759e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f375c0c4fa0 0x7f375c159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.590+0000 7f375a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c15c3a0 0x7f375c157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:37472/0 (socket says 172.21.15.102:37472) 2024-03-18T16:34:20.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.590+0000 7f375a800640 1 -- 172.21.15.102:0/2588591983 learned_addr learned my addr 172.21.15.102:0/2588591983 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:20.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.593+0000 7f3759400640 1 -- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c15c3a0 msgr2=0x7f375c157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:20.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.593+0000 7f3759400640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c15c3a0 0x7f375c157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:20.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.593+0000 7f3759400640 1 -- 172.21.15.102:0/2588591983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f375c0c4fa0 msgr2=0x7f375c159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:20.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.593+0000 7f3759400640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f375c0c4fa0 0x7f375c159d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:20.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.593+0000 7f3759400640 1 -- 172.21.15.102:0/2588591983 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375c1530c0 con 0x7f375c141650 2024-03-18T16:34:20.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.596+0000 7f3759400640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c141650 0x7f375c1573d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3748033000 tx=0x7f37480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:20.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.600+0000 7f3752000640 1 -- 172.21.15.102:0/2588591983 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748040070 con 0x7f375c141650 2024-03-18T16:34:20.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.600+0000 7f3752000640 1 -- 172.21.15.102:0/2588591983 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3748039cf0 con 0x7f375c141650 2024-03-18T16:34:20.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.600+0000 7f3752000640 1 -- 172.21.15.102:0/2588591983 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748035040 con 0x7f375c141650 2024-03-18T16:34:20.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.600+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375c0c43c0 con 0x7f375c141650 2024-03-18T16:34:20.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.600+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f375c1e5870 con 0x7f375c141650 2024-03-18T16:34:20.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.602+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f375c0b7f00 con 0x7f375c141650 2024-03-18T16:34:20.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.880+0000 7f3752000640 1 -- 172.21.15.102:0/2588591983 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3748038020 con 0x7f375c141650 2024-03-18T16:34:20.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.882+0000 7f3752000640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f373c06bc50 0x7f373c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:20.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.883+0000 7f3759e00640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f373c06bc50 0x7f373c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:20.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.883+0000 7f3759e00640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f373c06bc50 0x7f373c06e110 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f374c009b40 tx=0x7f374c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:20.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.884+0000 7f3752000640 1 -- 172.21.15.102:0/2588591983 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3748078ef0 con 0x7f375c141650 2024-03-18T16:34:20.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:20.895+0000 7f3752000640 1 -- 172.21.15.102:0/2588591983 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f374807ae50 con 0x7f375c141650 2024-03-18T16:34:21.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.188+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f375c0734f0 con 0x7f375c141650 2024-03-18T16:34:21.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.202+0000 7f3752000640 1 -- 172.21.15.102:0/2588591983 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f374807eef0 con 0x7f375c141650 2024-03-18T16:34:21.203 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:21.203 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:21.204 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f373c06bc50 msgr2=0x7f373c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f373c06bc50 0x7f373c06e110 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f374c009b40 tx=0x7f374c009190 comp rx=0 tx=0).stop 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c141650 msgr2=0x7f375c1573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c141650 0x7f375c1573d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3748033000 tx=0x7f37480046c0 comp rx=0 tx=0).stop 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 shutdown_connections 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f373c06bc50 0x7f373c06e110 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f375c15c3a0 0x7f375c157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f375c141650 0x7f375c1573d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 --2- 172.21.15.102:0/2588591983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f375c0c4fa0 0x7f375c159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:21.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 >> 172.21.15.102:0/2588591983 conn(0x7f375c0cb890 msgr2=0x7f375c140820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:21.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.204+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 shutdown_connections 2024-03-18T16:34:21.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:21.205+0000 7f375b200640 1 -- 172.21.15.102:0/2588591983 wait complete. 2024-03-18T16:34:21.827 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:21.827 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:22.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:23.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.002+0000 7f3d17200640 1 Processor -- start 2024-03-18T16:34:23.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.002+0000 7f3d17200640 1 -- start start 2024-03-18T16:34:23.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.003+0000 7f3d17200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3d1815a6f0 0x7f3d1815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:23.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.003+0000 7f3d17200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d1815baa0 0x7f3d1815bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:23.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.003+0000 7f3d17200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d180ca500 con 0x7f3d1815a310 2024-03-18T16:34:23.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.003+0000 7f3d17200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3d180cd410 con 0x7f3d1815a6f0 2024-03-18T16:34:23.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.003+0000 7f3d17200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3d1815b080 con 0x7f3d1815baa0 2024-03-18T16:34:23.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.007+0000 7f3d16800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d1815baa0 0x7f3d1815bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:23.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.008+0000 7f3d16800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d1815baa0 0x7f3d1815bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36442/0 (socket says 172.21.15.102:36442) 2024-03-18T16:34:23.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.008+0000 7f3d16800640 1 -- 172.21.15.102:0/2978533918 learned_addr learned my addr 172.21.15.102:0/2978533918 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:23.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.033+0000 7f3d16800640 1 -- 172.21.15.102:0/2978533918 >> v2:172.21.15.102:3300/0 conn(0x7f3d1815a6f0 msgr2=0x7f3d1815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:23.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.033+0000 7f3d16800640 1 --2- 172.21.15.102:0/2978533918 >> v2:172.21.15.102:3300/0 conn(0x7f3d1815a6f0 0x7f3d1815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.033+0000 7f3d16800640 1 -- 172.21.15.102:0/2978533918 >> v1:172.21.15.102:6789/0 conn(0x7f3d1815a310 legacy=0x7f3d1806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:23.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.033+0000 7f3d16800640 1 -- 172.21.15.102:0/2978533918 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1815c430 con 0x7f3d1815baa0 2024-03-18T16:34:23.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.040+0000 7f3d16800640 1 --2- 172.21.15.102:0/2978533918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815baa0 0x7f3d1815bef0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3d180d0c80 tx=0x7f3d1002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df3b9622051082f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.048+0000 7f3d14a00640 1 -- 172.21.15.102:0/2978533918 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1002ca80 con 0x7f3d1815baa0 2024-03-18T16:34:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.048+0000 7f3d14a00640 1 -- 172.21.15.102:0/2978533918 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1002cc20 con 0x7f3d1815baa0 2024-03-18T16:34:23.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.048+0000 7f3d14a00640 1 -- 172.21.15.102:0/2978533918 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d100343f0 con 0x7f3d1815baa0 2024-03-18T16:34:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.049+0000 7f3d17200640 1 -- 172.21.15.102:0/2978533918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815baa0 msgr2=0x7f3d1815bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.049+0000 7f3d17200640 1 --2- 172.21.15.102:0/2978533918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815baa0 0x7f3d1815bef0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3d180d0c80 tx=0x7f3d1002bee0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.050+0000 7f3d17200640 1 -- 172.21.15.102:0/2978533918 shutdown_connections 2024-03-18T16:34:23.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.050+0000 7f3d17200640 1 --2- 172.21.15.102:0/2978533918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815baa0 0x7f3d1815bef0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.050+0000 7f3d17200640 1 --2- 172.21.15.102:0/2978533918 >> v2:172.21.15.102:3300/0 conn(0x7f3d1815a6f0 0x7f3d1815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.050+0000 7f3d17200640 1 -- 172.21.15.102:0/2978533918 >> 172.21.15.102:0/2978533918 conn(0x7f3d1808c1a0 msgr2=0x7f3d18066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.050+0000 7f3d17200640 1 -- 172.21.15.102:0/2978533918 shutdown_connections 2024-03-18T16:34:23.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.050+0000 7f3d17200640 1 -- 172.21.15.102:0/2978533918 wait complete. 2024-03-18T16:34:23.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.052+0000 7f3d17200640 1 Processor -- start 2024-03-18T16:34:23.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.053+0000 7f3d17200640 1 -- start start 2024-03-18T16:34:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.054+0000 7f3d17200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d1815a310 0x7f3d181f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.054+0000 7f3d17200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d1815a6f0 0x7f3d180c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.054+0000 7f3d17200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815baa0 0x7f3d18150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.054+0000 7f3d17200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3d180cce50 con 0x7f3d1815a6f0 2024-03-18T16:34:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.054+0000 7f3d17200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d180caa80 con 0x7f3d1815baa0 2024-03-18T16:34:23.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.054+0000 7f3d17200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3d180cd130 con 0x7f3d1815a310 2024-03-18T16:34:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.055+0000 7f3d15e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d1815a310 0x7f3d181f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.055+0000 7f3d16800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815baa0 0x7f3d18150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.055+0000 7f3d15e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d1815a310 0x7f3d181f0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:36786/0 (socket says 172.21.15.102:36786) 2024-03-18T16:34:23.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.055+0000 7f3d15e00640 1 -- 172.21.15.102:0/3247872978 learned_addr learned my addr 172.21.15.102:0/3247872978 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:23.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.060+0000 7f3d15e00640 1 -- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815baa0 msgr2=0x7f3d18150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:23.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.060+0000 7f3d15e00640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815baa0 0x7f3d18150100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.060+0000 7f3d15e00640 1 -- 172.21.15.102:0/3247872978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d1815a6f0 msgr2=0x7f3d180c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.060+0000 7f3d15e00640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d1815a6f0 0x7f3d180c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.060+0000 7f3d15e00640 1 -- 172.21.15.102:0/3247872978 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d180c3850 con 0x7f3d1815a310 2024-03-18T16:34:23.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.062+0000 7f3d15e00640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d1815a310 0x7f3d181f0f40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800c440 tx=0x7f3d0800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.066+0000 7f3d0e000640 1 -- 172.21.15.102:0/3247872978 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08017070 con 0x7f3d1815a310 2024-03-18T16:34:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.066+0000 7f3d0e000640 1 -- 172.21.15.102:0/3247872978 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d080035f0 con 0x7f3d1815a310 2024-03-18T16:34:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.066+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d180c3af0 con 0x7f3d1815a310 2024-03-18T16:34:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.066+0000 7f3d0e000640 1 -- 172.21.15.102:0/3247872978 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d080039e0 con 0x7f3d1815a310 2024-03-18T16:34:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.067+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d180c40b0 con 0x7f3d1815a310 2024-03-18T16:34:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.068+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d18074fd0 con 0x7f3d1815a310 2024-03-18T16:34:23.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.088+0000 7f3d0e000640 1 -- 172.21.15.102:0/3247872978 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3d0801a070 con 0x7f3d1815a310 2024-03-18T16:34:23.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.089+0000 7f3d0e000640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cf806bc20 0x7f3cf806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:23.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.089+0000 7f3d15400640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cf806bc20 0x7f3cf806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:23.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.090+0000 7f3d15400640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cf806bc20 0x7f3cf806e0e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3d1815b5b0 tx=0x7f3d04006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:23.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.326+0000 7f3d0e000640 1 -- 172.21.15.102:0/3247872978 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3d08056540 con 0x7f3d1815a310 2024-03-18T16:34:23.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.337+0000 7f3d0e000640 1 -- 172.21.15.102:0/3247872978 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3d08021080 con 0x7f3d1815a310 2024-03-18T16:34:23.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.574+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d180caed0 con 0x7f3d1815a310 2024-03-18T16:34:23.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.589+0000 7f3d0e000640 1 -- 172.21.15.102:0/3247872978 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3d080584a0 con 0x7f3d1815a310 2024-03-18T16:34:23.591 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:23.591 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:23.591 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.592+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cf806bc20 msgr2=0x7f3cf806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.592+0000 7f3d17200640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cf806bc20 0x7f3cf806e0e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3d1815b5b0 tx=0x7f3d04006c90 comp rx=0 tx=0).stop 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.592+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d1815a310 msgr2=0x7f3d181f0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.593+0000 7f3d17200640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d1815a310 0x7f3d181f0f40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800c440 tx=0x7f3d0800c900 comp rx=0 tx=0).stop 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.593+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 shutdown_connections 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.593+0000 7f3d17200640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3cf806bc20 0x7f3cf806e0e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.593+0000 7f3d17200640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d1815baa0 0x7f3d18150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.593+0000 7f3d17200640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d1815a6f0 0x7f3d180c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.593+0000 7f3d17200640 1 --2- 172.21.15.102:0/3247872978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3d1815a310 0x7f3d181f0f40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.593+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 >> 172.21.15.102:0/3247872978 conn(0x7f3d1808c1a0 msgr2=0x7f3d1815d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.593+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 shutdown_connections 2024-03-18T16:34:23.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:23.593+0000 7f3d17200640 1 -- 172.21.15.102:0/3247872978 wait complete. 2024-03-18T16:34:23.923 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:34:23.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:34:24.223 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:24.223 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:25.224 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:25.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.426+0000 7f4d39400640 1 Processor -- start 2024-03-18T16:34:25.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.426+0000 7f4d39400640 1 -- start start 2024-03-18T16:34:25.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.427+0000 7f4d39400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4d340ca960 0x7f4d3406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:25.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.427+0000 7f4d39400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4d340cad40 0x7f4d3415e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:25.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.428+0000 7f4d39400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d3415e670 con 0x7f4d3415a1a0 2024-03-18T16:34:25.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.428+0000 7f4d39400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4d340c8680 con 0x7f4d340ca960 2024-03-18T16:34:25.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.428+0000 7f4d39400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4d34070cb0 con 0x7f4d340cad40 2024-03-18T16:34:25.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.428+0000 7f4d33400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4d340cad40 0x7f4d3415e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:25.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.429+0000 7f4d33400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4d340cad40 0x7f4d3415e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36460/0 (socket says 172.21.15.102:36460) 2024-03-18T16:34:25.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.429+0000 7f4d33400640 1 -- 172.21.15.102:0/313881577 learned_addr learned my addr 172.21.15.102:0/313881577 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:25.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.432+0000 7f4d33400640 1 -- 172.21.15.102:0/313881577 >> v2:172.21.15.102:3300/0 conn(0x7f4d340ca960 msgr2=0x7f4d3406c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:25.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.432+0000 7f4d33400640 1 --2- 172.21.15.102:0/313881577 >> v2:172.21.15.102:3300/0 conn(0x7f4d340ca960 0x7f4d3406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:25.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.432+0000 7f4d33400640 1 -- 172.21.15.102:0/313881577 >> v1:172.21.15.102:6789/0 conn(0x7f4d3415a1a0 legacy=0x7f4d340705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:25.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.432+0000 7f4d33400640 1 -- 172.21.15.102:0/313881577 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3406c9c0 con 0x7f4d340cad40 2024-03-18T16:34:25.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.434+0000 7f4d33400640 1 --2- 172.21.15.102:0/313881577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d340cad40 0x7f4d3415e130 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4d340cb220 tx=0x7f4d2002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3508e8ea63c40cfc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:25.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.440+0000 7f4d32a00640 1 -- 172.21.15.102:0/313881577 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20034070 con 0x7f4d340cad40 2024-03-18T16:34:25.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.440+0000 7f4d32a00640 1 -- 172.21.15.102:0/313881577 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d20030860 con 0x7f4d340cad40 2024-03-18T16:34:25.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.440+0000 7f4d32a00640 1 -- 172.21.15.102:0/313881577 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20030c30 con 0x7f4d340cad40 2024-03-18T16:34:25.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.441+0000 7f4d39400640 1 -- 172.21.15.102:0/313881577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d340cad40 msgr2=0x7f4d3415e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:25.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.441+0000 7f4d39400640 1 --2- 172.21.15.102:0/313881577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d340cad40 0x7f4d3415e130 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4d340cb220 tx=0x7f4d2002b9a0 comp rx=0 tx=0).stop 2024-03-18T16:34:25.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.441+0000 7f4d39400640 1 -- 172.21.15.102:0/313881577 shutdown_connections 2024-03-18T16:34:25.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.441+0000 7f4d39400640 1 --2- 172.21.15.102:0/313881577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d340cad40 0x7f4d3415e130 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:25.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.441+0000 7f4d39400640 1 --2- 172.21.15.102:0/313881577 >> v2:172.21.15.102:3300/0 conn(0x7f4d340ca960 0x7f4d3406c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:25.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.441+0000 7f4d39400640 1 -- 172.21.15.102:0/313881577 >> 172.21.15.102:0/313881577 conn(0x7f4d34067190 msgr2=0x7f4d340675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:25.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.441+0000 7f4d39400640 1 -- 172.21.15.102:0/313881577 shutdown_connections 2024-03-18T16:34:25.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.441+0000 7f4d39400640 1 -- 172.21.15.102:0/313881577 wait complete. 2024-03-18T16:34:25.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.443+0000 7f4d39400640 1 Processor -- start 2024-03-18T16:34:25.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.443+0000 7f4d39400640 1 -- start start 2024-03-18T16:34:25.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.444+0000 7f4d39400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d340ca960 0x7f4d340b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:25.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.444+0000 7f4d39400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d340cad40 0x7f4d3413e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:25.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.444+0000 7f4d39400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d3415a1a0 0x7f4d34071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:25.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.444+0000 7f4d39400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3415d930 con 0x7f4d340cad40 2024-03-18T16:34:25.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.444+0000 7f4d39400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3406cf80 con 0x7f4d3415a1a0 2024-03-18T16:34:25.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.444+0000 7f4d39400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3415dc10 con 0x7f4d340ca960 2024-03-18T16:34:25.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.445+0000 7f4d33e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d340ca960 0x7f4d340b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:25.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.445+0000 7f4d38a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d3415a1a0 0x7f4d34071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:25.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.445+0000 7f4d33400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d340cad40 0x7f4d3413e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:25.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.445+0000 7f4d38a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d3415a1a0 0x7f4d34071370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36464/0 (socket says 172.21.15.102:36464) 2024-03-18T16:34:25.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.445+0000 7f4d38a00640 1 -- 172.21.15.102:0/638448914 learned_addr learned my addr 172.21.15.102:0/638448914 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:25.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.449+0000 7f4d38a00640 1 -- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d340ca960 msgr2=0x7f4d340b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:25.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.449+0000 7f4d38a00640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d340ca960 0x7f4d340b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:25.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.449+0000 7f4d38a00640 1 -- 172.21.15.102:0/638448914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d340cad40 msgr2=0x7f4d3413e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:25.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.449+0000 7f4d38a00640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d340cad40 0x7f4d3413e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:25.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.449+0000 7f4d38a00640 1 -- 172.21.15.102:0/638448914 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d34140d30 con 0x7f4d3415a1a0 2024-03-18T16:34:25.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.449+0000 7f4d33e00640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d340ca960 0x7f4d340b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:34:25.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.451+0000 7f4d38a00640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d3415a1a0 0x7f4d34071370 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00b990 tx=0x7f4d2c00bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:25.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.455+0000 7f4d2be00640 1 -- 172.21.15.102:0/638448914 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c019070 con 0x7f4d3415a1a0 2024-03-18T16:34:25.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.455+0000 7f4d2be00640 1 -- 172.21.15.102:0/638448914 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d2c00ca40 con 0x7f4d3415a1a0 2024-03-18T16:34:25.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.455+0000 7f4d2be00640 1 -- 172.21.15.102:0/638448914 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c014440 con 0x7f4d3415a1a0 2024-03-18T16:34:25.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.455+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d340cb7d0 con 0x7f4d3415a1a0 2024-03-18T16:34:25.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.455+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d340cbd90 con 0x7f4d3415a1a0 2024-03-18T16:34:25.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.456+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d34074fd0 con 0x7f4d3415a1a0 2024-03-18T16:34:25.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.476+0000 7f4d2be00640 1 -- 172.21.15.102:0/638448914 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4d2c01c070 con 0x7f4d3415a1a0 2024-03-18T16:34:25.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.477+0000 7f4d2be00640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d1406bbd0 0x7f4d1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:25.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.478+0000 7f4d33e00640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d1406bbd0 0x7f4d1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:25.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.479+0000 7f4d33e00640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d1406bbd0 0x7f4d1406e090 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009d60 tx=0x7f4d24009110 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:25.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.481+0000 7f4d2be00640 1 -- 172.21.15.102:0/638448914 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f4d2c057080 con 0x7f4d3415a1a0 2024-03-18T16:34:25.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.732+0000 7f4d2be00640 1 -- 172.21.15.102:0/638448914 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4d2c023020 con 0x7f4d3415a1a0 2024-03-18T16:34:25.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.969+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d3406c6d0 con 0x7f4d3415a1a0 2024-03-18T16:34:25.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.983+0000 7f4d2be00640 1 -- 172.21.15.102:0/638448914 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f4d2c058fe0 con 0x7f4d3415a1a0 2024-03-18T16:34:25.985 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:25.985 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:25.985 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:25.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d1406bbd0 msgr2=0x7f4d1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d1406bbd0 0x7f4d1406e090 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009d60 tx=0x7f4d24009110 comp rx=0 tx=0).stop 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d3415a1a0 msgr2=0x7f4d34071370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d3415a1a0 0x7f4d34071370 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c00b990 tx=0x7f4d2c00bd90 comp rx=0 tx=0).stop 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 shutdown_connections 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d1406bbd0 0x7f4d1406e090 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4d24009d60 tx=0x7f4d24009110 comp rx=0 tx=0).stop 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d3415a1a0 0x7f4d34071370 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d340cad40 0x7f4d3413e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 --2- 172.21.15.102:0/638448914 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d340ca960 0x7f4d340b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 >> 172.21.15.102:0/638448914 conn(0x7f4d34067190 msgr2=0x7f4d34070250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 shutdown_connections 2024-03-18T16:34:25.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:25.986+0000 7f4d39400640 1 -- 172.21.15.102:0/638448914 wait complete. 2024-03-18T16:34:26.610 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:26.610 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:27.612 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:27.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.786+0000 7f7416600640 1 Processor -- start 2024-03-18T16:34:27.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.786+0000 7f7416600640 1 -- start start 2024-03-18T16:34:27.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.787+0000 7f7416600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7410158420 0x7f741015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:27.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.787+0000 7f7416600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74100c72d0 con 0x7f74100c4fa0 2024-03-18T16:34:27.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.787+0000 7f7416600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7410158840 con 0x7f7410141650 2024-03-18T16:34:27.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.787+0000 7f7416600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f741015c8e0 con 0x7f7410158420 2024-03-18T16:34:27.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.788+0000 7f7415200640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f7410141650 0x7f74100c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:50346/0 (socket says 172.21.15.102:50346) 2024-03-18T16:34:27.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.788+0000 7f7415200640 1 -- 172.21.15.102:0/2607995182 learned_addr learned my addr 172.21.15.102:0/2607995182 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:27.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.788+0000 7f740fe00640 1 --2- 172.21.15.102:0/2607995182 >> v2:172.21.15.102:3300/0 conn(0x7f7410158420 0x7f741015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:27.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.791+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3572651220 0 0) 0x7f7410158840 con 0x7f7410141650 2024-03-18T16:34:27.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.791+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73f8003520 con 0x7f7410141650 2024-03-18T16:34:27.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2573339139 0 0) 0x7f73f8003520 con 0x7f7410141650 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 >> v2:172.21.15.102:3300/0 conn(0x7f7410158420 msgr2=0x7f741015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 --2- 172.21.15.102:0/2607995182 >> v2:172.21.15.102:3300/0 conn(0x7f7410158420 0x7f741015c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 >> v1:172.21.15.102:6789/0 conn(0x7f74100c4fa0 legacy=0x7f7410162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7410158ad0 con 0x7f7410141650 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f7404001670 con 0x7f7410141650 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 >> v1:172.21.15.154:6789/0 conn(0x7f7410141650 legacy=0x7f74100c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73f8003a00 0x7f73f8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 0x7f73f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80051c0 0x7f73f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f741015c8e0 con 0x7f73f8003a00 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73f800c0c0 con 0x7f73f80051c0 2024-03-18T16:34:27.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.795+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f73f800c2c0 con 0x7f73f8004800 2024-03-18T16:34:27.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.796+0000 7f740fe00640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 0x7f73f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:27.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.799+0000 7f7415c00640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80051c0 0x7f73f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:27.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.799+0000 7f740fe00640 1 -- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80051c0 msgr2=0x7f73f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:27.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.799+0000 7f740fe00640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80051c0 0x7f73f800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:27.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.799+0000 7f740fe00640 1 -- 172.21.15.102:0/2607995182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73f8003a00 msgr2=0x7f73f8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:27.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.799+0000 7f740fe00640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73f8003a00 0x7f73f8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:27.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.799+0000 7f740fe00640 1 -- 172.21.15.102:0/2607995182 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f8003520 con 0x7f73f8004800 2024-03-18T16:34:27.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.799+0000 7f7415c00640 1 -- 172.21.15.102:0/2607995182 reap_dead start 2024-03-18T16:34:27.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.801+0000 7f740fe00640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 0x7f73f8004c80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f73fc009ac0 tx=0x7f73f8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d65cc76370e2df9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:27.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.805+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc039070 con 0x7f73f8004800 2024-03-18T16:34:27.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.805+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73fc02c6d0 con 0x7f73f8004800 2024-03-18T16:34:27.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.805+0000 7f740f400640 1 -- 172.21.15.102:0/2607995182 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc009690 con 0x7f73f8004800 2024-03-18T16:34:27.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.806+0000 7f7416600640 1 -- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 msgr2=0x7f73f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:27.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.806+0000 7f7416600640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 0x7f73f8004c80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f73fc009ac0 tx=0x7f73f8005830 comp rx=0 tx=0).stop 2024-03-18T16:34:27.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.806+0000 7f7416600640 1 -- 172.21.15.102:0/2607995182 shutdown_connections 2024-03-18T16:34:27.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.806+0000 7f7416600640 1 --2- 172.21.15.102:0/2607995182 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 0x7f73f8004c80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:27.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.806+0000 7f7416600640 1 -- 172.21.15.102:0/2607995182 >> 172.21.15.102:0/2607995182 conn(0x7f74100cb890 msgr2=0x7f74100cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:27.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.806+0000 7f7416600640 1 -- 172.21.15.102:0/2607995182 shutdown_connections 2024-03-18T16:34:27.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.807+0000 7f7416600640 1 -- 172.21.15.102:0/2607995182 wait complete. 2024-03-18T16:34:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.809+0000 7f7416600640 1 Processor -- start 2024-03-18T16:34:27.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.809+0000 7f7416600640 1 -- start start 2024-03-18T16:34:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.811+0000 7f7416600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 0x7f74101597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.811+0000 7f7416600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7410136430 0x7f741015adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:27.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.811+0000 7f7416600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7410137220 0x7f741015b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:27.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.811+0000 7f7416600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7410159090 con 0x7f7410137220 2024-03-18T16:34:27.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.811+0000 7f7416600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7410159370 con 0x7f7410136430 2024-03-18T16:34:27.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.811+0000 7f7416600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f74100c63c0 con 0x7f73f8004800 2024-03-18T16:34:27.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.812+0000 7f740fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7410136430 0x7f741015adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:27.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.812+0000 7f740fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7410136430 0x7f741015adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36484/0 (socket says 172.21.15.102:36484) 2024-03-18T16:34:27.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.812+0000 7f740fe00640 1 -- 172.21.15.102:0/953750086 learned_addr learned my addr 172.21.15.102:0/953750086 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:27.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.812+0000 7f7415c00640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7410137220 0x7f741015b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:27.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.813+0000 7f7415200640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 0x7f74101597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:27.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.817+0000 7f740fe00640 1 -- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 msgr2=0x7f74101597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:27.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.817+0000 7f740fe00640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 0x7f74101597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:27.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.817+0000 7f740fe00640 1 -- 172.21.15.102:0/953750086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7410137220 msgr2=0x7f741015b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:27.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.817+0000 7f740fe00640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7410137220 0x7f741015b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:27.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.817+0000 7f740fe00640 1 -- 172.21.15.102:0/953750086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7410138720 con 0x7f7410136430 2024-03-18T16:34:27.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.821+0000 7f740fe00640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7410136430 0x7f741015adb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f73fc009be0 tx=0x7f73fc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:27.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.825+0000 7f740cc00640 1 -- 172.21.15.102:0/953750086 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc042020 con 0x7f7410136430 2024-03-18T16:34:27.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.826+0000 7f740cc00640 1 -- 172.21.15.102:0/953750086 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73fc03a040 con 0x7f7410136430 2024-03-18T16:34:27.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.826+0000 7f740cc00640 1 -- 172.21.15.102:0/953750086 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc039070 con 0x7f7410136430 2024-03-18T16:34:27.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.826+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741015a5f0 con 0x7f7410136430 2024-03-18T16:34:27.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.826+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7410138160 con 0x7f7410136430 2024-03-18T16:34:27.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.827+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7410074fd0 con 0x7f7410136430 2024-03-18T16:34:27.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.858+0000 7f740cc00640 1 -- 172.21.15.102:0/953750086 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f73fc045020 con 0x7f7410136430 2024-03-18T16:34:27.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.860+0000 7f740cc00640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f73f006bc50 0x7f73f006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.860+0000 7f7415200640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f73f006bc50 0x7f73f006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:27.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.861+0000 7f7415200640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f73f006bc50 0x7f73f006e110 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7404000ff0 tx=0x7f74040019f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:27.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:27.862+0000 7f740cc00640 1 -- 172.21.15.102:0/953750086 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f73fc0788d0 con 0x7f7410136430 2024-03-18T16:34:28.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.118+0000 7f740cc00640 1 -- 172.21.15.102:0/953750086 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f73fc04c070 con 0x7f7410136430 2024-03-18T16:34:28.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.382+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7410152890 con 0x7f7410136430 2024-03-18T16:34:28.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.397+0000 7f740cc00640 1 -- 172.21.15.102:0/953750086 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f73fc07a830 con 0x7f7410136430 2024-03-18T16:34:28.400 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:28.400 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:28.400 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:28.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.400+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f73f006bc50 msgr2=0x7f73f006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:28.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.400+0000 7f7416600640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f73f006bc50 0x7f73f006e110 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7404000ff0 tx=0x7f74040019f0 comp rx=0 tx=0).stop 2024-03-18T16:34:28.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.400+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7410136430 msgr2=0x7f741015adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:28.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.400+0000 7f7416600640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7410136430 0x7f741015adb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f73fc009be0 tx=0x7f73fc004730 comp rx=0 tx=0).stop 2024-03-18T16:34:28.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.401+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 shutdown_connections 2024-03-18T16:34:28.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.401+0000 7f7416600640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f73f006bc50 0x7f73f006e110 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:28.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.401+0000 7f7416600640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7410137220 0x7f741015b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:28.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.401+0000 7f7416600640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7410136430 0x7f741015adb0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:28.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.401+0000 7f7416600640 1 --2- 172.21.15.102:0/953750086 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f73f8004800 0x7f74101597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:28.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.401+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 >> 172.21.15.102:0/953750086 conn(0x7f74100cb890 msgr2=0x7f741013e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:28.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.401+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 shutdown_connections 2024-03-18T16:34:28.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:28.401+0000 7f7416600640 1 -- 172.21.15.102:0/953750086 wait complete. 2024-03-18T16:34:29.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:29.024 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:30.025 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:30.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.195+0000 7f8cc6a00640 1 Processor -- start 2024-03-18T16:34:30.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.195+0000 7f8cc6a00640 1 -- start start 2024-03-18T16:34:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.196+0000 7f8cc6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0158420 0x7f8cc015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.196+0000 7f8cc6a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8cc00c4fa0 0x7f8cc00c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.196+0000 7f8cc6a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cc00c72d0 con 0x7f8cc0141650 2024-03-18T16:34:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.196+0000 7f8cc6a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8cc015c8e0 con 0x7f8cc00c4fa0 2024-03-18T16:34:30.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.196+0000 7f8cc6a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc0159370 con 0x7f8cc0158420 2024-03-18T16:34:30.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.197+0000 7f8cc4c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0158420 0x7f8cc015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:30.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.197+0000 7f8cc6000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8cc00c4fa0 0x7f8cc00c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:30.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.197+0000 7f8cc5600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8cc0141650 0x7f8cc00c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38562/0 (socket says 172.21.15.102:38562) 2024-03-18T16:34:30.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.197+0000 7f8cc5600640 1 -- 172.21.15.102:0/2565278601 learned_addr learned my addr 172.21.15.102:0/2565278601 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:30.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.200+0000 7f8cc4c00640 1 -- 172.21.15.102:0/2565278601 >> v2:172.21.15.154:3300/0 conn(0x7f8cc00c4fa0 msgr2=0x7f8cc00c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:30.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.200+0000 7f8cc4c00640 1 --2- 172.21.15.102:0/2565278601 >> v2:172.21.15.154:3300/0 conn(0x7f8cc00c4fa0 0x7f8cc00c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.200+0000 7f8cc4c00640 1 -- 172.21.15.102:0/2565278601 >> v1:172.21.15.102:6789/0 conn(0x7f8cc0141650 legacy=0x7f8cc00c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:30.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.200+0000 7f8cc4c00640 1 -- 172.21.15.102:0/2565278601 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc0158840 con 0x7f8cc0158420 2024-03-18T16:34:30.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.202+0000 7f8cc4c00640 1 --2- 172.21.15.102:0/2565278601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0158420 0x7f8cc015c3a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0097a0 tx=0x7f8cac02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc3515715154b9ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:30.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.206+0000 7f8cbfe00640 1 -- 172.21.15.102:0/2565278601 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac035020 con 0x7f8cc0158420 2024-03-18T16:34:30.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.206+0000 7f8cbfe00640 1 -- 172.21.15.102:0/2565278601 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cac031810 con 0x7f8cc0158420 2024-03-18T16:34:30.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.206+0000 7f8cbfe00640 1 -- 172.21.15.102:0/2565278601 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac031be0 con 0x7f8cc0158420 2024-03-18T16:34:30.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.207+0000 7f8cc6a00640 1 -- 172.21.15.102:0/2565278601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0158420 msgr2=0x7f8cc015c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:30.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.207+0000 7f8cc6a00640 1 --2- 172.21.15.102:0/2565278601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0158420 0x7f8cc015c3a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8cac0097a0 tx=0x7f8cac02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.207+0000 7f8cc6a00640 1 -- 172.21.15.102:0/2565278601 shutdown_connections 2024-03-18T16:34:30.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.207+0000 7f8cc6a00640 1 --2- 172.21.15.102:0/2565278601 >> v2:172.21.15.154:3300/0 conn(0x7f8cc00c4fa0 0x7f8cc00c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.207+0000 7f8cc6a00640 1 --2- 172.21.15.102:0/2565278601 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0158420 0x7f8cc015c3a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.207+0000 7f8cc6a00640 1 -- 172.21.15.102:0/2565278601 >> 172.21.15.102:0/2565278601 conn(0x7f8cc00cb890 msgr2=0x7f8cc00cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:30.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.207+0000 7f8cc6a00640 1 -- 172.21.15.102:0/2565278601 shutdown_connections 2024-03-18T16:34:30.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.207+0000 7f8cc6a00640 1 -- 172.21.15.102:0/2565278601 wait complete. 2024-03-18T16:34:30.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.209+0000 7f8cc6a00640 1 Processor -- start 2024-03-18T16:34:30.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.209+0000 7f8cc6a00640 1 -- start start 2024-03-18T16:34:30.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.210+0000 7f8cc6a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc00c4fa0 0x7f8cc015c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:30.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.210+0000 7f8cc6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0141650 0x7f8cc013d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:30.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.210+0000 7f8cc6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc0158420 0x7f8cc00c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:30.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.210+0000 7f8cc6a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc0159090 con 0x7f8cc0158420 2024-03-18T16:34:30.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.210+0000 7f8cc6a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc0158db0 con 0x7f8cc00c4fa0 2024-03-18T16:34:30.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.210+0000 7f8cc6a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc00c63c0 con 0x7f8cc0141650 2024-03-18T16:34:30.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.210+0000 7f8cc4c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0141650 0x7f8cc013d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:30.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.211+0000 7f8cc5600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc00c4fa0 0x7f8cc015c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:30.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.211+0000 7f8cc6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc0158420 0x7f8cc00c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:30.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.211+0000 7f8cc4c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0141650 0x7f8cc013d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:36842/0 (socket says 172.21.15.102:36842) 2024-03-18T16:34:30.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.211+0000 7f8cc4c00640 1 -- 172.21.15.102:0/978856891 learned_addr learned my addr 172.21.15.102:0/978856891 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:30.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.214+0000 7f8cc4c00640 1 -- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc00c4fa0 msgr2=0x7f8cc015c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:30.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.214+0000 7f8cc4c00640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc00c4fa0 0x7f8cc015c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.214+0000 7f8cc4c00640 1 -- 172.21.15.102:0/978856891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc0158420 msgr2=0x7f8cc00c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:30.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.214+0000 7f8cc4c00640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc0158420 0x7f8cc00c9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.214+0000 7f8cc4c00640 1 -- 172.21.15.102:0/978856891 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc00cb470 con 0x7f8cc0141650 2024-03-18T16:34:30.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.215+0000 7f8cc5600640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc00c4fa0 0x7f8cc015c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:34:30.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.217+0000 7f8cc4c00640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0141650 0x7f8cc013d6c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8cac034040 tx=0x7f8cac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:30.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.221+0000 7f8cbd600640 1 -- 172.21.15.102:0/978856891 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac040070 con 0x7f8cc0141650 2024-03-18T16:34:30.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.221+0000 7f8cbd600640 1 -- 172.21.15.102:0/978856891 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cac039cf0 con 0x7f8cc0141650 2024-03-18T16:34:30.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.221+0000 7f8cbd600640 1 -- 172.21.15.102:0/978856891 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cac035040 con 0x7f8cc0141650 2024-03-18T16:34:30.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.221+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc015b2d0 con 0x7f8cc0141650 2024-03-18T16:34:30.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.222+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc015b890 con 0x7f8cc0141650 2024-03-18T16:34:30.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.223+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc0074fd0 con 0x7f8cc0141650 2024-03-18T16:34:30.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.242+0000 7f8cbd600640 1 -- 172.21.15.102:0/978856891 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8cac038020 con 0x7f8cc0141650 2024-03-18T16:34:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.243+0000 7f8cbd600640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca006bc20 0x7f8ca006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.243+0000 7f8cc5600640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca006bc20 0x7f8ca006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.244+0000 7f8cc5600640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca006bc20 0x7f8ca006e0e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4007fd0 tx=0x7f8cb4007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:30.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.483+0000 7f8cbd600640 1 -- 172.21.15.102:0/978856891 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8cac078df0 con 0x7f8cc0141650 2024-03-18T16:34:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.495+0000 7f8cbd600640 1 -- 172.21.15.102:0/978856891 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8cac07ad50 con 0x7f8cc0141650 2024-03-18T16:34:30.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.795+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cc00b7f00 con 0x7f8cc0141650 2024-03-18T16:34:30.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.811+0000 7f8cbd600640 1 -- 172.21.15.102:0/978856891 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8cac07edf0 con 0x7f8cc0141650 2024-03-18T16:34:30.814 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:30.814 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:30.814 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:30.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.813+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca006bc20 msgr2=0x7f8ca006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:30.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.813+0000 7f8cc6a00640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca006bc20 0x7f8ca006e0e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4007fd0 tx=0x7f8cb4007e80 comp rx=0 tx=0).stop 2024-03-18T16:34:30.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.813+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0141650 msgr2=0x7f8cc013d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:30.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.813+0000 7f8cc6a00640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0141650 0x7f8cc013d6c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8cac034040 tx=0x7f8cac0046c0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.814+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 shutdown_connections 2024-03-18T16:34:30.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.814+0000 7f8cc6a00640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca006bc20 0x7f8ca006e0e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.814+0000 7f8cc6a00640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc0158420 0x7f8cc00c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.814+0000 7f8cc6a00640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc0141650 0x7f8cc013d6c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.814+0000 7f8cc6a00640 1 --2- 172.21.15.102:0/978856891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc00c4fa0 0x7f8cc015c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:30.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.814+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 >> 172.21.15.102:0/978856891 conn(0x7f8cc00cb890 msgr2=0x7f8cc013e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:30.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.814+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 shutdown_connections 2024-03-18T16:34:30.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:30.814+0000 7f8cc6a00640 1 -- 172.21.15.102:0/978856891 wait complete. 2024-03-18T16:34:31.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:31.437 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:32.438 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.610+0000 7fa83e000640 1 Processor -- start 2024-03-18T16:34:32.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.611+0000 7fa83e000640 1 -- start start 2024-03-18T16:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.612+0000 7fa83e000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa838141650 0x7fa838158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.612+0000 7fa83e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa8380c4fa0 0x7fa8380c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.612+0000 7fa83e000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8380c72d0 con 0x7fa83815c3a0 2024-03-18T16:34:32.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.612+0000 7fa83e000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa838159370 con 0x7fa8380c4fa0 2024-03-18T16:34:32.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.612+0000 7fa83e000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa838158ad0 con 0x7fa838141650 2024-03-18T16:34:32.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.613+0000 7fa83cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa838141650 0x7fa838158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:32.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.613+0000 7fa837e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa83815c3a0 0x7fa8380c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44348/0 (socket says 172.21.15.102:44348) 2024-03-18T16:34:32.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.613+0000 7fa837e00640 1 -- 172.21.15.102:0/3363614267 learned_addr learned my addr 172.21.15.102:0/3363614267 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:32.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.613+0000 7fa83d600640 1 --2- 172.21.15.102:0/3363614267 >> v2:172.21.15.102:3300/0 conn(0x7fa8380c4fa0 0x7fa8380c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:32.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.616+0000 7fa83d600640 1 -- 172.21.15.102:0/3363614267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa838141650 msgr2=0x7fa838158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:32.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.616+0000 7fa83d600640 1 --2- 172.21.15.102:0/3363614267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa838141650 0x7fa838158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:32.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.616+0000 7fa83d600640 1 -- 172.21.15.102:0/3363614267 >> v1:172.21.15.154:6789/0 conn(0x7fa83815c3a0 legacy=0x7fa8380c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:32.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.616+0000 7fa83d600640 1 -- 172.21.15.102:0/3363614267 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa838158db0 con 0x7fa8380c4fa0 2024-03-18T16:34:32.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.618+0000 7fa83d600640 1 --2- 172.21.15.102:0/3363614267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8380c4fa0 0x7fa8380c5430 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa830009d90 tx=0x7fa83002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a7c80a9de0dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.624+0000 7fa837400640 1 -- 172.21.15.102:0/3363614267 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa830039070 con 0x7fa8380c4fa0 2024-03-18T16:34:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.624+0000 7fa837400640 1 -- 172.21.15.102:0/3363614267 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa83002cc60 con 0x7fa8380c4fa0 2024-03-18T16:34:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.624+0000 7fa837400640 1 -- 172.21.15.102:0/3363614267 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa830034660 con 0x7fa8380c4fa0 2024-03-18T16:34:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.624+0000 7fa83e000640 1 -- 172.21.15.102:0/3363614267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8380c4fa0 msgr2=0x7fa8380c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.624+0000 7fa83e000640 1 --2- 172.21.15.102:0/3363614267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8380c4fa0 0x7fa8380c5430 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa830009d90 tx=0x7fa83002beb0 comp rx=0 tx=0).stop 2024-03-18T16:34:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.624+0000 7fa83e000640 1 -- 172.21.15.102:0/3363614267 shutdown_connections 2024-03-18T16:34:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.624+0000 7fa83e000640 1 --2- 172.21.15.102:0/3363614267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8380c4fa0 0x7fa8380c5430 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.624+0000 7fa83e000640 1 --2- 172.21.15.102:0/3363614267 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa838141650 0x7fa838158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.625+0000 7fa83e000640 1 -- 172.21.15.102:0/3363614267 >> 172.21.15.102:0/3363614267 conn(0x7fa8380cb890 msgr2=0x7fa8380cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.625+0000 7fa83e000640 1 -- 172.21.15.102:0/3363614267 shutdown_connections 2024-03-18T16:34:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.625+0000 7fa83e000640 1 -- 172.21.15.102:0/3363614267 wait complete. 2024-03-18T16:34:32.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.627+0000 7fa83e000640 1 Processor -- start 2024-03-18T16:34:32.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.627+0000 7fa83e000640 1 -- start start 2024-03-18T16:34:32.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.629+0000 7fa83e000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8380c4fa0 0x7fa838130010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:32.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.629+0000 7fa83e000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa838141650 0x7fa838156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:32.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.629+0000 7fa83e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa83815c3a0 0x7fa838157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:32.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.629+0000 7fa83e000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa8380c66a0 con 0x7fa83815c3a0 2024-03-18T16:34:32.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.629+0000 7fa83e000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8380c63c0 con 0x7fa838141650 2024-03-18T16:34:32.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.629+0000 7fa83e000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa83815c7c0 con 0x7fa8380c4fa0 2024-03-18T16:34:32.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.630+0000 7fa837e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa838141650 0x7fa838156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:32.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.630+0000 7fa83cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8380c4fa0 0x7fa838130010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:32.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.630+0000 7fa83cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8380c4fa0 0x7fa838130010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:59524/0 (socket says 172.21.15.102:59524) 2024-03-18T16:34:32.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.630+0000 7fa83cc00640 1 -- 172.21.15.102:0/3483789037 learned_addr learned my addr 172.21.15.102:0/3483789037 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:32.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.632+0000 7fa83d600640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa83815c3a0 0x7fa838157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:32.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.634+0000 7fa83cc00640 1 -- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa838141650 msgr2=0x7fa838156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:32.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.634+0000 7fa83cc00640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa838141650 0x7fa838156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:32.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.634+0000 7fa83cc00640 1 -- 172.21.15.102:0/3483789037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa83815c3a0 msgr2=0x7fa838157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:32.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.634+0000 7fa83cc00640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa83815c3a0 0x7fa838157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:32.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.634+0000 7fa83cc00640 1 -- 172.21.15.102:0/3483789037 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa838136450 con 0x7fa8380c4fa0 2024-03-18T16:34:32.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.637+0000 7fa83cc00640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8380c4fa0 0x7fa838130010 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00e9e0 tx=0x7fa82c00eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:32.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.643+0000 7fa834c00640 1 -- 172.21.15.102:0/3483789037 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c017070 con 0x7fa8380c4fa0 2024-03-18T16:34:32.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.644+0000 7fa834c00640 1 -- 172.21.15.102:0/3483789037 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82c00c9d0 con 0x7fa8380c4fa0 2024-03-18T16:34:32.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.644+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa838138490 con 0x7fa8380c4fa0 2024-03-18T16:34:32.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.644+0000 7fa834c00640 1 -- 172.21.15.102:0/3483789037 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c00cdc0 con 0x7fa8380c4fa0 2024-03-18T16:34:32.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.644+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa838135920 con 0x7fa8380c4fa0 2024-03-18T16:34:32.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.645+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa838074fd0 con 0x7fa8380c4fa0 2024-03-18T16:34:32.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.671+0000 7fa834c00640 1 -- 172.21.15.102:0/3483789037 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa82c01a070 con 0x7fa8380c4fa0 2024-03-18T16:34:32.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.673+0000 7fa834c00640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa81806bc50 0x7fa81806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:32.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.673+0000 7fa837e00640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa81806bc50 0x7fa81806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:32.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.674+0000 7fa837e00640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa81806bc50 0x7fa81806e110 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa838152310 tx=0x7fa824007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:32.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:32.676+0000 7fa834c00640 1 -- 172.21.15.102:0/3483789037 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa82c0576a0 con 0x7fa8380c4fa0 2024-03-18T16:34:33.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.002+0000 7fa834c00640 1 -- 172.21.15.102:0/3483789037 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa838136450 con 0x7fa8380c4fa0 2024-03-18T16:34:33.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.247+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa838156290 con 0x7fa8380c4fa0 2024-03-18T16:34:33.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.262+0000 7fa834c00640 1 -- 172.21.15.102:0/3483789037 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa82c059600 con 0x7fa8380c4fa0 2024-03-18T16:34:33.264 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:33.265 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:33.265 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa81806bc50 msgr2=0x7fa81806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa81806bc50 0x7fa81806e110 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa838152310 tx=0x7fa824007e80 comp rx=0 tx=0).stop 2024-03-18T16:34:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8380c4fa0 msgr2=0x7fa838130010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8380c4fa0 0x7fa838130010 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00e9e0 tx=0x7fa82c00eea0 comp rx=0 tx=0).stop 2024-03-18T16:34:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 shutdown_connections 2024-03-18T16:34:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa81806bc50 0x7fa81806e110 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:33.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa83815c3a0 0x7fa838157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa838141650 0x7fa838156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 --2- 172.21.15.102:0/3483789037 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8380c4fa0 0x7fa838130010 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 >> 172.21.15.102:0/3483789037 conn(0x7fa8380cb890 msgr2=0x7fa838071dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 shutdown_connections 2024-03-18T16:34:33.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:33.264+0000 7fa83e000640 1 -- 172.21.15.102:0/3483789037 wait complete. 2024-03-18T16:34:33.898 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:33.899 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:34.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:35.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.075+0000 7f22a6a00640 1 Processor -- start 2024-03-18T16:34:35.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.075+0000 7f22a6a00640 1 -- start start 2024-03-18T16:34:35.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.076+0000 7f22a6a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f22a00d0250 0x7f22a015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:35.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.076+0000 7f22a6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cc2f0 0x7f22a00cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:35.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.076+0000 7f22a6a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a00cabd0 con 0x7f22a00cfe70 2024-03-18T16:34:35.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.076+0000 7f22a6a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f22a00caea0 con 0x7f22a00d0250 2024-03-18T16:34:35.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.076+0000 7f22a6a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f22a00cb180 con 0x7f22a00cc2f0 2024-03-18T16:34:35.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.077+0000 7f22a4c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f22a00d0250 0x7f22a015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:35.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.077+0000 7f22a6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cc2f0 0x7f22a00cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:35.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.078+0000 7f22a6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cc2f0 0x7f22a00cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:59538/0 (socket says 172.21.15.102:59538) 2024-03-18T16:34:35.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.078+0000 7f22a6000640 1 -- 172.21.15.102:0/3854328132 learned_addr learned my addr 172.21.15.102:0/3854328132 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:35.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.080+0000 7f22a6000640 1 -- 172.21.15.102:0/3854328132 >> v2:172.21.15.154:3300/0 conn(0x7f22a00d0250 msgr2=0x7f22a015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:35.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.080+0000 7f22a6000640 1 --2- 172.21.15.102:0/3854328132 >> v2:172.21.15.154:3300/0 conn(0x7f22a00d0250 0x7f22a015ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.080+0000 7f22a6000640 1 -- 172.21.15.102:0/3854328132 >> v1:172.21.15.154:6789/0 conn(0x7f22a00cfe70 legacy=0x7f22a00c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:35.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.080+0000 7f22a6000640 1 -- 172.21.15.102:0/3854328132 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a00cb460 con 0x7f22a00cc2f0 2024-03-18T16:34:35.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.082+0000 7f22a6000640 1 --2- 172.21.15.102:0/3854328132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cc2f0 0x7f22a00cc710 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2298009e50 tx=0x7f229802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aed5ce3af2588e95 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:35.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.086+0000 7f229fe00640 1 -- 172.21.15.102:0/3854328132 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229802cb90 con 0x7f22a00cc2f0 2024-03-18T16:34:35.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.086+0000 7f229fe00640 1 -- 172.21.15.102:0/3854328132 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229802cd30 con 0x7f22a00cc2f0 2024-03-18T16:34:35.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.086+0000 7f229fe00640 1 -- 172.21.15.102:0/3854328132 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2298034610 con 0x7f22a00cc2f0 2024-03-18T16:34:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.087+0000 7f22a6a00640 1 -- 172.21.15.102:0/3854328132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cc2f0 msgr2=0x7f22a00cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.087+0000 7f22a6a00640 1 --2- 172.21.15.102:0/3854328132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cc2f0 0x7f22a00cc710 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2298009e50 tx=0x7f229802bff0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.087+0000 7f22a6a00640 1 -- 172.21.15.102:0/3854328132 shutdown_connections 2024-03-18T16:34:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.087+0000 7f22a6a00640 1 --2- 172.21.15.102:0/3854328132 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cc2f0 0x7f22a00cc710 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.087+0000 7f22a6a00640 1 --2- 172.21.15.102:0/3854328132 >> v2:172.21.15.154:3300/0 conn(0x7f22a00d0250 0x7f22a015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.087+0000 7f22a6a00640 1 -- 172.21.15.102:0/3854328132 >> 172.21.15.102:0/3854328132 conn(0x7f22a008c3e0 msgr2=0x7f22a006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.087+0000 7f22a6a00640 1 -- 172.21.15.102:0/3854328132 shutdown_connections 2024-03-18T16:34:35.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.087+0000 7f22a6a00640 1 -- 172.21.15.102:0/3854328132 wait complete. 2024-03-18T16:34:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.090+0000 7f22a6a00640 1 Processor -- start 2024-03-18T16:34:35.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.090+0000 7f22a6a00640 1 -- start start 2024-03-18T16:34:35.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.091+0000 7f22a6a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a00cc2f0 0x7f22a01e6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:35.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.091+0000 7f22a6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cfe70 0x7f22a01e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:35.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.092+0000 7f22a6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22a00d0250 0x7f22a0155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:35.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.092+0000 7f22a6a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f22a00cbd00 con 0x7f22a00d0250 2024-03-18T16:34:35.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.092+0000 7f22a6a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f22a00cba20 con 0x7f22a00cc2f0 2024-03-18T16:34:35.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.092+0000 7f22a6a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f22a00ccc50 con 0x7f22a00cfe70 2024-03-18T16:34:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.092+0000 7f22a4c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cfe70 0x7f22a01e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.093+0000 7f22a6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22a00d0250 0x7f22a0155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.093+0000 7f22a4c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cfe70 0x7f22a01e03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:59552/0 (socket says 172.21.15.102:59552) 2024-03-18T16:34:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.093+0000 7f22a4c00640 1 -- 172.21.15.102:0/2613802092 learned_addr learned my addr 172.21.15.102:0/2613802092 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.096+0000 7f22a4c00640 1 -- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a00cc2f0 msgr2=0x7f22a01e6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.096+0000 7f22a4c00640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a00cc2f0 0x7f22a01e6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.096+0000 7f22a4c00640 1 -- 172.21.15.102:0/2613802092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22a00d0250 msgr2=0x7f22a0155e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.096+0000 7f22a4c00640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22a00d0250 0x7f22a0155e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.096+0000 7f22a4c00640 1 -- 172.21.15.102:0/2613802092 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a01e0e40 con 0x7f22a00cfe70 2024-03-18T16:34:35.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.098+0000 7f22a4c00640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cfe70 0x7f22a01e03b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f228c00e870 tx=0x7f228c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.102+0000 7f229d600640 1 -- 172.21.15.102:0/2613802092 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c017070 con 0x7f22a00cfe70 2024-03-18T16:34:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.102+0000 7f229d600640 1 -- 172.21.15.102:0/2613802092 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f228c00c940 con 0x7f22a00cfe70 2024-03-18T16:34:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.102+0000 7f229d600640 1 -- 172.21.15.102:0/2613802092 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c00cd10 con 0x7f22a00cfe70 2024-03-18T16:34:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.102+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a01efd00 con 0x7f22a00cfe70 2024-03-18T16:34:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.102+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a01f0280 con 0x7f22a00cfe70 2024-03-18T16:34:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.103+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a0074fd0 con 0x7f22a00cfe70 2024-03-18T16:34:35.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.123+0000 7f229d600640 1 -- 172.21.15.102:0/2613802092 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f228c01a070 con 0x7f22a00cfe70 2024-03-18T16:34:35.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.125+0000 7f229d600640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f228006bbd0 0x7f228006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:35.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.125+0000 7f22a5600640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f228006bbd0 0x7f228006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:35.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.126+0000 7f22a5600640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f228006bbd0 0x7f228006e090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2294000f60 tx=0x7f22940079d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:35.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.127+0000 7f229d600640 1 -- 172.21.15.102:0/2613802092 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f228c057590 con 0x7f22a00cfe70 2024-03-18T16:34:35.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.377+0000 7f229d600640 1 -- 172.21.15.102:0/2613802092 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f228c0594f0 con 0x7f22a00cfe70 2024-03-18T16:34:35.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.617+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22a00c6760 con 0x7f22a00cfe70 2024-03-18T16:34:35.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.630+0000 7f229d600640 1 -- 172.21.15.102:0/2613802092 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f228c05d590 con 0x7f22a00cfe70 2024-03-18T16:34:35.633 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:35.633 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.632+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f228006bbd0 msgr2=0x7f228006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.632+0000 7f22a6a00640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f228006bbd0 0x7f228006e090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2294000f60 tx=0x7f22940079d0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.632+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cfe70 msgr2=0x7f22a01e03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.632+0000 7f22a6a00640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cfe70 0x7f22a01e03b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f228c00e870 tx=0x7f228c00ed30 comp rx=0 tx=0).stop 2024-03-18T16:34:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.633+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 shutdown_connections 2024-03-18T16:34:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.633+0000 7f22a6a00640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f228006bbd0 0x7f228006e090 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.633+0000 7f22a6a00640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22a00d0250 0x7f22a0155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.633+0000 7f22a6a00640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f22a00cfe70 0x7f22a01e03b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.633+0000 7f22a6a00640 1 --2- 172.21.15.102:0/2613802092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22a00cc2f0 0x7f22a01e6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.633+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 >> 172.21.15.102:0/2613802092 conn(0x7f22a008c3e0 msgr2=0x7f22a00c8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.633+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 shutdown_connections 2024-03-18T16:34:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:35.633+0000 7f22a6a00640 1 -- 172.21.15.102:0/2613802092 wait complete. 2024-03-18T16:34:36.256 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:36.256 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:37.257 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:37.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.428+0000 7fa8c9c00640 1 Processor -- start 2024-03-18T16:34:37.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.428+0000 7fa8c9c00640 1 -- start start 2024-03-18T16:34:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.430+0000 7fa8c9c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa8c40cad40 0x7fa8c406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.430+0000 7fa8c9c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c406c9c0 con 0x7fa8c415a1a0 2024-03-18T16:34:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.430+0000 7fa8c9c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c406cca0 con 0x7fa8c40ca960 2024-03-18T16:34:37.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.430+0000 7fa8c9c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c40c8680 con 0x7fa8c40cad40 2024-03-18T16:34:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.431+0000 7fa8c3e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa8c40ca960 0x7fa8c415e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44368/0 (socket says 172.21.15.102:44368) 2024-03-18T16:34:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.431+0000 7fa8c3e00640 1 -- 172.21.15.102:0/52738743 learned_addr learned my addr 172.21.15.102:0/52738743 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.434+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1006440949 0 0) 0x7fa8c406cca0 con 0x7fa8c40ca960 2024-03-18T16:34:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.434+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8a8003520 con 0x7fa8c40ca960 2024-03-18T16:34:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.434+0000 7fa8c3400640 1 --2- 172.21.15.102:0/52738743 >> v2:172.21.15.102:3300/0 conn(0x7fa8c40cad40 0x7fa8c406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:37.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2070162600 0 0) 0x7fa8a8003520 con 0x7fa8c40ca960 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 >> v2:172.21.15.102:3300/0 conn(0x7fa8c40cad40 msgr2=0x7fa8c406c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 --2- 172.21.15.102:0/52738743 >> v2:172.21.15.102:3300/0 conn(0x7fa8c40cad40 0x7fa8c406c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 >> v1:172.21.15.102:6789/0 conn(0x7fa8c415a1a0 legacy=0x7fa8c40705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c406cf80 con 0x7fa8c40ca960 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7fa8b4001670 con 0x7fa8c40ca960 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 >> v1:172.21.15.154:6789/0 conn(0x7fa8c40ca960 legacy=0x7fa8c415e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a8003a00 0x7fa8a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8004800 0x7fa8a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8a80051c0 0x7fa8a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a800c040 con 0x7fa8a80051c0 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a800c1f0 con 0x7fa8a8004800 2024-03-18T16:34:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.438+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa8a800c3f0 con 0x7fa8a8003a00 2024-03-18T16:34:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.439+0000 7fa8c3e00640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a8003a00 0x7fa8a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:37.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.441+0000 7fa8c3400640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8004800 0x7fa8a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.442+0000 7fa8c3e00640 1 -- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8004800 msgr2=0x7fa8a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.442+0000 7fa8c3e00640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8004800 0x7fa8a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.442+0000 7fa8c3e00640 1 -- 172.21.15.102:0/52738743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8a80051c0 msgr2=0x7fa8a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.442+0000 7fa8c3e00640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8a80051c0 0x7fa8a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.442+0000 7fa8c3e00640 1 -- 172.21.15.102:0/52738743 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a8003520 con 0x7fa8a8003a00 2024-03-18T16:34:37.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.442+0000 7fa8c9200640 1 -- 172.21.15.102:0/52738743 reap_dead start 2024-03-18T16:34:37.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.444+0000 7fa8c3e00640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a8003a00 0x7fa8a8003e00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400c190 tx=0x7fa8a8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce8ee98556d23575 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:37.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.448+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4007c60 con 0x7fa8a8003a00 2024-03-18T16:34:37.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.448+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b4030070 con 0x7fa8a8003a00 2024-03-18T16:34:37.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.448+0000 7fa8c2a00640 1 -- 172.21.15.102:0/52738743 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4005d30 con 0x7fa8a8003a00 2024-03-18T16:34:37.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.448+0000 7fa8c9c00640 1 -- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a8003a00 msgr2=0x7fa8a8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:37.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.448+0000 7fa8c9c00640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a8003a00 0x7fa8a8003e00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400c190 tx=0x7fa8a8005830 comp rx=0 tx=0).stop 2024-03-18T16:34:37.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.449+0000 7fa8c9c00640 1 -- 172.21.15.102:0/52738743 shutdown_connections 2024-03-18T16:34:37.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.449+0000 7fa8c9c00640 1 --2- 172.21.15.102:0/52738743 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8a8003a00 0x7fa8a8003e00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:37.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.449+0000 7fa8c9c00640 1 -- 172.21.15.102:0/52738743 >> 172.21.15.102:0/52738743 conn(0x7fa8c4067190 msgr2=0x7fa8c40675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:37.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.449+0000 7fa8c9c00640 1 -- 172.21.15.102:0/52738743 shutdown_connections 2024-03-18T16:34:37.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.449+0000 7fa8c9c00640 1 -- 172.21.15.102:0/52738743 wait complete. 2024-03-18T16:34:37.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.451+0000 7fa8c9c00640 1 Processor -- start 2024-03-18T16:34:37.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.452+0000 7fa8c9c00640 1 -- start start 2024-03-18T16:34:37.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.453+0000 7fa8c9c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8003a00 0x7fa8c413e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:37.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.453+0000 7fa8c9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8c415e320 0x7fa8c40cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:37.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.453+0000 7fa8c9c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8c413dc00 0x7fa8c41380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:37.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.453+0000 7fa8c9c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c406c310 con 0x7fa8c415e320 2024-03-18T16:34:37.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.453+0000 7fa8c9c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c415dc10 con 0x7fa8a8003a00 2024-03-18T16:34:37.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.453+0000 7fa8c9c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c406c490 con 0x7fa8c413dc00 2024-03-18T16:34:37.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.454+0000 7fa8c3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8003a00 0x7fa8c413e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:37.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.454+0000 7fa8c9200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8c413dc00 0x7fa8c41380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:37.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.454+0000 7fa8c3e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8003a00 0x7fa8c413e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36032/0 (socket says 172.21.15.102:36032) 2024-03-18T16:34:37.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.454+0000 7fa8c3e00640 1 -- 172.21.15.102:0/1256315549 learned_addr learned my addr 172.21.15.102:0/1256315549 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:37.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.454+0000 7fa8c3400640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8c415e320 0x7fa8c40cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:37.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.458+0000 7fa8c9200640 1 -- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8003a00 msgr2=0x7fa8c413e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:37.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.458+0000 7fa8c9200640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8003a00 0x7fa8c413e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:37.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.458+0000 7fa8c9200640 1 -- 172.21.15.102:0/1256315549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8c415e320 msgr2=0x7fa8c40cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:37.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.458+0000 7fa8c9200640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8c415e320 0x7fa8c40cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:37.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.458+0000 7fa8c9200640 1 -- 172.21.15.102:0/1256315549 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c413e890 con 0x7fa8c413dc00 2024-03-18T16:34:37.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.460+0000 7fa8c9200640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8c413dc00 0x7fa8c41380c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40c2b30 tx=0x7fa8bc00c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:37.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.464+0000 7fa8bbe00640 1 -- 172.21.15.102:0/1256315549 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc007030 con 0x7fa8c413dc00 2024-03-18T16:34:37.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.464+0000 7fa8bbe00640 1 -- 172.21.15.102:0/1256315549 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8bc0071f0 con 0x7fa8c413dc00 2024-03-18T16:34:37.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.464+0000 7fa8bbe00640 1 -- 172.21.15.102:0/1256315549 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8bc0075e0 con 0x7fa8c413dc00 2024-03-18T16:34:37.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.464+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c41f0480 con 0x7fa8c413dc00 2024-03-18T16:34:37.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.464+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c413d120 con 0x7fa8c413dc00 2024-03-18T16:34:37.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.466+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c4074fd0 con 0x7fa8c413dc00 2024-03-18T16:34:37.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.485+0000 7fa8bbe00640 1 -- 172.21.15.102:0/1256315549 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa8bc007780 con 0x7fa8c413dc00 2024-03-18T16:34:37.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.486+0000 7fa8bbe00640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa8a406bc20 0x7fa8a406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:37.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.487+0000 7fa8c3e00640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa8a406bc20 0x7fa8a406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:37.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.487+0000 7fa8bbe00640 1 -- 172.21.15.102:0/1256315549 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa8bc0566a0 con 0x7fa8c413dc00 2024-03-18T16:34:37.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.488+0000 7fa8c3e00640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa8a406bc20 0x7fa8a406e0e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4000fb0 tx=0x7fa8b4007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:37.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.742+0000 7fa8bbe00640 1 -- 172.21.15.102:0/1256315549 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa8bc058600 con 0x7fa8c413dc00 2024-03-18T16:34:37.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:37.981+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8c40c9c50 con 0x7fa8c413dc00 2024-03-18T16:34:38.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.094+0000 7fa8bbe00640 1 -- 172.21.15.102:0/1256315549 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa8bc05c6a0 con 0x7fa8c413dc00 2024-03-18T16:34:38.096 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:38.096 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:38.096 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa8a406bc20 msgr2=0x7fa8a406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa8a406bc20 0x7fa8a406e0e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4000fb0 tx=0x7fa8b4007000 comp rx=0 tx=0).stop 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8c413dc00 msgr2=0x7fa8c41380c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8c413dc00 0x7fa8c41380c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40c2b30 tx=0x7fa8bc00c890 comp rx=0 tx=0).stop 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 shutdown_connections 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa8a406bc20 0x7fa8a406e0e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa8c413dc00 0x7fa8c41380c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8c415e320 0x7fa8c40cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 --2- 172.21.15.102:0/1256315549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a8003a00 0x7fa8c413e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:38.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.098+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 >> 172.21.15.102:0/1256315549 conn(0x7fa8c4067190 msgr2=0x7fa8c4158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:38.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.099+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 shutdown_connections 2024-03-18T16:34:38.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:38.099+0000 7fa8c9c00640 1 -- 172.21.15.102:0/1256315549 wait complete. 2024-03-18T16:34:38.723 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:38.724 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:39.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:39.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.896+0000 7f3277e00640 1 Processor -- start 2024-03-18T16:34:39.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.896+0000 7f3277e00640 1 -- start start 2024-03-18T16:34:39.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.897+0000 7f3277e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f327815ad60 0x7f32780c5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:39.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.897+0000 7f3277e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f327806bf60 con 0x7f327806b470 2024-03-18T16:34:39.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.897+0000 7f3277e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32780c94b0 con 0x7f327806de50 2024-03-18T16:34:39.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.897+0000 7f3277e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f32780ca0a0 con 0x7f327815ad60 2024-03-18T16:34:39.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.898+0000 7f3276000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f327806de50 0x7f3278154150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44382/0 (socket says 172.21.15.102:44382) 2024-03-18T16:34:39.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.898+0000 7f3276000640 1 -- 172.21.15.102:0/3642712355 learned_addr learned my addr 172.21.15.102:0/3642712355 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:39.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.899+0000 7f3277400640 1 --2- 172.21.15.102:0/3642712355 >> v2:172.21.15.154:3300/0 conn(0x7f327815ad60 0x7f32780c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:39.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.901+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 595605441 0 0) 0x7f327806bf60 con 0x7f327806b470 2024-03-18T16:34:39.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.901+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f325c003520 con 0x7f327806b470 2024-03-18T16:34:39.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.901+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 227252469 0 0) 0x7f32780c94b0 con 0x7f327806de50 2024-03-18T16:34:39.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.901+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f327806bf60 con 0x7f327806de50 2024-03-18T16:34:39.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1515878901 0 0) 0x7f325c003520 con 0x7f327806b470 2024-03-18T16:34:39.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 >> v2:172.21.15.154:3300/0 conn(0x7f327815ad60 msgr2=0x7f32780c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:39.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 --2- 172.21.15.102:0/3642712355 >> v2:172.21.15.154:3300/0 conn(0x7f327815ad60 0x7f32780c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:39.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 >> v1:172.21.15.154:6789/0 conn(0x7f327806de50 legacy=0x7f3278154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:39.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32780c9770 con 0x7f327806b470 2024-03-18T16:34:39.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f3268001670 con 0x7f327806b470 2024-03-18T16:34:39.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 >> v1:172.21.15.102:6789/0 conn(0x7f327806b470 legacy=0x7f327806b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:39.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f325c003b90 0x7f325c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:39.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f325c004800 0x7f325c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:39.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f325c0051c0 0x7f325c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:39.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f32780ca0a0 con 0x7f325c004800 2024-03-18T16:34:39.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f325c00c090 con 0x7f325c003b90 2024-03-18T16:34:39.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.906+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f325c00c290 con 0x7f325c0051c0 2024-03-18T16:34:39.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.907+0000 7f3277400640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f325c0051c0 0x7f325c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:39.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.910+0000 7f3276a00640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f325c003b90 0x7f325c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:39.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.910+0000 7f3276000640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f325c004800 0x7f325c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:39.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.911+0000 7f3277400640 1 -- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f325c003b90 msgr2=0x7f325c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:39.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.911+0000 7f3277400640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f325c003b90 0x7f325c003f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:39.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.911+0000 7f3277400640 1 -- 172.21.15.102:0/3642712355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f325c004800 msgr2=0x7f325c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:39.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.911+0000 7f3277400640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f325c004800 0x7f325c004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:39.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.911+0000 7f3277400640 1 -- 172.21.15.102:0/3642712355 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f325c003520 con 0x7f325c0051c0 2024-03-18T16:34:39.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.911+0000 7f3277400640 1 -- 172.21.15.102:0/3642712355 reap_dead start 2024-03-18T16:34:39.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.913+0000 7f3277400640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f325c0051c0 0x7f325c00b9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3270009fc0 tx=0x7f325c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f619e37462e42bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:39.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.917+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270009c10 con 0x7f325c0051c0 2024-03-18T16:34:39.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.917+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f327002cc60 con 0x7f325c0051c0 2024-03-18T16:34:39.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.917+0000 7f3275600640 1 -- 172.21.15.102:0/3642712355 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270003bd0 con 0x7f325c0051c0 2024-03-18T16:34:39.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.918+0000 7f3277e00640 1 -- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f325c0051c0 msgr2=0x7f325c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:39.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.918+0000 7f3277e00640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f325c0051c0 0x7f325c00b9c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3270009fc0 tx=0x7f325c005830 comp rx=0 tx=0).stop 2024-03-18T16:34:39.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.918+0000 7f3277e00640 1 -- 172.21.15.102:0/3642712355 shutdown_connections 2024-03-18T16:34:39.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.918+0000 7f3277e00640 1 --2- 172.21.15.102:0/3642712355 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f325c0051c0 0x7f325c00b9c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:39.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.918+0000 7f3277e00640 1 -- 172.21.15.102:0/3642712355 >> 172.21.15.102:0/3642712355 conn(0x7f32780727e0 msgr2=0x7f3278072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:39.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.918+0000 7f3277e00640 1 -- 172.21.15.102:0/3642712355 shutdown_connections 2024-03-18T16:34:39.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.918+0000 7f3277e00640 1 -- 172.21.15.102:0/3642712355 wait complete. 2024-03-18T16:34:39.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.920+0000 7f3277e00640 1 Processor -- start 2024-03-18T16:34:39.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.921+0000 7f3277e00640 1 -- start start 2024-03-18T16:34:39.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.922+0000 7f3277e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f325c0051c0 0x7f32780c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:39.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.922+0000 7f3277e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3278155d90 0x7f32781e03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:39.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.922+0000 7f3277e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32781e0900 0x7f327814a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:39.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.922+0000 7f3277e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f327806b850 con 0x7f32781e0900 2024-03-18T16:34:39.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.922+0000 7f3277e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32780c9d30 con 0x7f325c0051c0 2024-03-18T16:34:39.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.922+0000 7f3277e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f327806b9d0 con 0x7f3278155d90 2024-03-18T16:34:39.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.923+0000 7f3276a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f325c0051c0 0x7f32780c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:39.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.923+0000 7f3276000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3278155d90 0x7f32781e03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:39.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.923+0000 7f3276000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3278155d90 0x7f32781e03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:59576/0 (socket says 172.21.15.102:59576) 2024-03-18T16:34:39.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.923+0000 7f3276000640 1 -- 172.21.15.102:0/852094226 learned_addr learned my addr 172.21.15.102:0/852094226 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:39.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.923+0000 7f3277400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32781e0900 0x7f327814a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:39.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.927+0000 7f3276000640 1 -- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f325c0051c0 msgr2=0x7f32780c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:39.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.927+0000 7f3276000640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f325c0051c0 0x7f32780c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:39.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.927+0000 7f3276000640 1 -- 172.21.15.102:0/852094226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32781e0900 msgr2=0x7f327814a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:39.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.927+0000 7f3276000640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32781e0900 0x7f327814a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:39.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.927+0000 7f3276000640 1 -- 172.21.15.102:0/852094226 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32781e2d00 con 0x7f3278155d90 2024-03-18T16:34:39.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.929+0000 7f3276000640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3278155d90 0x7f32781e03c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f326400d7a0 tx=0x7f326400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:39.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.933+0000 7f326ea00640 1 -- 172.21.15.102:0/852094226 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3264005ce0 con 0x7f3278155d90 2024-03-18T16:34:39.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.933+0000 7f326ea00640 1 -- 172.21.15.102:0/852094226 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3264017070 con 0x7f3278155d90 2024-03-18T16:34:39.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.933+0000 7f326ea00640 1 -- 172.21.15.102:0/852094226 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3264007270 con 0x7f3278155d90 2024-03-18T16:34:39.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.933+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3278154cf0 con 0x7f3278155d90 2024-03-18T16:34:39.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.934+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32781552b0 con 0x7f3278155d90 2024-03-18T16:34:39.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.935+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3278073310 con 0x7f3278155d90 2024-03-18T16:34:39.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.954+0000 7f326ea00640 1 -- 172.21.15.102:0/852094226 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3264007410 con 0x7f3278155d90 2024-03-18T16:34:39.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.954+0000 7f326ea00640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f325806bc20 0x7f325806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:39.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.955+0000 7f3276a00640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f325806bc20 0x7f325806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:39.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.956+0000 7f3276a00640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f325806bc20 0x7f325806e0e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3268000c00 tx=0x7f327806ec30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:39.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:39.956+0000 7f326ea00640 1 -- 172.21.15.102:0/852094226 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f32640012d0 con 0x7f3278155d90 2024-03-18T16:34:40.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.208+0000 7f326ea00640 1 -- 172.21.15.102:0/852094226 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3278073310 con 0x7f3278155d90 2024-03-18T16:34:40.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.456+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32780cd710 con 0x7f3278155d90 2024-03-18T16:34:40.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.470+0000 7f326ea00640 1 -- 172.21.15.102:0/852094226 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f32780cd710 con 0x7f3278155d90 2024-03-18T16:34:40.472 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:40.472 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:40.472 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:40.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.472+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f325806bc20 msgr2=0x7f325806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:40.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.472+0000 7f3277e00640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f325806bc20 0x7f325806e0e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3268000c00 tx=0x7f327806ec30 comp rx=0 tx=0).stop 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.472+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3278155d90 msgr2=0x7f32781e03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.472+0000 7f3277e00640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3278155d90 0x7f32781e03c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f326400d7a0 tx=0x7f326400dc60 comp rx=0 tx=0).stop 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.473+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 shutdown_connections 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.473+0000 7f3277e00640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f325806bc20 0x7f325806e0e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.473+0000 7f3277e00640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f32781e0900 0x7f327814a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.473+0000 7f3277e00640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3278155d90 0x7f32781e03c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.473+0000 7f3277e00640 1 --2- 172.21.15.102:0/852094226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f325c0051c0 0x7f32780c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.473+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 >> 172.21.15.102:0/852094226 conn(0x7f32780727e0 msgr2=0x7f32780c45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.473+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 shutdown_connections 2024-03-18T16:34:40.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:40.473+0000 7f3277e00640 1 -- 172.21.15.102:0/852094226 wait complete. 2024-03-18T16:34:41.096 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:41.096 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:42.097 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:42.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.287+0000 7fd297e00640 1 Processor -- start 2024-03-18T16:34:42.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.287+0000 7fd297e00640 1 -- start start 2024-03-18T16:34:42.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.289+0000 7fd297e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd2980ca960 0x7fd29806c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:42.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.289+0000 7fd297e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd29815a1a0 0x7fd2980705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:42.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.289+0000 7fd297e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd29806c9c0 con 0x7fd2980cad40 2024-03-18T16:34:42.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.289+0000 7fd297e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd2980c8680 con 0x7fd2980ca960 2024-03-18T16:34:42.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.289+0000 7fd297e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd29806cca0 con 0x7fd29815a1a0 2024-03-18T16:34:42.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.290+0000 7fd297400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd29815a1a0 0x7fd2980705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:42.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.290+0000 7fd296a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd2980ca960 0x7fd29806c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:42.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.290+0000 7fd296000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd2980cad40 0x7fd29815e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:36266/0 (socket says 172.21.15.102:36266) 2024-03-18T16:34:42.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.290+0000 7fd296000640 1 -- 172.21.15.102:0/3254317673 learned_addr learned my addr 172.21.15.102:0/3254317673 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:42.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.293+0000 7fd297400640 1 -- 172.21.15.102:0/3254317673 >> v2:172.21.15.102:3300/0 conn(0x7fd2980ca960 msgr2=0x7fd29806c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:42.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.293+0000 7fd297400640 1 --2- 172.21.15.102:0/3254317673 >> v2:172.21.15.102:3300/0 conn(0x7fd2980ca960 0x7fd29806c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:42.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.293+0000 7fd297400640 1 -- 172.21.15.102:0/3254317673 >> v1:172.21.15.154:6789/0 conn(0x7fd2980cad40 legacy=0x7fd29815e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:42.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.293+0000 7fd297400640 1 -- 172.21.15.102:0/3254317673 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd29806cf80 con 0x7fd29815a1a0 2024-03-18T16:34:42.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.295+0000 7fd297400640 1 --2- 172.21.15.102:0/3254317673 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd29815a1a0 0x7fd2980705a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd290009d90 tx=0x7fd29002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81381b61903d1ec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:42.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.299+0000 7fd295600640 1 -- 172.21.15.102:0/3254317673 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290039070 con 0x7fd29815a1a0 2024-03-18T16:34:42.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.299+0000 7fd295600640 1 -- 172.21.15.102:0/3254317673 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd29002cce0 con 0x7fd29815a1a0 2024-03-18T16:34:42.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.299+0000 7fd295600640 1 -- 172.21.15.102:0/3254317673 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd290034660 con 0x7fd29815a1a0 2024-03-18T16:34:42.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.300+0000 7fd297e00640 1 -- 172.21.15.102:0/3254317673 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd29815a1a0 msgr2=0x7fd2980705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:42.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.300+0000 7fd297e00640 1 --2- 172.21.15.102:0/3254317673 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd29815a1a0 0x7fd2980705a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd290009d90 tx=0x7fd29002bf30 comp rx=0 tx=0).stop 2024-03-18T16:34:42.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.300+0000 7fd297e00640 1 -- 172.21.15.102:0/3254317673 shutdown_connections 2024-03-18T16:34:42.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.300+0000 7fd297e00640 1 --2- 172.21.15.102:0/3254317673 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd29815a1a0 0x7fd2980705a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:42.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.300+0000 7fd297e00640 1 --2- 172.21.15.102:0/3254317673 >> v2:172.21.15.102:3300/0 conn(0x7fd2980ca960 0x7fd29806c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:42.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.300+0000 7fd297e00640 1 -- 172.21.15.102:0/3254317673 >> 172.21.15.102:0/3254317673 conn(0x7fd298067190 msgr2=0x7fd2980675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:42.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.300+0000 7fd297e00640 1 -- 172.21.15.102:0/3254317673 shutdown_connections 2024-03-18T16:34:42.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.300+0000 7fd297e00640 1 -- 172.21.15.102:0/3254317673 wait complete. 2024-03-18T16:34:42.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.302+0000 7fd297e00640 1 Processor -- start 2024-03-18T16:34:42.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.302+0000 7fd297e00640 1 -- start start 2024-03-18T16:34:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.303+0000 7fd297e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2980ca960 0x7fd298071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.303+0000 7fd297e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980cad40 0x7fd2981e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.303+0000 7fd297e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd29815a1a0 0x7fd2980cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.303+0000 7fd297e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd298070ae0 con 0x7fd2980cad40 2024-03-18T16:34:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.303+0000 7fd297e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd29815dc10 con 0x7fd29815a1a0 2024-03-18T16:34:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.303+0000 7fd297e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd298070c60 con 0x7fd2980ca960 2024-03-18T16:34:42.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.304+0000 7fd297400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd29815a1a0 0x7fd2980cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:42.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.304+0000 7fd296a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2980ca960 0x7fd298071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:42.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.304+0000 7fd296000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980cad40 0x7fd2981e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:42.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.304+0000 7fd296a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2980ca960 0x7fd298071ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46028/0 (socket says 172.21.15.102:46028) 2024-03-18T16:34:42.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.304+0000 7fd296a00640 1 -- 172.21.15.102:0/1214058028 learned_addr learned my addr 172.21.15.102:0/1214058028 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.308+0000 7fd296a00640 1 -- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd29815a1a0 msgr2=0x7fd2980cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.308+0000 7fd296a00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd29815a1a0 0x7fd2980cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.308+0000 7fd296a00640 1 -- 172.21.15.102:0/1214058028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980cad40 msgr2=0x7fd2981e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.308+0000 7fd296a00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980cad40 0x7fd2981e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.308+0000 7fd296a00640 1 -- 172.21.15.102:0/1214058028 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2981f3d20 con 0x7fd2980ca960 2024-03-18T16:34:42.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.310+0000 7fd296a00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2980ca960 0x7fd298071ae0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd28800e8a0 tx=0x7fd28800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.313+0000 7fd28ea00640 1 -- 172.21.15.102:0/1214058028 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288017070 con 0x7fd2980ca960 2024-03-18T16:34:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.313+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2981f4000 con 0x7fd2980ca960 2024-03-18T16:34:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.314+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd298155f70 con 0x7fd2980ca960 2024-03-18T16:34:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.314+0000 7fd28ea00640 1 -- 172.21.15.102:0/1214058028 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd28800c940 con 0x7fd2980ca960 2024-03-18T16:34:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.314+0000 7fd28ea00640 1 -- 172.21.15.102:0/1214058028 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288021060 con 0x7fd2980ca960 2024-03-18T16:34:42.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.315+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd298074fd0 con 0x7fd2980ca960 2024-03-18T16:34:42.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.336+0000 7fd28ea00640 1 -- 172.21.15.102:0/1214058028 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd28801a070 con 0x7fd2980ca960 2024-03-18T16:34:42.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.338+0000 7fd28ea00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd27806bbd0 0x7fd27806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:42.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.338+0000 7fd296000640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd27806bbd0 0x7fd27806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:42.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.339+0000 7fd296000640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd27806bbd0 0x7fd27806e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd298067a50 tx=0x7fd284007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:42.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.385+0000 7fd28ea00640 1 -- 172.21.15.102:0/1214058028 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd2880576d0 con 0x7fd2980ca960 2024-03-18T16:34:42.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.590+0000 7fd28ea00640 1 -- 172.21.15.102:0/1214058028 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd288059630 con 0x7fd2980ca960 2024-03-18T16:34:42.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.876+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd298158440 con 0x7fd2980ca960 2024-03-18T16:34:42.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.890+0000 7fd28ea00640 1 -- 172.21.15.102:0/1214058028 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd28805d6d0 con 0x7fd2980ca960 2024-03-18T16:34:42.892 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:42.892 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:42.892 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:42.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.892+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd27806bbd0 msgr2=0x7fd27806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:42.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.892+0000 7fd297e00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd27806bbd0 0x7fd27806e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd298067a50 tx=0x7fd284007e80 comp rx=0 tx=0).stop 2024-03-18T16:34:42.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.892+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2980ca960 msgr2=0x7fd298071ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:42.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.892+0000 7fd297e00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2980ca960 0x7fd298071ae0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd28800e8a0 tx=0x7fd28800ed60 comp rx=0 tx=0).stop 2024-03-18T16:34:42.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.893+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 shutdown_connections 2024-03-18T16:34:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.893+0000 7fd297e00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd27806bbd0 0x7fd27806e090 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.893+0000 7fd297e00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd29815a1a0 0x7fd2980cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.893+0000 7fd297e00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2980cad40 0x7fd2981e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.893+0000 7fd297e00640 1 --2- 172.21.15.102:0/1214058028 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd2980ca960 0x7fd298071ae0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.893+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 >> 172.21.15.102:0/1214058028 conn(0x7fd298067190 msgr2=0x7fd298071100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.893+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 shutdown_connections 2024-03-18T16:34:42.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:42.893+0000 7fd297e00640 1 -- 172.21.15.102:0/1214058028 wait complete. 2024-03-18T16:34:43.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:43.515 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:44.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:44.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.689+0000 7f52e6000640 1 Processor -- start 2024-03-18T16:34:44.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.689+0000 7f52e6000640 1 -- start start 2024-03-18T16:34:44.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.690+0000 7f52e6000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f52e00d0250 0x7f52e015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:44.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.690+0000 7f52e6000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52e00cc2f0 0x7f52e00cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:44.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.690+0000 7f52e6000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e00ccc80 con 0x7f52e00cfe70 2024-03-18T16:34:44.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.690+0000 7f52e6000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f52e00cabd0 con 0x7f52e00d0250 2024-03-18T16:34:44.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.690+0000 7f52e6000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f52e00caea0 con 0x7f52e00cc2f0 2024-03-18T16:34:44.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.691+0000 7f52e5600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52e00cc2f0 0x7f52e00cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:44.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.691+0000 7f52dfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f52e00d0250 0x7f52e015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:44.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.692+0000 7f52dfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f52e00d0250 0x7f52e015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54582/0 (socket says 172.21.15.102:54582) 2024-03-18T16:34:44.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.692+0000 7f52dfe00640 1 -- 172.21.15.102:0/4061286943 learned_addr learned my addr 172.21.15.102:0/4061286943 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:44.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.694+0000 7f52e5600640 1 -- 172.21.15.102:0/4061286943 >> v2:172.21.15.102:3300/0 conn(0x7f52e00d0250 msgr2=0x7f52e015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:44.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.694+0000 7f52e5600640 1 --2- 172.21.15.102:0/4061286943 >> v2:172.21.15.102:3300/0 conn(0x7f52e00d0250 0x7f52e015ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:44.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.694+0000 7f52e5600640 1 -- 172.21.15.102:0/4061286943 >> v1:172.21.15.102:6789/0 conn(0x7f52e00cfe70 legacy=0x7f52e00c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:44.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.694+0000 7f52e5600640 1 -- 172.21.15.102:0/4061286943 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e00cb180 con 0x7f52e00cc2f0 2024-03-18T16:34:44.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.696+0000 7f52e5600640 1 --2- 172.21.15.102:0/4061286943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00cc2f0 0x7f52e00cc740 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f52e00ca160 tx=0x7f52d802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9050c07989d82f32 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:44.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.699+0000 7f52df400640 1 -- 172.21.15.102:0/4061286943 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8039070 con 0x7f52e00cc2f0 2024-03-18T16:34:44.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.700+0000 7f52df400640 1 -- 172.21.15.102:0/4061286943 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d802cd10 con 0x7f52e00cc2f0 2024-03-18T16:34:44.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.700+0000 7f52df400640 1 -- 172.21.15.102:0/4061286943 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8034660 con 0x7f52e00cc2f0 2024-03-18T16:34:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.700+0000 7f52e6000640 1 -- 172.21.15.102:0/4061286943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00cc2f0 msgr2=0x7f52e00cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.700+0000 7f52e6000640 1 --2- 172.21.15.102:0/4061286943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00cc2f0 0x7f52e00cc740 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f52e00ca160 tx=0x7f52d802bf60 comp rx=0 tx=0).stop 2024-03-18T16:34:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.700+0000 7f52e6000640 1 -- 172.21.15.102:0/4061286943 shutdown_connections 2024-03-18T16:34:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.700+0000 7f52e6000640 1 --2- 172.21.15.102:0/4061286943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00cc2f0 0x7f52e00cc740 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.700+0000 7f52e6000640 1 --2- 172.21.15.102:0/4061286943 >> v2:172.21.15.102:3300/0 conn(0x7f52e00d0250 0x7f52e015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:44.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.700+0000 7f52e6000640 1 -- 172.21.15.102:0/4061286943 >> 172.21.15.102:0/4061286943 conn(0x7f52e008c3e0 msgr2=0x7f52e006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:44.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.701+0000 7f52e6000640 1 -- 172.21.15.102:0/4061286943 shutdown_connections 2024-03-18T16:34:44.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.701+0000 7f52e6000640 1 -- 172.21.15.102:0/4061286943 wait complete. 2024-03-18T16:34:44.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.702+0000 7f52e6000640 1 Processor -- start 2024-03-18T16:34:44.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.703+0000 7f52e6000640 1 -- start start 2024-03-18T16:34:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.704+0000 7f52e6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52e00cc2f0 0x7f52e01e18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.704+0000 7f52e6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52e00cfe70 0x7f52e01f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.704+0000 7f52e6000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00d0250 0x7f52e01557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:44.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.704+0000 7f52e6000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f52e00cba20 con 0x7f52e00cfe70 2024-03-18T16:34:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.704+0000 7f52e6000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52e00cb740 con 0x7f52e00d0250 2024-03-18T16:34:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.704+0000 7f52e6000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f52e00cbd00 con 0x7f52e00cc2f0 2024-03-18T16:34:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.704+0000 7f52e4c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52e00cc2f0 0x7f52e01e18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.704+0000 7f52e5600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00d0250 0x7f52e01557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:44.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.705+0000 7f52dfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52e00cfe70 0x7f52e01f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:44.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.705+0000 7f52e5600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00d0250 0x7f52e01557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:53382/0 (socket says 172.21.15.102:53382) 2024-03-18T16:34:44.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.705+0000 7f52e5600640 1 -- 172.21.15.102:0/3775097408 learned_addr learned my addr 172.21.15.102:0/3775097408 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:44.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.708+0000 7f52e5600640 1 -- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52e00cc2f0 msgr2=0x7f52e01e18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:44.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.708+0000 7f52e5600640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52e00cc2f0 0x7f52e01e18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:44.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.708+0000 7f52e5600640 1 -- 172.21.15.102:0/3775097408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52e00cfe70 msgr2=0x7f52e01f0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:44.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.708+0000 7f52e5600640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52e00cfe70 0x7f52e01f0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:44.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.708+0000 7f52e5600640 1 -- 172.21.15.102:0/3775097408 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e01e03b0 con 0x7f52e00d0250 2024-03-18T16:34:44.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.711+0000 7f52e5600640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00d0250 0x7f52e01557c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009e80 tx=0x7f52d8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:44.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.716+0000 7f52dcc00640 1 -- 172.21.15.102:0/3775097408 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8039050 con 0x7f52e00d0250 2024-03-18T16:34:44.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.716+0000 7f52dcc00640 1 -- 172.21.15.102:0/3775097408 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d8045070 con 0x7f52e00d0250 2024-03-18T16:34:44.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.716+0000 7f52dcc00640 1 -- 172.21.15.102:0/3775097408 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8009450 con 0x7f52e00d0250 2024-03-18T16:34:44.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.716+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e01e0680 con 0x7f52e00d0250 2024-03-18T16:34:44.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.716+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e01512f0 con 0x7f52e00d0250 2024-03-18T16:34:44.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:44.717+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e0074fb0 con 0x7f52e00d0250 2024-03-18T16:34:45.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.002+0000 7f52dcc00640 1 -- 172.21.15.102:0/3775097408 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f52d803e850 con 0x7f52e00d0250 2024-03-18T16:34:45.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.003+0000 7f52dcc00640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52c006bc50 0x7f52c006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:45.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.003+0000 7f52e4c00640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52c006bc50 0x7f52c006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:45.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.004+0000 7f52e4c00640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52c006bc50 0x7f52c006e110 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f52d4001780 tx=0x7f52d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:45.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.005+0000 7f52dcc00640 1 -- 172.21.15.102:0/3775097408 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f52d807a070 con 0x7f52e00d0250 2024-03-18T16:34:45.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.020+0000 7f52dcc00640 1 -- 172.21.15.102:0/3775097408 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f52d804e020 con 0x7f52e00d0250 2024-03-18T16:34:45.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.268+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52e015a4f0 con 0x7f52e00d0250 2024-03-18T16:34:45.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.324+0000 7f52dcc00640 1 -- 172.21.15.102:0/3775097408 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f52d807bfd0 con 0x7f52e00d0250 2024-03-18T16:34:45.326 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:45.326 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:45.327 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.328+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52c006bc50 msgr2=0x7f52c006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.328+0000 7f52e6000640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52c006bc50 0x7f52c006e110 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f52d4001780 tx=0x7f52d4009040 comp rx=0 tx=0).stop 2024-03-18T16:34:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.328+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00d0250 msgr2=0x7f52e01557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.328+0000 7f52e6000640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00d0250 0x7f52e01557c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009e80 tx=0x7f52d8004a40 comp rx=0 tx=0).stop 2024-03-18T16:34:45.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.328+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 shutdown_connections 2024-03-18T16:34:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.329+0000 7f52e6000640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52c006bc50 0x7f52c006e110 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.329+0000 7f52e6000640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52e00d0250 0x7f52e01557c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.329+0000 7f52e6000640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52e00cfe70 0x7f52e01f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.329+0000 7f52e6000640 1 --2- 172.21.15.102:0/3775097408 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52e00cc2f0 0x7f52e01e18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.329+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 >> 172.21.15.102:0/3775097408 conn(0x7f52e008c3e0 msgr2=0x7f52e00cce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.329+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 shutdown_connections 2024-03-18T16:34:45.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:45.329+0000 7f52e6000640 1 -- 172.21.15.102:0/3775097408 wait complete. 2024-03-18T16:34:45.956 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:45.957 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:46.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.132+0000 7f24f6600640 1 Processor -- start 2024-03-18T16:34:47.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.132+0000 7f24f6600640 1 -- start start 2024-03-18T16:34:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.133+0000 7f24f6600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24f0158420 0x7f24f015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.133+0000 7f24f6600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f00c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.133+0000 7f24f6600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f00c72d0 con 0x7f24f0141650 2024-03-18T16:34:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.133+0000 7f24f6600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f24f015c8e0 con 0x7f24f0158420 2024-03-18T16:34:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.133+0000 7f24f6600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24f0159370 con 0x7f24f00c4fa0 2024-03-18T16:34:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.134+0000 7f24efe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24f0158420 0x7f24f015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:47.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.134+0000 7f24f5c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f00c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.135+0000 7f24efe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24f0158420 0x7f24f015c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54594/0 (socket says 172.21.15.102:54594) 2024-03-18T16:34:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.135+0000 7f24f5c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f00c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46046/0 (socket says 172.21.15.102:46046) 2024-03-18T16:34:47.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.135+0000 7f24efe00640 1 -- 172.21.15.102:0/1434337109 learned_addr learned my addr 172.21.15.102:0/1434337109 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:47.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.138+0000 7f24efe00640 1 -- 172.21.15.102:0/1434337109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 msgr2=0x7f24f00c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:47.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.138+0000 7f24efe00640 1 --2- 172.21.15.102:0/1434337109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f00c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.138+0000 7f24efe00640 1 -- 172.21.15.102:0/1434337109 >> v1:172.21.15.102:6789/0 conn(0x7f24f0141650 legacy=0x7f24f00c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:47.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.138+0000 7f24efe00640 1 -- 172.21.15.102:0/1434337109 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f0158840 con 0x7f24f0158420 2024-03-18T16:34:47.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.141+0000 7f24efe00640 1 --2- 172.21.15.102:0/1434337109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f0158420 0x7f24f015c3a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009870 tx=0x7f24dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ac3eb6b3e2d044 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:47.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.147+0000 7f24ef400640 1 -- 172.21.15.102:0/1434337109 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc035020 con 0x7f24f0158420 2024-03-18T16:34:47.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.147+0000 7f24ef400640 1 -- 172.21.15.102:0/1434337109 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24dc031640 con 0x7f24f0158420 2024-03-18T16:34:47.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.147+0000 7f24ef400640 1 -- 172.21.15.102:0/1434337109 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc02d040 con 0x7f24f0158420 2024-03-18T16:34:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.148+0000 7f24f6600640 1 -- 172.21.15.102:0/1434337109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f0158420 msgr2=0x7f24f015c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.148+0000 7f24f6600640 1 --2- 172.21.15.102:0/1434337109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f0158420 0x7f24f015c3a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009870 tx=0x7f24dc02b8f0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.149+0000 7f24f6600640 1 -- 172.21.15.102:0/1434337109 shutdown_connections 2024-03-18T16:34:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.149+0000 7f24f6600640 1 --2- 172.21.15.102:0/1434337109 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f00c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.149+0000 7f24f6600640 1 --2- 172.21.15.102:0/1434337109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f0158420 0x7f24f015c3a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.149+0000 7f24f6600640 1 -- 172.21.15.102:0/1434337109 >> 172.21.15.102:0/1434337109 conn(0x7f24f00cb890 msgr2=0x7f24f00cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:47.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.149+0000 7f24f6600640 1 -- 172.21.15.102:0/1434337109 shutdown_connections 2024-03-18T16:34:47.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.149+0000 7f24f6600640 1 -- 172.21.15.102:0/1434337109 wait complete. 2024-03-18T16:34:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.151+0000 7f24f6600640 1 Processor -- start 2024-03-18T16:34:47.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.151+0000 7f24f6600640 1 -- start start 2024-03-18T16:34:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.152+0000 7f24f6600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f0137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.152+0000 7f24f6600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f0141650 0x7f24f00c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.152+0000 7f24f6600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24f0158420 0x7f24f01351d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:47.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.152+0000 7f24f6600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f24f0159090 con 0x7f24f0141650 2024-03-18T16:34:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.152+0000 7f24f6600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24f0158db0 con 0x7f24f0158420 2024-03-18T16:34:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.152+0000 7f24f6600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f24f00c63c0 con 0x7f24f00c4fa0 2024-03-18T16:34:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.152+0000 7f24f5200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f0137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.153+0000 7f24efe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f0141650 0x7f24f00c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.153+0000 7f24f5200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f0137230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46054/0 (socket says 172.21.15.102:46054) 2024-03-18T16:34:47.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.153+0000 7f24f5200640 1 -- 172.21.15.102:0/3873551523 learned_addr learned my addr 172.21.15.102:0/3873551523 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.158+0000 7f24f5200640 1 -- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24f0158420 msgr2=0x7f24f01351d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.158+0000 7f24f5200640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24f0158420 0x7f24f01351d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.158+0000 7f24f5200640 1 -- 172.21.15.102:0/3873551523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f0141650 msgr2=0x7f24f00c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.158+0000 7f24f5200640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f0141650 0x7f24f00c9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.158+0000 7f24f5200640 1 -- 172.21.15.102:0/3873551523 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f0135710 con 0x7f24f00c4fa0 2024-03-18T16:34:47.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.161+0000 7f24f5200640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f0137230 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f24e400c430 tx=0x7f24e400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.166+0000 7f24ecc00640 1 -- 172.21.15.102:0/3873551523 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4011790 con 0x7f24f00c4fa0 2024-03-18T16:34:47.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.166+0000 7f24ecc00640 1 -- 172.21.15.102:0/3873551523 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e4017070 con 0x7f24f00c4fa0 2024-03-18T16:34:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.166+0000 7f24ecc00640 1 -- 172.21.15.102:0/3873551523 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e4011bb0 con 0x7f24f00c4fa0 2024-03-18T16:34:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.166+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f01364b0 con 0x7f24f00c4fa0 2024-03-18T16:34:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.166+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f0136750 con 0x7f24f00c4fa0 2024-03-18T16:34:47.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.167+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f00c5f80 con 0x7f24f00c4fa0 2024-03-18T16:34:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.186+0000 7f24ecc00640 1 -- 172.21.15.102:0/3873551523 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f24e4007020 con 0x7f24f00c4fa0 2024-03-18T16:34:47.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.187+0000 7f24ecc00640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24d006bc20 0x7f24d006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:47.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.188+0000 7f24efe00640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24d006bc20 0x7f24d006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:47.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.188+0000 7f24efe00640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24d006bc20 0x7f24d006e0e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f24dc036000 tx=0x7f24dc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:47.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.190+0000 7f24ecc00640 1 -- 172.21.15.102:0/3873551523 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f24e40574b0 con 0x7f24f00c4fa0 2024-03-18T16:34:47.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.438+0000 7f24ecc00640 1 -- 172.21.15.102:0/3873551523 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f24f00c5f80 con 0x7f24f00c4fa0 2024-03-18T16:34:47.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.679+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24f0074fd0 con 0x7f24f00c4fa0 2024-03-18T16:34:47.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.693+0000 7f24ecc00640 1 -- 172.21.15.102:0/3873551523 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f24e4021020 con 0x7f24f00c4fa0 2024-03-18T16:34:47.695 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:47.695 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:47.695 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:47.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.695+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24d006bc20 msgr2=0x7f24d006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:47.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.695+0000 7f24f6600640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24d006bc20 0x7f24d006e0e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f24dc036000 tx=0x7f24dc03c040 comp rx=0 tx=0).stop 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.695+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 msgr2=0x7f24f0137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.695+0000 7f24f6600640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f0137230 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f24e400c430 tx=0x7f24e400c8f0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.696+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 shutdown_connections 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.696+0000 7f24f6600640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f24d006bc20 0x7f24d006e0e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.696+0000 7f24f6600640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24f0158420 0x7f24f01351d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.696+0000 7f24f6600640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24f0141650 0x7f24f00c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.696+0000 7f24f6600640 1 --2- 172.21.15.102:0/3873551523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f24f00c4fa0 0x7f24f0137230 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.696+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 >> 172.21.15.102:0/3873551523 conn(0x7f24f00cb890 msgr2=0x7f24f0157640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.696+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 shutdown_connections 2024-03-18T16:34:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:47.696+0000 7f24f6600640 1 -- 172.21.15.102:0/3873551523 wait complete. 2024-03-18T16:34:48.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:48.318 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:49.318 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:49.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.506+0000 7f15fdc00640 1 Processor -- start 2024-03-18T16:34:49.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.507+0000 7f15fdc00640 1 -- start start 2024-03-18T16:34:49.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.508+0000 7f15fdc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f81524c0 0x7f15f8072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.508+0000 7f15fdc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f15f815b8e0 0x7f15f80c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.508+0000 7f15fdc00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15f815c200 con 0x7f15f8158ee0 2024-03-18T16:34:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.508+0000 7f15fdc00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f15f8159300 con 0x7f15f815b8e0 2024-03-18T16:34:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.508+0000 7f15fdc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f15f815c4e0 con 0x7f15f81524c0 2024-03-18T16:34:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.509+0000 7f15f6000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f15f8158ee0 0x7f15f806e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:36270/0 (socket says 172.21.15.102:36270) 2024-03-18T16:34:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.509+0000 7f15f6000640 1 -- 172.21.15.102:0/1891954995 learned_addr learned my addr 172.21.15.102:0/1891954995 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:49.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.509+0000 7f15f6a00640 1 --2- 172.21.15.102:0/1891954995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f81524c0 0x7f15f8072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:49.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.510+0000 7f15f7400640 1 --2- 172.21.15.102:0/1891954995 >> v2:172.21.15.154:3300/0 conn(0x7f15f815b8e0 0x7f15f80c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:49.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.512+0000 7f15f6a00640 1 -- 172.21.15.102:0/1891954995 >> v2:172.21.15.154:3300/0 conn(0x7f15f815b8e0 msgr2=0x7f15f80c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:49.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.512+0000 7f15f6a00640 1 --2- 172.21.15.102:0/1891954995 >> v2:172.21.15.154:3300/0 conn(0x7f15f815b8e0 0x7f15f80c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:49.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.512+0000 7f15f6a00640 1 -- 172.21.15.102:0/1891954995 >> v1:172.21.15.154:6789/0 conn(0x7f15f8158ee0 legacy=0x7f15f806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:49.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.512+0000 7f15f6a00640 1 -- 172.21.15.102:0/1891954995 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f815c7c0 con 0x7f15f81524c0 2024-03-18T16:34:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.514+0000 7f15f6a00640 1 --2- 172.21.15.102:0/1891954995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f81524c0 0x7f15f8072700 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f15e80097a0 tx=0x7f15e802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c68e6d5ac72447d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.519+0000 7f15f5600640 1 -- 172.21.15.102:0/1891954995 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8035020 con 0x7f15f81524c0 2024-03-18T16:34:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.519+0000 7f15f5600640 1 -- 172.21.15.102:0/1891954995 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15e8031830 con 0x7f15f81524c0 2024-03-18T16:34:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.519+0000 7f15f5600640 1 -- 172.21.15.102:0/1891954995 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8031c40 con 0x7f15f81524c0 2024-03-18T16:34:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.520+0000 7f15fdc00640 1 -- 172.21.15.102:0/1891954995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f81524c0 msgr2=0x7f15f8072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.520+0000 7f15fdc00640 1 --2- 172.21.15.102:0/1891954995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f81524c0 0x7f15f8072700 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f15e80097a0 tx=0x7f15e802b8d0 comp rx=0 tx=0).stop 2024-03-18T16:34:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.520+0000 7f15fdc00640 1 -- 172.21.15.102:0/1891954995 shutdown_connections 2024-03-18T16:34:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.520+0000 7f15fdc00640 1 --2- 172.21.15.102:0/1891954995 >> v2:172.21.15.154:3300/0 conn(0x7f15f815b8e0 0x7f15f80c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.520+0000 7f15fdc00640 1 --2- 172.21.15.102:0/1891954995 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f81524c0 0x7f15f8072700 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.520+0000 7f15fdc00640 1 -- 172.21.15.102:0/1891954995 >> 172.21.15.102:0/1891954995 conn(0x7f15f80700d0 msgr2=0x7f15f806b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.520+0000 7f15fdc00640 1 -- 172.21.15.102:0/1891954995 shutdown_connections 2024-03-18T16:34:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.520+0000 7f15fdc00640 1 -- 172.21.15.102:0/1891954995 wait complete. 2024-03-18T16:34:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.522+0000 7f15fdc00640 1 Processor -- start 2024-03-18T16:34:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.522+0000 7f15fdc00640 1 -- start start 2024-03-18T16:34:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.523+0000 7f15fdc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f81524c0 0x7f15f81569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.523+0000 7f15fdc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f8158ee0 0x7f15f81f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.523+0000 7f15fdc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f815b8e0 0x7f15f81f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.523+0000 7f15fdc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f15f81589c0 con 0x7f15f815b8e0 2024-03-18T16:34:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.523+0000 7f15fdc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15f81586e0 con 0x7f15f81524c0 2024-03-18T16:34:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.523+0000 7f15fdc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f15f81528a0 con 0x7f15f8158ee0 2024-03-18T16:34:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.524+0000 7f15f6a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f81524c0 0x7f15f81569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.524+0000 7f15f6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f8158ee0 0x7f15f81f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.524+0000 7f15f6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f8158ee0 0x7f15f81f3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:46076/0 (socket says 172.21.15.102:46076) 2024-03-18T16:34:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.524+0000 7f15f6000640 1 -- 172.21.15.102:0/2810329207 learned_addr learned my addr 172.21.15.102:0/2810329207 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.529+0000 7f15f6a00640 1 -- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f8158ee0 msgr2=0x7f15f81f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.529+0000 7f15f6a00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f8158ee0 0x7f15f81f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.529+0000 7f15f6a00640 1 -- 172.21.15.102:0/2810329207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f815b8e0 msgr2=0x7f15f81f3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.529+0000 7f15f6a00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f815b8e0 0x7f15f81f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:49.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.529+0000 7f15f6a00640 1 -- 172.21.15.102:0/2810329207 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f80c6b10 con 0x7f15f81524c0 2024-03-18T16:34:49.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.533+0000 7f15f6a00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f81524c0 0x7f15f81569b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f15e8033000 tx=0x7f15e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:49.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.537+0000 7f15eea00640 1 -- 172.21.15.102:0/2810329207 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8040070 con 0x7f15f81524c0 2024-03-18T16:34:49.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.537+0000 7f15eea00640 1 -- 172.21.15.102:0/2810329207 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15e803a910 con 0x7f15f81524c0 2024-03-18T16:34:49.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.537+0000 7f15eea00640 1 -- 172.21.15.102:0/2810329207 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e8035040 con 0x7f15f81524c0 2024-03-18T16:34:49.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.537+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f8152ce0 con 0x7f15f81524c0 2024-03-18T16:34:49.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.537+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f81532a0 con 0x7f15f81524c0 2024-03-18T16:34:49.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.538+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f8159c40 con 0x7f15f81524c0 2024-03-18T16:34:49.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.558+0000 7f15eea00640 1 -- 172.21.15.102:0/2810329207 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f15e8038020 con 0x7f15f81524c0 2024-03-18T16:34:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.559+0000 7f15eea00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15d806bc50 0x7f15d806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.559+0000 7f15f6000640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15d806bc50 0x7f15d806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:49.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.560+0000 7f15f6000640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15d806bc50 0x7f15d806e110 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f15f81481b0 tx=0x7f15e4007f10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:49.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.809+0000 7f15eea00640 1 -- 172.21.15.102:0/2810329207 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f15e80317e0 con 0x7f15f81524c0 2024-03-18T16:34:49.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:49.820+0000 7f15eea00640 1 -- 172.21.15.102:0/2810329207 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f15f8159c40 con 0x7f15f81524c0 2024-03-18T16:34:50.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.056+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15f8074fd0 con 0x7f15f81524c0 2024-03-18T16:34:50.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.069+0000 7f15eea00640 1 -- 172.21.15.102:0/2810329207 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f15e807b210 con 0x7f15f81524c0 2024-03-18T16:34:50.071 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:50.071 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:50.071 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:50.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15d806bc50 msgr2=0x7f15d806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:50.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15d806bc50 0x7f15d806e110 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f15f81481b0 tx=0x7f15e4007f10 comp rx=0 tx=0).stop 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f81524c0 msgr2=0x7f15f81569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f81524c0 0x7f15f81569b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f15e8033000 tx=0x7f15e80046c0 comp rx=0 tx=0).stop 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 shutdown_connections 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15d806bc50 0x7f15d806e110 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f815b8e0 0x7f15f81f3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15f8158ee0 0x7f15f81f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 --2- 172.21.15.102:0/2810329207 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f81524c0 0x7f15f81569b0 secure :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f15e8033000 tx=0x7f15e80046c0 comp rx=0 tx=0).stop 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 >> 172.21.15.102:0/2810329207 conn(0x7f15f80700d0 msgr2=0x7f15f815a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 shutdown_connections 2024-03-18T16:34:50.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:50.073+0000 7f15fdc00640 1 -- 172.21.15.102:0/2810329207 wait complete. 2024-03-18T16:34:50.697 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:50.697 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:51.698 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:51.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.874+0000 7f7286400640 1 Processor -- start 2024-03-18T16:34:51.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.874+0000 7f7286400640 1 -- start start 2024-03-18T16:34:51.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.875+0000 7f7286400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7280141650 0x7f7280158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:51.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.875+0000 7f7286400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f72800c4fa0 0x7f72800c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:51.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.875+0000 7f7286400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72800c72d0 con 0x7f728015c3a0 2024-03-18T16:34:51.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.875+0000 7f7286400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7280159370 con 0x7f7280141650 2024-03-18T16:34:51.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.875+0000 7f7286400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7280158ad0 con 0x7f72800c4fa0 2024-03-18T16:34:51.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.876+0000 7f7285000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7280141650 0x7f7280158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:51.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.876+0000 7f7285a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f72800c4fa0 0x7f72800c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:51.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.877+0000 7f7285000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7280141650 0x7f7280158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57056/0 (socket says 172.21.15.102:57056) 2024-03-18T16:34:51.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.877+0000 7f7285000640 1 -- 172.21.15.102:0/4121172774 learned_addr learned my addr 172.21.15.102:0/4121172774 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.879+0000 7f7285a00640 1 -- 172.21.15.102:0/4121172774 >> v2:172.21.15.102:3300/0 conn(0x7f7280141650 msgr2=0x7f7280158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.879+0000 7f7285a00640 1 --2- 172.21.15.102:0/4121172774 >> v2:172.21.15.102:3300/0 conn(0x7f7280141650 0x7f7280158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.879+0000 7f7285a00640 1 -- 172.21.15.102:0/4121172774 >> v1:172.21.15.102:6789/0 conn(0x7f728015c3a0 legacy=0x7f72800c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:51.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.879+0000 7f7285a00640 1 -- 172.21.15.102:0/4121172774 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7280158db0 con 0x7f72800c4fa0 2024-03-18T16:34:51.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.881+0000 7f7285a00640 1 --2- 172.21.15.102:0/4121172774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72800c4fa0 0x7f72800c5410 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f72800c9720 tx=0x7f727802bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dda688854a033af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:51.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.886+0000 7f727f400640 1 -- 172.21.15.102:0/4121172774 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7278039070 con 0x7f72800c4fa0 2024-03-18T16:34:51.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.886+0000 7f727f400640 1 -- 172.21.15.102:0/4121172774 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f727802cc70 con 0x7f72800c4fa0 2024-03-18T16:34:51.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.886+0000 7f727f400640 1 -- 172.21.15.102:0/4121172774 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7278034660 con 0x7f72800c4fa0 2024-03-18T16:34:51.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.887+0000 7f7286400640 1 -- 172.21.15.102:0/4121172774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72800c4fa0 msgr2=0x7f72800c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.887+0000 7f7286400640 1 --2- 172.21.15.102:0/4121172774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72800c4fa0 0x7f72800c5410 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f72800c9720 tx=0x7f727802bec0 comp rx=0 tx=0).stop 2024-03-18T16:34:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.887+0000 7f7286400640 1 -- 172.21.15.102:0/4121172774 shutdown_connections 2024-03-18T16:34:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.887+0000 7f7286400640 1 --2- 172.21.15.102:0/4121172774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72800c4fa0 0x7f72800c5410 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.887+0000 7f7286400640 1 --2- 172.21.15.102:0/4121172774 >> v2:172.21.15.102:3300/0 conn(0x7f7280141650 0x7f7280158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.887+0000 7f7286400640 1 -- 172.21.15.102:0/4121172774 >> 172.21.15.102:0/4121172774 conn(0x7f72800cb890 msgr2=0x7f72800cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.887+0000 7f7286400640 1 -- 172.21.15.102:0/4121172774 shutdown_connections 2024-03-18T16:34:51.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.887+0000 7f7286400640 1 -- 172.21.15.102:0/4121172774 wait complete. 2024-03-18T16:34:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.889+0000 7f7286400640 1 Processor -- start 2024-03-18T16:34:51.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.889+0000 7f7286400640 1 -- start start 2024-03-18T16:34:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.890+0000 7f7286400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72800c4fa0 0x7f728013bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.890+0000 7f7286400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7280141650 0x7f72801f14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.890+0000 7f7286400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f728015c3a0 0x7f72801f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.890+0000 7f7286400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f72800c66a0 con 0x7f72800c4fa0 2024-03-18T16:34:51.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.890+0000 7f7286400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f72800c63c0 con 0x7f7280141650 2024-03-18T16:34:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.890+0000 7f7286400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f728015c7c0 con 0x7f728015c3a0 2024-03-18T16:34:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.890+0000 7f727fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7280141650 0x7f72801f14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.891+0000 7f7285a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f728015c3a0 0x7f72801f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.891+0000 7f7285000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72800c4fa0 0x7f728013bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.891+0000 7f727fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7280141650 0x7f72801f14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:36166/0 (socket says 172.21.15.102:36166) 2024-03-18T16:34:51.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.891+0000 7f727fe00640 1 -- 172.21.15.102:0/1449178395 learned_addr learned my addr 172.21.15.102:0/1449178395 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:51.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.894+0000 7f7285a00640 1 -- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7280141650 msgr2=0x7f72801f14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:51.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.894+0000 7f7285a00640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7280141650 0x7f72801f14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:51.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.894+0000 7f7285a00640 1 -- 172.21.15.102:0/1449178395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72800c4fa0 msgr2=0x7f728013bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:51.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.894+0000 7f7285a00640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72800c4fa0 0x7f728013bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:51.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.894+0000 7f7285a00640 1 -- 172.21.15.102:0/1449178395 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728015bfa0 con 0x7f728015c3a0 2024-03-18T16:34:51.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.897+0000 7f7285a00640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f728015c3a0 0x7f72801f19f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f72801594f0 tx=0x7f7278004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:51.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.899+0000 7f727cc00640 1 -- 172.21.15.102:0/1449178395 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7278039050 con 0x7f728015c3a0 2024-03-18T16:34:51.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.899+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728015bcc0 con 0x7f728015c3a0 2024-03-18T16:34:51.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.899+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f728013ad80 con 0x7f728015c3a0 2024-03-18T16:34:51.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:51.900+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7280074fd0 con 0x7f728015c3a0 2024-03-18T16:34:52.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.106+0000 7f727cc00640 1 -- 172.21.15.102:0/1449178395 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7278045070 con 0x7f728015c3a0 2024-03-18T16:34:52.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.210+0000 7f727cc00640 1 -- 172.21.15.102:0/1449178395 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7278009450 con 0x7f728015c3a0 2024-03-18T16:34:52.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.212+0000 7f727cc00640 1 -- 172.21.15.102:0/1449178395 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f727803e890 con 0x7f728015c3a0 2024-03-18T16:34:52.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.214+0000 7f727cc00640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f726006bbd0 0x7f726006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:52.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.215+0000 7f7285000640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f726006bbd0 0x7f726006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:52.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.215+0000 7f727cc00640 1 -- 172.21.15.102:0/1449178395 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f72780ae510 con 0x7f728015c3a0 2024-03-18T16:34:52.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.215+0000 7f7285000640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f726006bbd0 0x7f726006e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7274009bd0 tx=0x7f7274009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:52.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.225+0000 7f727cc00640 1 -- 172.21.15.102:0/1449178395 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f727802ce20 con 0x7f728015c3a0 2024-03-18T16:34:52.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.463+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72800b7f00 con 0x7f728015c3a0 2024-03-18T16:34:52.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.475+0000 7f727cc00640 1 -- 172.21.15.102:0/1449178395 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f727807c4c0 con 0x7f728015c3a0 2024-03-18T16:34:52.477 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:52.477 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:52.477 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f726006bbd0 msgr2=0x7f726006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f726006bbd0 0x7f726006e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7274009bd0 tx=0x7f7274009240 comp rx=0 tx=0).stop 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f728015c3a0 msgr2=0x7f72801f19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f728015c3a0 0x7f72801f19f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f72801594f0 tx=0x7f7278004a40 comp rx=0 tx=0).stop 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 shutdown_connections 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f726006bbd0 0x7f726006e090 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f728015c3a0 0x7f72801f19f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7280141650 0x7f72801f14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 --2- 172.21.15.102:0/1449178395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72800c4fa0 0x7f728013bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:52.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 >> 172.21.15.102:0/1449178395 conn(0x7f72800cb890 msgr2=0x7f72800cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:52.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 shutdown_connections 2024-03-18T16:34:52.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:52.479+0000 7f7286400640 1 -- 172.21.15.102:0/1449178395 wait complete. 2024-03-18T16:34:53.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:53.101 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:54.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:34:54.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:34:54.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.279+0000 7f4d69c00640 1 Processor -- start 2024-03-18T16:34:54.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.279+0000 7f4d69c00640 1 -- start start 2024-03-18T16:34:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.281+0000 7f4d69c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4d640cc380 0x7f4d64156830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.281+0000 7f4d69c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d641504d0 0x7f4d64150960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.281+0000 7f4d69c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d6406ffb0 con 0x7f4d640c7820 2024-03-18T16:34:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.281+0000 7f4d69c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4d640c2d90 con 0x7f4d640cc380 2024-03-18T16:34:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.281+0000 7f4d69c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4d640c2fe0 con 0x7f4d641504d0 2024-03-18T16:34:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.282+0000 7f4d69200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d641504d0 0x7f4d64150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:54.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.282+0000 7f4d63400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4d640cc380 0x7f4d64156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.282+0000 7f4d63e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4d640c7820 0x7f4d640c7c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:39708/0 (socket says 172.21.15.102:39708) 2024-03-18T16:34:54.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.282+0000 7f4d63e00640 1 -- 172.21.15.102:0/504927687 learned_addr learned my addr 172.21.15.102:0/504927687 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.285+0000 7f4d69200640 1 -- 172.21.15.102:0/504927687 >> v2:172.21.15.102:3300/0 conn(0x7f4d640cc380 msgr2=0x7f4d64156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.285+0000 7f4d69200640 1 --2- 172.21.15.102:0/504927687 >> v2:172.21.15.102:3300/0 conn(0x7f4d640cc380 0x7f4d64156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.285+0000 7f4d69200640 1 -- 172.21.15.102:0/504927687 >> v1:172.21.15.154:6789/0 conn(0x7f4d640c7820 legacy=0x7f4d640c7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.285+0000 7f4d69200640 1 -- 172.21.15.102:0/504927687 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d640c32c0 con 0x7f4d641504d0 2024-03-18T16:34:54.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.285+0000 7f4d62a00640 1 -- 172.21.15.102:0/504927687 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1319135355 0 0) 0x7f4d6406ffb0 con 0x7f4d640c7820 2024-03-18T16:34:54.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.287+0000 7f4d69200640 1 --2- 172.21.15.102:0/504927687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d641504d0 0x7f4d64150960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009d90 tx=0x7f4d5c02cfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=282a4b1140a7264 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.291+0000 7f4d62a00640 1 -- 172.21.15.102:0/504927687 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c03a070 con 0x7f4d641504d0 2024-03-18T16:34:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.291+0000 7f4d62a00640 1 -- 172.21.15.102:0/504927687 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d5c02dd70 con 0x7f4d641504d0 2024-03-18T16:34:54.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.291+0000 7f4d62a00640 1 -- 172.21.15.102:0/504927687 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c035660 con 0x7f4d641504d0 2024-03-18T16:34:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.292+0000 7f4d69c00640 1 -- 172.21.15.102:0/504927687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d641504d0 msgr2=0x7f4d64150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.292+0000 7f4d69c00640 1 --2- 172.21.15.102:0/504927687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d641504d0 0x7f4d64150960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009d90 tx=0x7f4d5c02cfc0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.292+0000 7f4d69c00640 1 -- 172.21.15.102:0/504927687 shutdown_connections 2024-03-18T16:34:54.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.292+0000 7f4d69c00640 1 --2- 172.21.15.102:0/504927687 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d641504d0 0x7f4d64150960 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.292+0000 7f4d69c00640 1 --2- 172.21.15.102:0/504927687 >> v2:172.21.15.102:3300/0 conn(0x7f4d640cc380 0x7f4d64156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.292+0000 7f4d69c00640 1 -- 172.21.15.102:0/504927687 >> 172.21.15.102:0/504927687 conn(0x7f4d640cb890 msgr2=0x7f4d640cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.293+0000 7f4d69c00640 1 -- 172.21.15.102:0/504927687 shutdown_connections 2024-03-18T16:34:54.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.293+0000 7f4d69c00640 1 -- 172.21.15.102:0/504927687 wait complete. 2024-03-18T16:34:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.295+0000 7f4d69c00640 1 Processor -- start 2024-03-18T16:34:54.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.295+0000 7f4d69c00640 1 -- start start 2024-03-18T16:34:54.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.297+0000 7f4d69c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d640c7820 0x7f4d641e0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.297+0000 7f4d69c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d640cc380 0x7f4d641ecc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.297+0000 7f4d69c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d641504d0 0x7f4d6406f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.297+0000 7f4d69c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6406fca0 con 0x7f4d640cc380 2024-03-18T16:34:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.297+0000 7f4d69c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4d640c4c90 con 0x7f4d641504d0 2024-03-18T16:34:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.297+0000 7f4d69c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6406f960 con 0x7f4d640c7820 2024-03-18T16:34:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.298+0000 7f4d63e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d640c7820 0x7f4d641e0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:54.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.298+0000 7f4d63400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d640cc380 0x7f4d641ecc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.298+0000 7f4d69200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d641504d0 0x7f4d6406f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.298+0000 7f4d63e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d640c7820 0x7f4d641e0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:33136/0 (socket says 172.21.15.102:33136) 2024-03-18T16:34:54.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.298+0000 7f4d63e00640 1 -- 172.21.15.102:0/3861086926 learned_addr learned my addr 172.21.15.102:0/3861086926 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.302+0000 7f4d63e00640 1 -- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d641504d0 msgr2=0x7f4d6406f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.302+0000 7f4d63e00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d641504d0 0x7f4d6406f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.302+0000 7f4d63e00640 1 -- 172.21.15.102:0/3861086926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d640cc380 msgr2=0x7f4d641ecc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.302+0000 7f4d63e00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d640cc380 0x7f4d641ecc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.302+0000 7f4d63e00640 1 -- 172.21.15.102:0/3861086926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d641ec720 con 0x7f4d640c7820 2024-03-18T16:34:54.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.304+0000 7f4d63e00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d640c7820 0x7f4d641e0cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400c490 tx=0x7f4d5400c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.309+0000 7f4d5be00640 1 -- 172.21.15.102:0/3861086926 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d540074e0 con 0x7f4d640c7820 2024-03-18T16:34:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.309+0000 7f4d5be00640 1 -- 172.21.15.102:0/3861086926 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d540047e0 con 0x7f4d640c7820 2024-03-18T16:34:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.309+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d641ec160 con 0x7f4d640c7820 2024-03-18T16:34:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.309+0000 7f4d5be00640 1 -- 172.21.15.102:0/3861086926 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d54004bd0 con 0x7f4d640c7820 2024-03-18T16:34:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.309+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d641e01d0 con 0x7f4d640c7820 2024-03-18T16:34:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.310+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d64074fd0 con 0x7f4d640c7820 2024-03-18T16:34:54.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.330+0000 7f4d5be00640 1 -- 172.21.15.102:0/3861086926 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4d54011070 con 0x7f4d640c7820 2024-03-18T16:34:54.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.331+0000 7f4d5be00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d4406bbd0 0x7f4d4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:54.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.332+0000 7f4d63400640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d4406bbd0 0x7f4d4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:54.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.333+0000 7f4d63400640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d4406bbd0 0x7f4d4406e090 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4d64157bd0 tx=0x7f4d50009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:54.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.572+0000 7f4d5be00640 1 -- 172.21.15.102:0/3861086926 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f4d54057130 con 0x7f4d640c7820 2024-03-18T16:34:54.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.592+0000 7f4d5be00640 1 -- 172.21.15.102:0/3861086926 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4d54059090 con 0x7f4d640c7820 2024-03-18T16:34:54.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.831+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d640c5b60 con 0x7f4d640c7820 2024-03-18T16:34:54.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.846+0000 7f4d5be00640 1 -- 172.21.15.102:0/3861086926 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f4d5405d130 con 0x7f4d640c7820 2024-03-18T16:34:54.847 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:54.847 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:54.848 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d4406bbd0 msgr2=0x7f4d4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d4406bbd0 0x7f4d4406e090 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4d64157bd0 tx=0x7f4d50009190 comp rx=0 tx=0).stop 2024-03-18T16:34:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d640c7820 msgr2=0x7f4d641e0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d640c7820 0x7f4d641e0cb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400c490 tx=0x7f4d5400c950 comp rx=0 tx=0).stop 2024-03-18T16:34:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 shutdown_connections 2024-03-18T16:34:54.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4d4406bbd0 0x7f4d4406e090 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d641504d0 0x7f4d6406f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d640cc380 0x7f4d641ecc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 --2- 172.21.15.102:0/3861086926 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4d640c7820 0x7f4d641e0cb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.848+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 >> 172.21.15.102:0/3861086926 conn(0x7f4d640cb890 msgr2=0x7f4d640c6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.849+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 shutdown_connections 2024-03-18T16:34:54.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:54.849+0000 7f4d69c00640 1 -- 172.21.15.102:0/3861086926 wait complete. 2024-03-18T16:34:55.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:55.473 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:56.474 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:56.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.647+0000 7ff88be00640 1 Processor -- start 2024-03-18T16:34:56.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.648+0000 7ff88be00640 1 -- start start 2024-03-18T16:34:56.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.649+0000 7ff88be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff88c161d30 0x7ff88c16a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:56.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.649+0000 7ff88be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff88c0c4190 con 0x7ff88c158c00 2024-03-18T16:34:56.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.649+0000 7ff88be00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff88c06f060 con 0x7ff88c071700 2024-03-18T16:34:56.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.649+0000 7ff88be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff88c158fe0 con 0x7ff88c161d30 2024-03-18T16:34:56.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.650+0000 7ff88a000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff88c071700 0x7ff88c072a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:39712/0 (socket says 172.21.15.102:39712) 2024-03-18T16:34:56.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.650+0000 7ff88a000640 1 -- 172.21.15.102:0/3723907672 learned_addr learned my addr 172.21.15.102:0/3723907672 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:56.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.650+0000 7ff88b400640 1 --2- 172.21.15.102:0/3723907672 >> v2:172.21.15.102:3300/0 conn(0x7ff88c161d30 0x7ff88c16a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:56.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.652+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4226198547 0 0) 0x7ff88c06f060 con 0x7ff88c071700 2024-03-18T16:34:56.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.652+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff870003520 con 0x7ff88c071700 2024-03-18T16:34:56.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.655+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1302424169 0 0) 0x7ff88c0c4190 con 0x7ff88c158c00 2024-03-18T16:34:56.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.655+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff88c06f060 con 0x7ff88c158c00 2024-03-18T16:34:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1151504111 0 0) 0x7ff870003520 con 0x7ff88c071700 2024-03-18T16:34:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 >> v2:172.21.15.102:3300/0 conn(0x7ff88c161d30 msgr2=0x7ff88c16a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 --2- 172.21.15.102:0/3723907672 >> v2:172.21.15.102:3300/0 conn(0x7ff88c161d30 0x7ff88c16a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 >> v1:172.21.15.102:6789/0 conn(0x7ff88c158c00 legacy=0x7ff88c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c071cb0 con 0x7ff88c071700 2024-03-18T16:34:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7ff878001670 con 0x7ff88c071700 2024-03-18T16:34:56.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 >> v1:172.21.15.154:6789/0 conn(0x7ff88c071700 legacy=0x7ff88c072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff870003b40 0x7ff870003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff870004800 0x7ff870004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8700051c0 0x7ff87000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c158fe0 con 0x7ff8700051c0 2024-03-18T16:34:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff87000c040 con 0x7ff870004800 2024-03-18T16:34:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.656+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff87000c240 con 0x7ff870003b40 2024-03-18T16:34:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.657+0000 7ff88aa00640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff870003b40 0x7ff870003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:56.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.658+0000 7ff88a000640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff870004800 0x7ff870004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:56.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.661+0000 7ff88aa00640 1 -- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff870004800 msgr2=0x7ff870004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.661+0000 7ff88aa00640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff870004800 0x7ff870004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.661+0000 7ff88aa00640 1 -- 172.21.15.102:0/3723907672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8700051c0 msgr2=0x7ff87000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.661+0000 7ff88aa00640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8700051c0 0x7ff87000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.661+0000 7ff88aa00640 1 -- 172.21.15.102:0/3723907672 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c071cb0 con 0x7ff870003b40 2024-03-18T16:34:56.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.661+0000 7ff88b400640 1 -- 172.21.15.102:0/3723907672 reap_dead start 2024-03-18T16:34:56.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.664+0000 7ff88aa00640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff870003b40 0x7ff870003f40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0018d0 tx=0x7ff870005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=762fbf4818b0e449 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:56.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.668+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c037070 con 0x7ff870003b40 2024-03-18T16:34:56.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.669+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff87c0052d0 con 0x7ff870003b40 2024-03-18T16:34:56.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.669+0000 7ff889600640 1 -- 172.21.15.102:0/3723907672 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c0014f0 con 0x7ff870003b40 2024-03-18T16:34:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.669+0000 7ff88be00640 1 -- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff870003b40 msgr2=0x7ff870003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.669+0000 7ff88be00640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff870003b40 0x7ff870003f40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0018d0 tx=0x7ff870005830 comp rx=0 tx=0).stop 2024-03-18T16:34:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.670+0000 7ff88be00640 1 -- 172.21.15.102:0/3723907672 shutdown_connections 2024-03-18T16:34:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.670+0000 7ff88be00640 1 --2- 172.21.15.102:0/3723907672 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff870003b40 0x7ff870003f40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.670+0000 7ff88be00640 1 -- 172.21.15.102:0/3723907672 >> 172.21.15.102:0/3723907672 conn(0x7ff88c0cb890 msgr2=0x7ff88c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:56.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.670+0000 7ff88be00640 1 -- 172.21.15.102:0/3723907672 shutdown_connections 2024-03-18T16:34:56.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.670+0000 7ff88be00640 1 -- 172.21.15.102:0/3723907672 wait complete. 2024-03-18T16:34:56.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.672+0000 7ff88be00640 1 Processor -- start 2024-03-18T16:34:56.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.673+0000 7ff88be00640 1 -- start start 2024-03-18T16:34:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.674+0000 7ff88be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff870003b40 0x7ff88c14a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.674+0000 7ff88be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0c6b40 0x7ff88c1f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.674+0000 7ff88be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c153790 0x7ff88c1f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.674+0000 7ff88be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c072270 con 0x7ff870003b40 2024-03-18T16:34:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.674+0000 7ff88be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c06f3b0 con 0x7ff88c153790 2024-03-18T16:34:56.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.674+0000 7ff88be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff88c072550 con 0x7ff88c0c6b40 2024-03-18T16:34:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.675+0000 7ff88a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0c6b40 0x7ff88c1f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.675+0000 7ff88b400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c153790 0x7ff88c1f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.675+0000 7ff88aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff870003b40 0x7ff88c14a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.675+0000 7ff88a000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0c6b40 0x7ff88c1f47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:33158/0 (socket says 172.21.15.102:33158) 2024-03-18T16:34:56.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.675+0000 7ff88a000640 1 -- 172.21.15.102:0/652179686 learned_addr learned my addr 172.21.15.102:0/652179686 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:56.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.679+0000 7ff88b400640 1 -- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0c6b40 msgr2=0x7ff88c1f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:56.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.679+0000 7ff88b400640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0c6b40 0x7ff88c1f47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:56.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.679+0000 7ff88b400640 1 -- 172.21.15.102:0/652179686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff870003b40 msgr2=0x7ff88c14a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:56.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.679+0000 7ff88b400640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff870003b40 0x7ff88c14a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:56.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.679+0000 7ff88b400640 1 -- 172.21.15.102:0/652179686 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c145860 con 0x7ff88c153790 2024-03-18T16:34:56.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.682+0000 7ff88b400640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c153790 0x7ff88c1f4ce0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0c2130 tx=0x7ff88400ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.687+0000 7ff883400640 1 -- 172.21.15.102:0/652179686 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88400bb10 con 0x7ff88c153790 2024-03-18T16:34:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.687+0000 7ff883400640 1 -- 172.21.15.102:0/652179686 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88401a070 con 0x7ff88c153790 2024-03-18T16:34:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.687+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88c1f42a0 con 0x7ff88c153790 2024-03-18T16:34:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.687+0000 7ff883400640 1 -- 172.21.15.102:0/652179686 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884014400 con 0x7ff88c153790 2024-03-18T16:34:56.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.687+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff88c1f7c90 con 0x7ff88c153790 2024-03-18T16:34:56.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.689+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88c074fd0 con 0x7ff88c153790 2024-03-18T16:34:56.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.708+0000 7ff883400640 1 -- 172.21.15.102:0/652179686 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff884013070 con 0x7ff88c153790 2024-03-18T16:34:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.709+0000 7ff883400640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff86c06bc20 0x7ff86c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:56.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.710+0000 7ff88aa00640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff86c06bc20 0x7ff86c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:56.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.711+0000 7ff883400640 1 -- 172.21.15.102:0/652179686 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff884057b10 con 0x7ff88c153790 2024-03-18T16:34:56.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.711+0000 7ff88aa00640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff86c06bc20 0x7ff86c06e0e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff87c034000 tx=0x7ff87c031000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:56.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:56.963+0000 7ff883400640 1 -- 172.21.15.102:0/652179686 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff884059a70 con 0x7ff88c153790 2024-03-18T16:34:57.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.341+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff88c0c4190 con 0x7ff88c153790 2024-03-18T16:34:57.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.357+0000 7ff883400640 1 -- 172.21.15.102:0/652179686 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff88405db10 con 0x7ff88c153790 2024-03-18T16:34:57.359 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:57.359 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:57.359 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:57.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff86c06bc20 msgr2=0x7ff86c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff86c06bc20 0x7ff86c06e0e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff87c034000 tx=0x7ff87c031000 comp rx=0 tx=0).stop 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c153790 msgr2=0x7ff88c1f4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c153790 0x7ff88c1f4ce0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0c2130 tx=0x7ff88400ef30 comp rx=0 tx=0).stop 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 shutdown_connections 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff86c06bc20 0x7ff86c06e0e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff88c153790 0x7ff88c1f4ce0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff88c0c6b40 0x7ff88c1f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 --2- 172.21.15.102:0/652179686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff870003b40 0x7ff88c14a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 >> 172.21.15.102:0/652179686 conn(0x7ff88c0cb890 msgr2=0x7ff88c1568d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 shutdown_connections 2024-03-18T16:34:57.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:57.360+0000 7ff88be00640 1 -- 172.21.15.102:0/652179686 wait complete. 2024-03-18T16:34:57.982 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:34:57.982 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:34:58.983 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:34:59.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.172+0000 7f0791600640 1 Processor -- start 2024-03-18T16:34:59.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.172+0000 7f0791600640 1 -- start start 2024-03-18T16:34:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.173+0000 7f0791600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f078c0cfe70 0x7f078c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.174+0000 7f0791600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078c0cabd0 con 0x7f078c0cc2f0 2024-03-18T16:34:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.174+0000 7f0791600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078c0caea0 con 0x7f078c0d0250 2024-03-18T16:34:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.174+0000 7f0791600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f078c0cb180 con 0x7f078c0cfe70 2024-03-18T16:34:59.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.174+0000 7f078be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f078c0cfe70 0x7f078c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:59.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.175+0000 7f078b400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f078c0d0250 0x7f078c0c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:39720/0 (socket says 172.21.15.102:39720) 2024-03-18T16:34:59.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.175+0000 7f078b400640 1 -- 172.21.15.102:0/2913630121 learned_addr learned my addr 172.21.15.102:0/2913630121 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:34:59.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.177+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2282717791 0 0) 0x7f078c0caea0 con 0x7f078c0d0250 2024-03-18T16:34:59.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.177+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0770003520 con 0x7f078c0d0250 2024-03-18T16:34:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2048214415 0 0) 0x7f0770003520 con 0x7f078c0d0250 2024-03-18T16:34:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 >> v2:172.21.15.102:3300/0 conn(0x7f078c0cfe70 msgr2=0x7f078c15ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 --2- 172.21.15.102:0/2913630121 >> v2:172.21.15.102:3300/0 conn(0x7f078c0cfe70 0x7f078c15ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 >> v1:172.21.15.102:6789/0 conn(0x7f078c0cc2f0 legacy=0x7f078c0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:59.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c0cb460 con 0x7f078c0d0250 2024-03-18T16:34:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f0778001670 con 0x7f078c0d0250 2024-03-18T16:34:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 >> v1:172.21.15.154:6789/0 conn(0x7f078c0d0250 legacy=0x7f078c0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 --2- 172.21.15.102:0/2913630121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07700039b0 0x7f0770003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 --2- 172.21.15.102:0/2913630121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0770004800 0x7f0770004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 --2- 172.21.15.102:0/2913630121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07700051c0 0x7f077000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0cb180 con 0x7f0770004800 2024-03-18T16:34:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f077000c070 con 0x7f07700039b0 2024-03-18T16:34:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.181+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f077000c270 con 0x7f07700051c0 2024-03-18T16:34:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.182+0000 7f0790c00640 1 --2- 172.21.15.102:0/2913630121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07700051c0 0x7f077000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.185+0000 7f0790c00640 1 -- 172.21.15.102:0/2913630121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07700039b0 msgr2=0x7f0770003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.185+0000 7f0790c00640 1 --2- 172.21.15.102:0/2913630121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07700039b0 0x7f0770003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.185+0000 7f0790c00640 1 -- 172.21.15.102:0/2913630121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0770004800 msgr2=0x7f0770004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:34:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.185+0000 7f0790c00640 1 --2- 172.21.15.102:0/2913630121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0770004800 0x7f0770004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.185+0000 7f0790c00640 1 -- 172.21.15.102:0/2913630121 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0770003520 con 0x7f07700051c0 2024-03-18T16:34:59.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.187+0000 7f0790c00640 1 -- 172.21.15.102:0/2913630121 reap_dead start 2024-03-18T16:34:59.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.188+0000 7f0790c00640 1 --2- 172.21.15.102:0/2913630121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07700051c0 0x7f077000b9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0784001e50 tx=0x7f0770005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd0adfa8759b536e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:59.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.191+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784001a70 con 0x7f07700051c0 2024-03-18T16:34:59.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.192+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0784003bb0 con 0x7f07700051c0 2024-03-18T16:34:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.193+0000 7f078aa00640 1 -- 172.21.15.102:0/2913630121 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07840050d0 con 0x7f07700051c0 2024-03-18T16:34:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.193+0000 7f0791600640 1 -- 172.21.15.102:0/2913630121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07700051c0 msgr2=0x7f077000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.193+0000 7f0791600640 1 --2- 172.21.15.102:0/2913630121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07700051c0 0x7f077000b9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0784001e50 tx=0x7f0770005830 comp rx=0 tx=0).stop 2024-03-18T16:34:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.193+0000 7f0791600640 1 -- 172.21.15.102:0/2913630121 shutdown_connections 2024-03-18T16:34:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.193+0000 7f0791600640 1 --2- 172.21.15.102:0/2913630121 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f07700051c0 0x7f077000b9c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.193+0000 7f0791600640 1 -- 172.21.15.102:0/2913630121 >> 172.21.15.102:0/2913630121 conn(0x7f078c08c3e0 msgr2=0x7f078c06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.193+0000 7f0791600640 1 -- 172.21.15.102:0/2913630121 shutdown_connections 2024-03-18T16:34:59.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.193+0000 7f0791600640 1 -- 172.21.15.102:0/2913630121 wait complete. 2024-03-18T16:34:59.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.196+0000 7f0791600640 1 Processor -- start 2024-03-18T16:34:59.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.196+0000 7f0791600640 1 -- start start 2024-03-18T16:34:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.198+0000 7f0791600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07700051c0 0x7f078c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.198+0000 7f0791600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f078c1500e0 0x7f078c158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.198+0000 7f0791600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078c1f04c0 0x7f078c14d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.198+0000 7f0791600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0cbd00 con 0x7f078c1500e0 2024-03-18T16:34:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.198+0000 7f0791600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0cba20 con 0x7f07700051c0 2024-03-18T16:34:59.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.198+0000 7f0791600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f078c15ae40 con 0x7f078c1f04c0 2024-03-18T16:34:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.198+0000 7f078be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07700051c0 0x7f078c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.198+0000 7f0790c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078c1f04c0 0x7f078c14d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.199+0000 7f078b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f078c1500e0 0x7f078c158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.199+0000 7f0790c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078c1f04c0 0x7f078c14d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:33172/0 (socket says 172.21.15.102:33172) 2024-03-18T16:34:59.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.199+0000 7f0790c00640 1 -- 172.21.15.102:0/2198379233 learned_addr learned my addr 172.21.15.102:0/2198379233 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:34:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.202+0000 7f0790c00640 1 -- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07700051c0 msgr2=0x7f078c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.202+0000 7f0790c00640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07700051c0 0x7f078c157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.202+0000 7f0790c00640 1 -- 172.21.15.102:0/2198379233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f078c1500e0 msgr2=0x7f078c158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.202+0000 7f0790c00640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f078c1500e0 0x7f078c158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.202+0000 7f0790c00640 1 -- 172.21.15.102:0/2198379233 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c14f060 con 0x7f078c1f04c0 2024-03-18T16:34:59.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.205+0000 7f0790c00640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078c1f04c0 0x7f078c14d710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f078402c060 tx=0x7f0784001510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:59.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.209+0000 7f0783e00640 1 -- 172.21.15.102:0/2198379233 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784006d50 con 0x7f078c1f04c0 2024-03-18T16:34:59.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.209+0000 7f0783e00640 1 -- 172.21.15.102:0/2198379233 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078402f670 con 0x7f078c1f04c0 2024-03-18T16:34:59.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.209+0000 7f0783e00640 1 -- 172.21.15.102:0/2198379233 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078402fa20 con 0x7f078c1f04c0 2024-03-18T16:34:59.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.209+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078c14f300 con 0x7f078c1f04c0 2024-03-18T16:34:59.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.209+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f078c14f8c0 con 0x7f078c1f04c0 2024-03-18T16:34:59.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.210+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078c074fd0 con 0x7f078c1f04c0 2024-03-18T16:34:59.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.230+0000 7f0783e00640 1 -- 172.21.15.102:0/2198379233 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f078402fbc0 con 0x7f078c1f04c0 2024-03-18T16:34:59.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.232+0000 7f0783e00640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f076c06bc50 0x7f076c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:34:59.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.232+0000 7f078be00640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f076c06bc50 0x7f076c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:34:59.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.233+0000 7f078be00640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f076c06bc50 0x7f076c06e110 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f077c0093d0 tx=0x7f077c0094a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:34:59.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.233+0000 7f0783e00640 1 -- 172.21.15.102:0/2198379233 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f0784078ab0 con 0x7f078c1f04c0 2024-03-18T16:34:59.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.481+0000 7f0783e00640 1 -- 172.21.15.102:0/2198379233 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0784033030 con 0x7f078c1f04c0 2024-03-18T16:34:59.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.759+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f078c0c7240 con 0x7f078c1f04c0 2024-03-18T16:34:59.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.772+0000 7f0783e00640 1 -- 172.21.15.102:0/2198379233 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f078407aa10 con 0x7f078c1f04c0 2024-03-18T16:34:59.774 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:34:59.774 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:34:59.774 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:34:59.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f076c06bc50 msgr2=0x7f076c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:59.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f076c06bc50 0x7f076c06e110 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f077c0093d0 tx=0x7f077c0094a0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078c1f04c0 msgr2=0x7f078c14d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:34:59.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078c1f04c0 0x7f078c14d710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f078402c060 tx=0x7f0784001510 comp rx=0 tx=0).stop 2024-03-18T16:34:59.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 shutdown_connections 2024-03-18T16:34:59.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f076c06bc50 0x7f076c06e110 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f078c1f04c0 0x7f078c14d710 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f078c1500e0 0x7f078c158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 --2- 172.21.15.102:0/2198379233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07700051c0 0x7f078c157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:34:59.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.775+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 >> 172.21.15.102:0/2198379233 conn(0x7f078c08c3e0 msgr2=0x7f078c15a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:34:59.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.776+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 shutdown_connections 2024-03-18T16:34:59.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:34:59.776+0000 7f0791600640 1 -- 172.21.15.102:0/2198379233 wait complete. 2024-03-18T16:35:00.397 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:00.397 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:01.398 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:01.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.593+0000 7f146cc00640 1 Processor -- start 2024-03-18T16:35:01.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.593+0000 7f146cc00640 1 -- start start 2024-03-18T16:35:01.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.595+0000 7f146cc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1468161d30 0x7f146816a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:01.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.595+0000 7f146cc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14680c4190 con 0x7f1468158c00 2024-03-18T16:35:01.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.595+0000 7f146cc00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f146806f060 con 0x7f1468071700 2024-03-18T16:35:01.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.595+0000 7f146cc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1468158fe0 con 0x7f1468161d30 2024-03-18T16:35:01.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.595+0000 7f1466a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f1468071700 0x7f1468072a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:40620/0 (socket says 172.21.15.102:40620) 2024-03-18T16:35:01.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.595+0000 7f1466a00640 1 -- 172.21.15.102:0/3320136354 learned_addr learned my addr 172.21.15.102:0/3320136354 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:01.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.596+0000 7f1467e00640 1 --2- 172.21.15.102:0/3320136354 >> v2:172.21.15.102:3300/0 conn(0x7f1468161d30 0x7f146816a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:01.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.598+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968096697 0 0) 0x7f146806f060 con 0x7f1468071700 2024-03-18T16:35:01.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.598+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f144c003520 con 0x7f1468071700 2024-03-18T16:35:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.601+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2936281339 0 0) 0x7f144c003520 con 0x7f1468071700 2024-03-18T16:35:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.601+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 >> v2:172.21.15.102:3300/0 conn(0x7f1468161d30 msgr2=0x7f146816a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.601+0000 7f1466000640 1 --2- 172.21.15.102:0/3320136354 >> v2:172.21.15.102:3300/0 conn(0x7f1468161d30 0x7f146816a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:01.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.601+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 >> v1:172.21.15.102:6789/0 conn(0x7f1468158c00 legacy=0x7f146806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.601+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1468071cb0 con 0x7f1468071700 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.601+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7f1454001670 con 0x7f1468071700 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.601+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 >> v1:172.21.15.154:6789/0 conn(0x7f1468071700 legacy=0x7f1468072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.602+0000 7f1466000640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0039b0 0x7f144c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.602+0000 7f1466000640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f144c004800 0x7f144c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.602+0000 7f1466000640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f144c0051c0 0x7f144c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.602+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1468158fe0 con 0x7f144c004800 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.602+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f144c00c060 con 0x7f144c0039b0 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.602+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f144c00c260 con 0x7f144c0051c0 2024-03-18T16:35:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.603+0000 7f1467e00640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f144c0051c0 0x7f144c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.605+0000 7f1467400640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0039b0 0x7f144c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.606+0000 7f1467e00640 1 -- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0039b0 msgr2=0x7f144c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.606+0000 7f1467e00640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0039b0 0x7f144c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.606+0000 7f1467e00640 1 -- 172.21.15.102:0/3320136354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f144c004800 msgr2=0x7f144c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.606+0000 7f1467e00640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f144c004800 0x7f144c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.606+0000 7f1467e00640 1 -- 172.21.15.102:0/3320136354 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1468071cb0 con 0x7f144c0051c0 2024-03-18T16:35:01.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.607+0000 7f1467e00640 1 -- 172.21.15.102:0/3320136354 reap_dead start 2024-03-18T16:35:01.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.611+0000 7f1467e00640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f144c0051c0 0x7f144c00b9c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1460009fc0 tx=0x7f144c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abdaba411a8bd742 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:01.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.615+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460009c10 con 0x7f144c0051c0 2024-03-18T16:35:01.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.615+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f146002cc60 con 0x7f144c0051c0 2024-03-18T16:35:01.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.615+0000 7f1466000640 1 -- 172.21.15.102:0/3320136354 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460003bd0 con 0x7f144c0051c0 2024-03-18T16:35:01.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.616+0000 7f146cc00640 1 -- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f144c0051c0 msgr2=0x7f144c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:01.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.616+0000 7f146cc00640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f144c0051c0 0x7f144c00b9c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1460009fc0 tx=0x7f144c005830 comp rx=0 tx=0).stop 2024-03-18T16:35:01.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.616+0000 7f146cc00640 1 -- 172.21.15.102:0/3320136354 shutdown_connections 2024-03-18T16:35:01.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.616+0000 7f146cc00640 1 --2- 172.21.15.102:0/3320136354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f144c0051c0 0x7f144c00b9c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:01.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.616+0000 7f146cc00640 1 -- 172.21.15.102:0/3320136354 >> 172.21.15.102:0/3320136354 conn(0x7f14680cb890 msgr2=0x7f14680cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:01.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.617+0000 7f146cc00640 1 -- 172.21.15.102:0/3320136354 shutdown_connections 2024-03-18T16:35:01.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.617+0000 7f146cc00640 1 -- 172.21.15.102:0/3320136354 wait complete. 2024-03-18T16:35:01.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.619+0000 7f146cc00640 1 Processor -- start 2024-03-18T16:35:01.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.620+0000 7f146cc00640 1 -- start start 2024-03-18T16:35:01.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.621+0000 7f146cc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0051c0 0x7f14681473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:01.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.621+0000 7f146cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14681ea790 0x7f1468147900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:01.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.621+0000 7f146cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14680c98b0 0x7f14680c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:01.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.621+0000 7f146cc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1468072270 con 0x7f14681ea790 2024-03-18T16:35:01.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.621+0000 7f146cc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f146806f3b0 con 0x7f144c0051c0 2024-03-18T16:35:01.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.621+0000 7f146cc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1468072550 con 0x7f14680c98b0 2024-03-18T16:35:01.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.622+0000 7f1467e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14680c98b0 0x7f14680c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:01.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.622+0000 7f1467400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0051c0 0x7f14681473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:01.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.622+0000 7f1466a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14681ea790 0x7f1468147900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:01.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.622+0000 7f1467400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0051c0 0x7f14681473c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:55630/0 (socket says 172.21.15.102:55630) 2024-03-18T16:35:01.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.622+0000 7f1467400640 1 -- 172.21.15.102:0/3019206847 learned_addr learned my addr 172.21.15.102:0/3019206847 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:01.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.625+0000 7f1467400640 1 -- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14680c98b0 msgr2=0x7f14680c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:01.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.625+0000 7f1467400640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14680c98b0 0x7f14680c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:01.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.625+0000 7f1467400640 1 -- 172.21.15.102:0/3019206847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14681ea790 msgr2=0x7f1468147900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:01.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.625+0000 7f1467400640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14681ea790 0x7f1468147900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:01.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.625+0000 7f1467400640 1 -- 172.21.15.102:0/3019206847 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14680ca270 con 0x7f144c0051c0 2024-03-18T16:35:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.628+0000 7f1467400640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0051c0 0x7f14681473c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f145800e4d0 tx=0x7f145800e990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.634+0000 7f145f400640 1 -- 172.21.15.102:0/3019206847 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14580052e0 con 0x7f144c0051c0 2024-03-18T16:35:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.634+0000 7f145f400640 1 -- 172.21.15.102:0/3019206847 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1458018070 con 0x7f144c0051c0 2024-03-18T16:35:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.634+0000 7f145f400640 1 -- 172.21.15.102:0/3019206847 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458003810 con 0x7f144c0051c0 2024-03-18T16:35:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.635+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14681f4520 con 0x7f144c0051c0 2024-03-18T16:35:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.635+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14681f4aa0 con 0x7f144c0051c0 2024-03-18T16:35:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.636+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14680c5ce0 con 0x7f144c0051c0 2024-03-18T16:35:01.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.655+0000 7f145f400640 1 -- 172.21.15.102:0/3019206847 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f14580039b0 con 0x7f144c0051c0 2024-03-18T16:35:01.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.656+0000 7f145f400640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f144806bc20 0x7f144806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:01.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.657+0000 7f1466a00640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f144806bc20 0x7f144806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:01.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.657+0000 7f1466a00640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f144806bc20 0x7f144806e0e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f14680c5b20 tx=0x7f14540019f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:01.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.897+0000 7f145f400640 1 -- 172.21.15.102:0/3019206847 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f14580575f0 con 0x7f144c0051c0 2024-03-18T16:35:01.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:01.908+0000 7f145f400640 1 -- 172.21.15.102:0/3019206847 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f14680c5ce0 con 0x7f144c0051c0 2024-03-18T16:35:02.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.170+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1468074fd0 con 0x7f144c0051c0 2024-03-18T16:35:02.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.250+0000 7f145f400640 1 -- 172.21.15.102:0/3019206847 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f14680ca270 con 0x7f144c0051c0 2024-03-18T16:35:02.252 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:02.252 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:02.252 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f144806bc20 msgr2=0x7f144806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f144806bc20 0x7f144806e0e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f14680c5b20 tx=0x7f14540019f0 comp rx=0 tx=0).stop 2024-03-18T16:35:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0051c0 msgr2=0x7f14681473c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0051c0 0x7f14681473c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f145800e4d0 tx=0x7f145800e990 comp rx=0 tx=0).stop 2024-03-18T16:35:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 shutdown_connections 2024-03-18T16:35:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f144806bc20 0x7f144806e0e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14680c98b0 0x7f14680c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14681ea790 0x7f1468147900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:02.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 --2- 172.21.15.102:0/3019206847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144c0051c0 0x7f14681473c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.253+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 >> 172.21.15.102:0/3019206847 conn(0x7f14680cb890 msgr2=0x7f146806f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.254+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 shutdown_connections 2024-03-18T16:35:02.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:02.254+0000 7f146cc00640 1 -- 172.21.15.102:0/3019206847 wait complete. 2024-03-18T16:35:02.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:02.877 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:03.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.049+0000 7f18bf200640 1 Processor -- start 2024-03-18T16:35:04.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.049+0000 7f18bf200640 1 -- start start 2024-03-18T16:35:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.051+0000 7f18bf200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 0x7f18b80ca260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.051+0000 7f18bf200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f18b806f8f0 0x7f18b8167d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.052+0000 7f18bf200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b80cb310 con 0x7f18b80c4150 2024-03-18T16:35:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.052+0000 7f18bf200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f18b80cab00 con 0x7f18b806f8f0 2024-03-18T16:35:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.052+0000 7f18bf200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f18b80c9dd0 con 0x7f18b8164b50 2024-03-18T16:35:04.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.052+0000 7f18bd400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f18b806f8f0 0x7f18b8167d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:04.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.052+0000 7f18bde00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 0x7f18b80ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:04.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.052+0000 7f18be800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f18b80c4150 0x7f18b816a870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47332/0 (socket says 172.21.15.102:47332) 2024-03-18T16:35:04.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.052+0000 7f18be800640 1 -- 172.21.15.102:0/2467263252 learned_addr learned my addr 172.21.15.102:0/2467263252 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:04.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.056+0000 7f18bd400640 1 -- 172.21.15.102:0/2467263252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 msgr2=0x7f18b80ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:04.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.056+0000 7f18bd400640 1 --2- 172.21.15.102:0/2467263252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 0x7f18b80ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.056+0000 7f18bd400640 1 -- 172.21.15.102:0/2467263252 >> v1:172.21.15.102:6789/0 conn(0x7f18b80c4150 legacy=0x7f18b816a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:04.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.056+0000 7f18bd400640 1 -- 172.21.15.102:0/2467263252 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b8168270 con 0x7f18b806f8f0 2024-03-18T16:35:04.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.058+0000 7f18bd400640 1 --2- 172.21.15.102:0/2467263252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b806f8f0 0x7f18b8167d30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f18a80097a0 tx=0x7f18a802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5edeba79e5cfbd2d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:04.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.060+0000 7f18bca00640 1 -- 172.21.15.102:0/2467263252 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a8035020 con 0x7f18b806f8f0 2024-03-18T16:35:04.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.062+0000 7f18bca00640 1 -- 172.21.15.102:0/2467263252 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a8031810 con 0x7f18b806f8f0 2024-03-18T16:35:04.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.062+0000 7f18bca00640 1 -- 172.21.15.102:0/2467263252 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a8031be0 con 0x7f18b806f8f0 2024-03-18T16:35:04.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.063+0000 7f18bf200640 1 -- 172.21.15.102:0/2467263252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b806f8f0 msgr2=0x7f18b8167d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:04.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.063+0000 7f18bf200640 1 --2- 172.21.15.102:0/2467263252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b806f8f0 0x7f18b8167d30 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f18a80097a0 tx=0x7f18a802b8b0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.063+0000 7f18bf200640 1 -- 172.21.15.102:0/2467263252 shutdown_connections 2024-03-18T16:35:04.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.063+0000 7f18bf200640 1 --2- 172.21.15.102:0/2467263252 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b806f8f0 0x7f18b8167d30 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.063+0000 7f18bf200640 1 --2- 172.21.15.102:0/2467263252 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 0x7f18b80ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.063+0000 7f18bf200640 1 -- 172.21.15.102:0/2467263252 >> 172.21.15.102:0/2467263252 conn(0x7f18b80cb890 msgr2=0x7f18b80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:04.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.063+0000 7f18bf200640 1 -- 172.21.15.102:0/2467263252 shutdown_connections 2024-03-18T16:35:04.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.063+0000 7f18bf200640 1 -- 172.21.15.102:0/2467263252 wait complete. 2024-03-18T16:35:04.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.065+0000 7f18bf200640 1 Processor -- start 2024-03-18T16:35:04.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.066+0000 7f18bf200640 1 -- start start 2024-03-18T16:35:04.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.067+0000 7f18bf200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b806f8f0 0x7f18b8201320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:04.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.067+0000 7f18bf200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b80c4150 0x7f18b808b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:04.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.067+0000 7f18bf200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 0x7f18b8176340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:04.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.067+0000 7f18bf200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f18b80cb030 con 0x7f18b806f8f0 2024-03-18T16:35:04.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.067+0000 7f18bf200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f18b80c97c0 con 0x7f18b80c4150 2024-03-18T16:35:04.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.067+0000 7f18bf200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f18b80c9a60 con 0x7f18b8164b50 2024-03-18T16:35:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.068+0000 7f18bd400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b80c4150 0x7f18b808b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.068+0000 7f18bde00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b806f8f0 0x7f18b8201320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.068+0000 7f18be800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 0x7f18b8176340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.068+0000 7f18bd400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b80c4150 0x7f18b808b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:55650/0 (socket says 172.21.15.102:55650) 2024-03-18T16:35:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.068+0000 7f18bd400640 1 -- 172.21.15.102:0/1909007634 learned_addr learned my addr 172.21.15.102:0/1909007634 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:04.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.072+0000 7f18be800640 1 -- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b80c4150 msgr2=0x7f18b808b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:04.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.072+0000 7f18be800640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b80c4150 0x7f18b808b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.072+0000 7f18be800640 1 -- 172.21.15.102:0/1909007634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b806f8f0 msgr2=0x7f18b8201320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:04.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.072+0000 7f18be800640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b806f8f0 0x7f18b8201320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.072+0000 7f18be800640 1 -- 172.21.15.102:0/1909007634 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b808b860 con 0x7f18b8164b50 2024-03-18T16:35:04.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.072+0000 7f18bd400640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b80c4150 0x7f18b808b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:35:04.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.074+0000 7f18be800640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 0x7f18b8176340 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f18b400caa0 tx=0x7f18b400cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:04.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.078+0000 7f18b2000640 1 -- 172.21.15.102:0/1909007634 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4017070 con 0x7f18b8164b50 2024-03-18T16:35:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.078+0000 7f18b2000640 1 -- 172.21.15.102:0/1909007634 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b400ac00 con 0x7f18b8164b50 2024-03-18T16:35:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.078+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b8167090 con 0x7f18b8164b50 2024-03-18T16:35:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.078+0000 7f18b2000640 1 -- 172.21.15.102:0/1909007634 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b4012400 con 0x7f18b8164b50 2024-03-18T16:35:04.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.078+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b815ab90 con 0x7f18b8164b50 2024-03-18T16:35:04.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.079+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b8074fd0 con 0x7f18b8164b50 2024-03-18T16:35:04.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.336+0000 7f18b2000640 1 -- 172.21.15.102:0/1909007634 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f18b401a070 con 0x7f18b8164b50 2024-03-18T16:35:04.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.339+0000 7f18b2000640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f189c06bbd0 0x7f189c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:04.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.339+0000 7f18bde00640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f189c06bbd0 0x7f189c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:04.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.340+0000 7f18bde00640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f189c06bbd0 0x7f189c06e090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009aa0 tx=0x7f18ac009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:04.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.341+0000 7f18b2000640 1 -- 172.21.15.102:0/1909007634 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f18b40570c0 con 0x7f18b8164b50 2024-03-18T16:35:04.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.352+0000 7f18b2000640 1 -- 172.21.15.102:0/1909007634 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f18b4059020 con 0x7f18b8164b50 2024-03-18T16:35:04.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.594+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18b80c3920 con 0x7f18b8164b50 2024-03-18T16:35:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.614+0000 7f18b2000640 1 -- 172.21.15.102:0/1909007634 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f18b405d0c0 con 0x7f18b8164b50 2024-03-18T16:35:04.616 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:04.616 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f189c06bbd0 msgr2=0x7f189c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f189c06bbd0 0x7f189c06e090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f18ac009aa0 tx=0x7f18ac009190 comp rx=0 tx=0).stop 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 msgr2=0x7f18b8176340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 0x7f18b8176340 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f18b400caa0 tx=0x7f18b400cf60 comp rx=0 tx=0).stop 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 shutdown_connections 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f189c06bbd0 0x7f189c06e090 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f18b8164b50 0x7f18b8176340 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18b80c4150 0x7f18b808b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 --2- 172.21.15.102:0/1909007634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18b806f8f0 0x7f18b8201320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 >> 172.21.15.102:0/1909007634 conn(0x7f18b80cb890 msgr2=0x7f18b8162a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 shutdown_connections 2024-03-18T16:35:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:04.617+0000 7f18bf200640 1 -- 172.21.15.102:0/1909007634 wait complete. 2024-03-18T16:35:05.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:05.240 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:06.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:06.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.411+0000 7ff4ce000640 1 Processor -- start 2024-03-18T16:35:06.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.411+0000 7ff4ce000640 1 -- start start 2024-03-18T16:35:06.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.412+0000 7ff4ce000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815a310 0x7ff4c806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:06.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.412+0000 7ff4ce000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff4c815baa0 0x7ff4c815bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:06.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.412+0000 7ff4ce000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c815c470 con 0x7ff4c815a6f0 2024-03-18T16:35:06.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.412+0000 7ff4ce000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c80cd410 con 0x7ff4c815baa0 2024-03-18T16:35:06.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.412+0000 7ff4ce000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c80ca500 con 0x7ff4c815a310 2024-03-18T16:35:06.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.413+0000 7ff4c7e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff4c815a6f0 0x7ff4c815ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:40630/0 (socket says 172.21.15.102:40630) 2024-03-18T16:35:06.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.413+0000 7ff4c7e00640 1 -- 172.21.15.102:0/1513378612 learned_addr learned my addr 172.21.15.102:0/1513378612 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:06.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.413+0000 7ff4ccc00640 1 --2- 172.21.15.102:0/1513378612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815a310 0x7ff4c806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:06.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.413+0000 7ff4cd600640 1 --2- 172.21.15.102:0/1513378612 >> v2:172.21.15.102:3300/0 conn(0x7ff4c815baa0 0x7ff4c815bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:06.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.416+0000 7ff4ccc00640 1 -- 172.21.15.102:0/1513378612 >> v2:172.21.15.102:3300/0 conn(0x7ff4c815baa0 msgr2=0x7ff4c815bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:06.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.416+0000 7ff4ccc00640 1 --2- 172.21.15.102:0/1513378612 >> v2:172.21.15.102:3300/0 conn(0x7ff4c815baa0 0x7ff4c815bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.416+0000 7ff4ccc00640 1 -- 172.21.15.102:0/1513378612 >> v1:172.21.15.154:6789/0 conn(0x7ff4c815a6f0 legacy=0x7ff4c815ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:06.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.416+0000 7ff4ccc00640 1 -- 172.21.15.102:0/1513378612 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c80ca7a0 con 0x7ff4c815a310 2024-03-18T16:35:06.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.418+0000 7ff4ccc00640 1 --2- 172.21.15.102:0/1513378612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815a310 0x7ff4c806d3e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0097a0 tx=0x7ff4bc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=518d5f936ac77563 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:06.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.421+0000 7ff4c7400640 1 -- 172.21.15.102:0/1513378612 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc035020 con 0x7ff4c815a310 2024-03-18T16:35:06.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.421+0000 7ff4c7400640 1 -- 172.21.15.102:0/1513378612 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4bc031810 con 0x7ff4c815a310 2024-03-18T16:35:06.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.421+0000 7ff4c7400640 1 -- 172.21.15.102:0/1513378612 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc031be0 con 0x7ff4c815a310 2024-03-18T16:35:06.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.422+0000 7ff4ce000640 1 -- 172.21.15.102:0/1513378612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815a310 msgr2=0x7ff4c806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:06.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.422+0000 7ff4ce000640 1 --2- 172.21.15.102:0/1513378612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815a310 0x7ff4c806d3e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0097a0 tx=0x7ff4bc02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.422+0000 7ff4ce000640 1 -- 172.21.15.102:0/1513378612 shutdown_connections 2024-03-18T16:35:06.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.422+0000 7ff4ce000640 1 --2- 172.21.15.102:0/1513378612 >> v2:172.21.15.102:3300/0 conn(0x7ff4c815baa0 0x7ff4c815bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.422+0000 7ff4ce000640 1 --2- 172.21.15.102:0/1513378612 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815a310 0x7ff4c806d3e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.422+0000 7ff4ce000640 1 -- 172.21.15.102:0/1513378612 >> 172.21.15.102:0/1513378612 conn(0x7ff4c808c1a0 msgr2=0x7ff4c8066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:06.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.422+0000 7ff4ce000640 1 -- 172.21.15.102:0/1513378612 shutdown_connections 2024-03-18T16:35:06.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.422+0000 7ff4ce000640 1 -- 172.21.15.102:0/1513378612 wait complete. 2024-03-18T16:35:06.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.424+0000 7ff4ce000640 1 Processor -- start 2024-03-18T16:35:06.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.424+0000 7ff4ce000640 1 -- start start 2024-03-18T16:35:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.425+0000 7ff4ce000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c815a310 0x7ff4c814c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:06.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.425+0000 7ff4ce000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c815a6f0 0x7ff4c80c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:06.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.425+0000 7ff4ce000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815baa0 0x7ff4c8150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:06.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.425+0000 7ff4ce000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c80cd130 con 0x7ff4c815a6f0 2024-03-18T16:35:06.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.425+0000 7ff4ce000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c80cce50 con 0x7ff4c815a310 2024-03-18T16:35:06.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.425+0000 7ff4ce000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c806d920 con 0x7ff4c815baa0 2024-03-18T16:35:06.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.426+0000 7ff4ccc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c815a310 0x7ff4c814c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:06.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.426+0000 7ff4c7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c815a6f0 0x7ff4c80c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.426+0000 7ff4cd600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815baa0 0x7ff4c8150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.426+0000 7ff4ccc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c815a310 0x7ff4c814c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:55656/0 (socket says 172.21.15.102:55656) 2024-03-18T16:35:06.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.426+0000 7ff4ccc00640 1 -- 172.21.15.102:0/165999854 learned_addr learned my addr 172.21.15.102:0/165999854 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.430+0000 7ff4cd600640 1 -- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c815a310 msgr2=0x7ff4c814c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.430+0000 7ff4cd600640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c815a310 0x7ff4c814c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.430+0000 7ff4cd600640 1 -- 172.21.15.102:0/165999854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c815a6f0 msgr2=0x7ff4c80c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.430+0000 7ff4cd600640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c815a6f0 0x7ff4c80c42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.430+0000 7ff4cd600640 1 -- 172.21.15.102:0/165999854 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c80c4820 con 0x7ff4c815baa0 2024-03-18T16:35:06.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.433+0000 7ff4cd600640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815baa0 0x7ff4c8150df0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000efc0 tx=0x7ff4c000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:06.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.436+0000 7ff4c4c00640 1 -- 172.21.15.102:0/165999854 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0018020 con 0x7ff4c815baa0 2024-03-18T16:35:06.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.437+0000 7ff4c4c00640 1 -- 172.21.15.102:0/165999854 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c0014480 con 0x7ff4c815baa0 2024-03-18T16:35:06.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.437+0000 7ff4c4c00640 1 -- 172.21.15.102:0/165999854 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0014870 con 0x7ff4c815baa0 2024-03-18T16:35:06.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.437+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c80c6fb0 con 0x7ff4c815baa0 2024-03-18T16:35:06.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.437+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c80c7570 con 0x7ff4c815baa0 2024-03-18T16:35:06.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.438+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c8074fd0 con 0x7ff4c815baa0 2024-03-18T16:35:06.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.457+0000 7ff4c4c00640 1 -- 172.21.15.102:0/165999854 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff4c0005d10 con 0x7ff4c815baa0 2024-03-18T16:35:06.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.458+0000 7ff4c4c00640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff4a806bc20 0x7ff4a806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:06.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.458+0000 7ff4ccc00640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff4a806bc20 0x7ff4a806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:06.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.459+0000 7ff4ccc00640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff4a806bc20 0x7ff4a806e0e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc02b8f0 tx=0x7ff4bc03c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:06.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.460+0000 7ff4c4c00640 1 -- 172.21.15.102:0/165999854 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff4c0058210 con 0x7ff4c815baa0 2024-03-18T16:35:06.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.711+0000 7ff4c4c00640 1 -- 172.21.15.102:0/165999854 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff4c0022030 con 0x7ff4c815baa0 2024-03-18T16:35:06.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.950+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4c80caed0 con 0x7ff4c815baa0 2024-03-18T16:35:06.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.966+0000 7ff4c4c00640 1 -- 172.21.15.102:0/165999854 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff4c0010040 con 0x7ff4c815baa0 2024-03-18T16:35:06.968 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:06.968 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:06.968 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:06.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff4a806bc20 msgr2=0x7ff4a806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff4a806bc20 0x7ff4a806e0e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc02b8f0 tx=0x7ff4bc03c040 comp rx=0 tx=0).stop 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815baa0 msgr2=0x7ff4c8150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815baa0 0x7ff4c8150df0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000efc0 tx=0x7ff4c000c490 comp rx=0 tx=0).stop 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 shutdown_connections 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff4a806bc20 0x7ff4a806e0e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff4c815baa0 0x7ff4c8150df0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4c815a6f0 0x7ff4c80c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 --2- 172.21.15.102:0/165999854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c815a310 0x7ff4c814c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 >> 172.21.15.102:0/165999854 conn(0x7ff4c808c1a0 msgr2=0x7ff4c814ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 shutdown_connections 2024-03-18T16:35:06.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:06.969+0000 7ff4ce000640 1 -- 172.21.15.102:0/165999854 wait complete. 2024-03-18T16:35:07.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:07.591 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:08.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:08.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.763+0000 7f04ae000640 1 Processor -- start 2024-03-18T16:35:08.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.763+0000 7f04ae000640 1 -- start start 2024-03-18T16:35:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.764+0000 7f04ae000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.764+0000 7f04ae000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f04a80cc2f0 0x7f04a80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.764+0000 7f04ae000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04a80ccca0 con 0x7f04a80cfe70 2024-03-18T16:35:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.764+0000 7f04ae000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f04a80cabd0 con 0x7f04a80cc2f0 2024-03-18T16:35:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.764+0000 7f04ae000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f04a80caea0 con 0x7f04a80d0250 2024-03-18T16:35:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.765+0000 7f04a7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.765+0000 7f04ad600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f04a80cc2f0 0x7f04a80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.766+0000 7f04a7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a815ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50692/0 (socket says 172.21.15.102:50692) 2024-03-18T16:35:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.766+0000 7f04acc00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f04a80cfe70 0x7f04a80c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47346/0 (socket says 172.21.15.102:47346) 2024-03-18T16:35:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.766+0000 7f04a7e00640 1 -- 172.21.15.102:0/467161590 learned_addr learned my addr 172.21.15.102:0/467161590 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:08.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.768+0000 7f04a7e00640 1 -- 172.21.15.102:0/467161590 >> v2:172.21.15.102:3300/0 conn(0x7f04a80cc2f0 msgr2=0x7f04a80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:08.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.768+0000 7f04a7e00640 1 --2- 172.21.15.102:0/467161590 >> v2:172.21.15.102:3300/0 conn(0x7f04a80cc2f0 0x7f04a80cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:08.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.768+0000 7f04a7e00640 1 -- 172.21.15.102:0/467161590 >> v1:172.21.15.102:6789/0 conn(0x7f04a80cfe70 legacy=0x7f04a80c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:08.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.768+0000 7f04a7e00640 1 -- 172.21.15.102:0/467161590 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a80cb180 con 0x7f04a80d0250 2024-03-18T16:35:08.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.771+0000 7f04a7e00640 1 --2- 172.21.15.102:0/467161590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a815ae40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0494009870 tx=0x7f049402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5d4727bc6025a24 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:08.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.773+0000 7f04a7400640 1 -- 172.21.15.102:0/467161590 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049402f830 con 0x7f04a80d0250 2024-03-18T16:35:08.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.774+0000 7f04a7400640 1 -- 172.21.15.102:0/467161590 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049402f9d0 con 0x7f04a80d0250 2024-03-18T16:35:08.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.774+0000 7f04a7400640 1 -- 172.21.15.102:0/467161590 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049402fda0 con 0x7f04a80d0250 2024-03-18T16:35:08.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.774+0000 7f04ae000640 1 -- 172.21.15.102:0/467161590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 msgr2=0x7f04a815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:08.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.775+0000 7f04ae000640 1 --2- 172.21.15.102:0/467161590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a815ae40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0494009870 tx=0x7f049402ba10 comp rx=0 tx=0).stop 2024-03-18T16:35:08.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.775+0000 7f04ae000640 1 -- 172.21.15.102:0/467161590 shutdown_connections 2024-03-18T16:35:08.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.775+0000 7f04ae000640 1 --2- 172.21.15.102:0/467161590 >> v2:172.21.15.102:3300/0 conn(0x7f04a80cc2f0 0x7f04a80cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:08.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.775+0000 7f04ae000640 1 --2- 172.21.15.102:0/467161590 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a815ae40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:08.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.775+0000 7f04ae000640 1 -- 172.21.15.102:0/467161590 >> 172.21.15.102:0/467161590 conn(0x7f04a808c3e0 msgr2=0x7f04a806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:08.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.775+0000 7f04ae000640 1 -- 172.21.15.102:0/467161590 shutdown_connections 2024-03-18T16:35:08.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.775+0000 7f04ae000640 1 -- 172.21.15.102:0/467161590 wait complete. 2024-03-18T16:35:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.777+0000 7f04ae000640 1 Processor -- start 2024-03-18T16:35:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.777+0000 7f04ae000640 1 -- start start 2024-03-18T16:35:08.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.778+0000 7f04ae000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a80cc2f0 0x7f04a8155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:08.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.778+0000 7f04ae000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04a80cfe70 0x7f04a81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:08.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.778+0000 7f04ae000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a81f1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:08.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.778+0000 7f04ae000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f04a80cba20 con 0x7f04a80cfe70 2024-03-18T16:35:08.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.778+0000 7f04ae000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04a80cb740 con 0x7f04a80cc2f0 2024-03-18T16:35:08.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.778+0000 7f04ae000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f04a80cbd00 con 0x7f04a80d0250 2024-03-18T16:35:08.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.779+0000 7f04ad600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a81f1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:08.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.779+0000 7f04acc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a80cc2f0 0x7f04a8155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:08.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.779+0000 7f04a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04a80cfe70 0x7f04a81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:08.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.779+0000 7f04acc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a80cc2f0 0x7f04a8155470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:55668/0 (socket says 172.21.15.102:55668) 2024-03-18T16:35:08.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.779+0000 7f04acc00640 1 -- 172.21.15.102:0/3140218198 learned_addr learned my addr 172.21.15.102:0/3140218198 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:08.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.784+0000 7f04ad600640 1 -- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a80cc2f0 msgr2=0x7f04a8155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:08.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.784+0000 7f04ad600640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a80cc2f0 0x7f04a8155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:08.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.784+0000 7f04ad600640 1 -- 172.21.15.102:0/3140218198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04a80cfe70 msgr2=0x7f04a81559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:08.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.784+0000 7f04ad600640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04a80cfe70 0x7f04a81559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:08.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.784+0000 7f04ad600640 1 -- 172.21.15.102:0/3140218198 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a81f0d40 con 0x7f04a80d0250 2024-03-18T16:35:08.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.787+0000 7f04ad600640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a81f1500 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f04a000efc0 tx=0x7f04a000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.791+0000 7f04a4c00640 1 -- 172.21.15.102:0/3140218198 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0018020 con 0x7f04a80d0250 2024-03-18T16:35:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.791+0000 7f04a4c00640 1 -- 172.21.15.102:0/3140218198 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a0014480 con 0x7f04a80d0250 2024-03-18T16:35:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.791+0000 7f04a4c00640 1 -- 172.21.15.102:0/3140218198 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0014870 con 0x7f04a80d0250 2024-03-18T16:35:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.792+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a81e0ec0 con 0x7f04a80d0250 2024-03-18T16:35:08.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.792+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a81f0fe0 con 0x7f04a80d0250 2024-03-18T16:35:08.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.793+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a81e11a0 con 0x7f04a80d0250 2024-03-18T16:35:08.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.813+0000 7f04a4c00640 1 -- 172.21.15.102:0/3140218198 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f04a0005d10 con 0x7f04a80d0250 2024-03-18T16:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.814+0000 7f04a4c00640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f048806bc20 0x7f048806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:08.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.814+0000 7f04acc00640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f048806bc20 0x7f048806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:08.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.815+0000 7f04acc00640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f048806bc20 0x7f048806e0e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f049c004de0 tx=0x7f049c007f10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:08.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:08.816+0000 7f04a4c00640 1 -- 172.21.15.102:0/3140218198 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f04a0058210 con 0x7f04a80d0250 2024-03-18T16:35:09.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.067+0000 7f04a4c00640 1 -- 172.21.15.102:0/3140218198 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f04a81e11a0 con 0x7f04a80d0250 2024-03-18T16:35:09.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.308+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04a8074fd0 con 0x7f04a80d0250 2024-03-18T16:35:09.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.324+0000 7f04a4c00640 1 -- 172.21.15.102:0/3140218198 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f04a0022030 con 0x7f04a80d0250 2024-03-18T16:35:09.326 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:09.326 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:09.326 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:09.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f048806bc20 msgr2=0x7f048806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:09.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f048806bc20 0x7f048806e0e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f049c004de0 tx=0x7f049c007f10 comp rx=0 tx=0).stop 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 msgr2=0x7f04a81f1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a81f1500 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f04a000efc0 tx=0x7f04a000c490 comp rx=0 tx=0).stop 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 shutdown_connections 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f048806bc20 0x7f048806e0e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04a80d0250 0x7f04a81f1500 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04a80cfe70 0x7f04a81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 --2- 172.21.15.102:0/3140218198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04a80cc2f0 0x7f04a8155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 >> 172.21.15.102:0/3140218198 conn(0x7f04a808c3e0 msgr2=0x7f04a80ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 shutdown_connections 2024-03-18T16:35:09.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:09.328+0000 7f04ae000640 1 -- 172.21.15.102:0/3140218198 wait complete. 2024-03-18T16:35:09.957 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:09.957 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:10.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.131+0000 7f52f1600640 1 Processor -- start 2024-03-18T16:35:11.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.131+0000 7f52f1600640 1 -- start start 2024-03-18T16:35:11.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.132+0000 7f52f1600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec141650 0x7f52ec158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:11.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.132+0000 7f52f1600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52ec0c72d0 con 0x7f52ec0c4fa0 2024-03-18T16:35:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.132+0000 7f52f1600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52ec158ad0 con 0x7f52ec15c3a0 2024-03-18T16:35:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.132+0000 7f52f1600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f52ec159370 con 0x7f52ec141650 2024-03-18T16:35:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.133+0000 7f52ebe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec141650 0x7f52ec158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:11.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.133+0000 7f52f0c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f52ec0c4fa0 0x7f52ec162cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55320/0 (socket says 172.21.15.102:55320) 2024-03-18T16:35:11.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.133+0000 7f52f0c00640 1 -- 172.21.15.102:0/1638674692 learned_addr learned my addr 172.21.15.102:0/1638674692 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:11.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.136+0000 7f52ebe00640 1 -- 172.21.15.102:0/1638674692 >> v1:172.21.15.154:6789/0 conn(0x7f52ec15c3a0 legacy=0x7f52ec0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:11.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.136+0000 7f52ebe00640 1 -- 172.21.15.102:0/1638674692 >> v1:172.21.15.102:6789/0 conn(0x7f52ec0c4fa0 legacy=0x7f52ec162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:11.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.136+0000 7f52ebe00640 1 -- 172.21.15.102:0/1638674692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ec158db0 con 0x7f52ec141650 2024-03-18T16:35:11.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.139+0000 7f52ebe00640 1 --2- 172.21.15.102:0/1638674692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec141650 0x7f52ec158420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f52dc0097d0 tx=0x7f52dc02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=673c3107a47eb4ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:11.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.142+0000 7f52eaa00640 1 -- 172.21.15.102:0/1638674692 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc035020 con 0x7f52ec141650 2024-03-18T16:35:11.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.142+0000 7f52eaa00640 1 -- 172.21.15.102:0/1638674692 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc031810 con 0x7f52ec141650 2024-03-18T16:35:11.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.142+0000 7f52eaa00640 1 -- 172.21.15.102:0/1638674692 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc031be0 con 0x7f52ec141650 2024-03-18T16:35:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.143+0000 7f52f1600640 1 -- 172.21.15.102:0/1638674692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec141650 msgr2=0x7f52ec158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.143+0000 7f52f1600640 1 --2- 172.21.15.102:0/1638674692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec141650 0x7f52ec158420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f52dc0097d0 tx=0x7f52dc02b8b0 comp rx=0 tx=0).stop 2024-03-18T16:35:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.143+0000 7f52f1600640 1 -- 172.21.15.102:0/1638674692 shutdown_connections 2024-03-18T16:35:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.143+0000 7f52f1600640 1 --2- 172.21.15.102:0/1638674692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec141650 0x7f52ec158420 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.143+0000 7f52f1600640 1 -- 172.21.15.102:0/1638674692 >> 172.21.15.102:0/1638674692 conn(0x7f52ec0cb890 msgr2=0x7f52ec0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.143+0000 7f52f1600640 1 -- 172.21.15.102:0/1638674692 shutdown_connections 2024-03-18T16:35:11.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.143+0000 7f52f1600640 1 -- 172.21.15.102:0/1638674692 wait complete. 2024-03-18T16:35:11.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.146+0000 7f52f1600640 1 Processor -- start 2024-03-18T16:35:11.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.146+0000 7f52f1600640 1 -- start start 2024-03-18T16:35:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.148+0000 7f52f1600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52ec0c4fa0 0x7f52ec1f1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.148+0000 7f52f1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ec141650 0x7f52ec1f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.148+0000 7f52f1600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec15c3a0 0x7f52ec0cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.148+0000 7f52f1600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec0c66a0 con 0x7f52ec141650 2024-03-18T16:35:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.148+0000 7f52f1600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52ec0c63c0 con 0x7f52ec0c4fa0 2024-03-18T16:35:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.148+0000 7f52f1600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f52ec15c7c0 con 0x7f52ec15c3a0 2024-03-18T16:35:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.148+0000 7f52f0c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec15c3a0 0x7f52ec0cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.148+0000 7f52ebe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52ec0c4fa0 0x7f52ec1f1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.149+0000 7f52eb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ec141650 0x7f52ec1f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:11.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.149+0000 7f52f0c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec15c3a0 0x7f52ec0cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57790/0 (socket says 172.21.15.102:57790) 2024-03-18T16:35:11.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.149+0000 7f52f0c00640 1 -- 172.21.15.102:0/743898873 learned_addr learned my addr 172.21.15.102:0/743898873 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:11.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.152+0000 7f52f0c00640 1 -- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52ec0c4fa0 msgr2=0x7f52ec1f1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:11.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.152+0000 7f52f0c00640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52ec0c4fa0 0x7f52ec1f1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:11.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.152+0000 7f52f0c00640 1 -- 172.21.15.102:0/743898873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ec141650 msgr2=0x7f52ec1f4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:11.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.152+0000 7f52f0c00640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ec141650 0x7f52ec1f4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:11.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.152+0000 7f52f0c00640 1 -- 172.21.15.102:0/743898873 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ec0c3a90 con 0x7f52ec15c3a0 2024-03-18T16:35:11.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.154+0000 7f52f0c00640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec15c3a0 0x7f52ec0cad40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f52e400cad0 tx=0x7f52e400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.158+0000 7f52e3e00640 1 -- 172.21.15.102:0/743898873 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e4017070 con 0x7f52ec15c3a0 2024-03-18T16:35:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.158+0000 7f52e3e00640 1 -- 172.21.15.102:0/743898873 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e400ac30 con 0x7f52ec15c3a0 2024-03-18T16:35:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.158+0000 7f52e3e00640 1 -- 172.21.15.102:0/743898873 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e4012620 con 0x7f52ec15c3a0 2024-03-18T16:35:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.158+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52ec0c3d70 con 0x7f52ec15c3a0 2024-03-18T16:35:11.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.158+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52ec0c4330 con 0x7f52ec15c3a0 2024-03-18T16:35:11.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.160+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52ec152590 con 0x7f52ec15c3a0 2024-03-18T16:35:11.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.179+0000 7f52e3e00640 1 -- 172.21.15.102:0/743898873 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f52e401a070 con 0x7f52ec15c3a0 2024-03-18T16:35:11.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.180+0000 7f52e3e00640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52cc06bc50 0x7f52cc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:11.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.181+0000 7f52ebe00640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52cc06bc50 0x7f52cc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:11.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.182+0000 7f52ebe00640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52cc06bc50 0x7f52cc06e110 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f52dc036000 tx=0x7f52dc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:11.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.182+0000 7f52e3e00640 1 -- 172.21.15.102:0/743898873 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f52e4057080 con 0x7f52ec15c3a0 2024-03-18T16:35:11.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.446+0000 7f52e3e00640 1 -- 172.21.15.102:0/743898873 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f52ec152590 con 0x7f52ec15c3a0 2024-03-18T16:35:11.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.733+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52ec067c20 con 0x7f52ec15c3a0 2024-03-18T16:35:11.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.748+0000 7f52e3e00640 1 -- 172.21.15.102:0/743898873 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f52ec067c20 con 0x7f52ec15c3a0 2024-03-18T16:35:11.750 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:11.750 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:11.750 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:11.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.751+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52cc06bc50 msgr2=0x7f52cc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:11.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.751+0000 7f52f1600640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52cc06bc50 0x7f52cc06e110 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f52dc036000 tx=0x7f52dc008030 comp rx=0 tx=0).stop 2024-03-18T16:35:11.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.751+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec15c3a0 msgr2=0x7f52ec0cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:11.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.751+0000 7f52f1600640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec15c3a0 0x7f52ec0cad40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f52e400cad0 tx=0x7f52e400cf90 comp rx=0 tx=0).stop 2024-03-18T16:35:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.752+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 shutdown_connections 2024-03-18T16:35:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.752+0000 7f52f1600640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f52cc06bc50 0x7f52cc06e110 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.752+0000 7f52f1600640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f52ec15c3a0 0x7f52ec0cad40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.752+0000 7f52f1600640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52ec141650 0x7f52ec1f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.752+0000 7f52f1600640 1 --2- 172.21.15.102:0/743898873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52ec0c4fa0 0x7f52ec1f1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.752+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 >> 172.21.15.102:0/743898873 conn(0x7f52ec0cb890 msgr2=0x7f52ec140910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.752+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 shutdown_connections 2024-03-18T16:35:11.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:11.752+0000 7f52f1600640 1 -- 172.21.15.102:0/743898873 wait complete. 2024-03-18T16:35:12.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:12.372 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:13.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:13.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.549+0000 7f8be6200640 1 Processor -- start 2024-03-18T16:35:13.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.549+0000 7f8be6200640 1 -- start start 2024-03-18T16:35:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.550+0000 7f8be6200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00cfe70 0x7f8be015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.551+0000 7f8be6200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8be00cabd0 con 0x7f8be00d0250 2024-03-18T16:35:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.551+0000 7f8be6200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8be00caea0 con 0x7f8be00cc2f0 2024-03-18T16:35:13.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.551+0000 7f8be6200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8be00cb180 con 0x7f8be00cfe70 2024-03-18T16:35:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.551+0000 7f8be4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00cfe70 0x7f8be015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.552+0000 7f8be4e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00cfe70 0x7f8be015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57798/0 (socket says 172.21.15.102:57798) 2024-03-18T16:35:13.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.552+0000 7f8be4e00640 1 -- 172.21.15.102:0/2925186755 learned_addr learned my addr 172.21.15.102:0/2925186755 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:13.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.554+0000 7f8be4e00640 1 -- 172.21.15.102:0/2925186755 >> v1:172.21.15.154:6789/0 conn(0x7f8be00cc2f0 legacy=0x7f8be00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:13.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.554+0000 7f8be4e00640 1 -- 172.21.15.102:0/2925186755 >> v1:172.21.15.102:6789/0 conn(0x7f8be00d0250 legacy=0x7f8be00c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:13.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.554+0000 7f8be4e00640 1 -- 172.21.15.102:0/2925186755 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be00cb460 con 0x7f8be00cfe70 2024-03-18T16:35:13.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.556+0000 7f8be4e00640 1 --2- 172.21.15.102:0/2925186755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00cfe70 0x7f8be015ae40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8bd40098a0 tx=0x7f8bd402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58a360a5b9b04381 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:13.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.561+0000 7f8bdf400640 1 -- 172.21.15.102:0/2925186755 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4034070 con 0x7f8be00cfe70 2024-03-18T16:35:13.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.561+0000 7f8bdf400640 1 -- 172.21.15.102:0/2925186755 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bd4030840 con 0x7f8be00cfe70 2024-03-18T16:35:13.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.561+0000 7f8bdf400640 1 -- 172.21.15.102:0/2925186755 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4030c10 con 0x7f8be00cfe70 2024-03-18T16:35:13.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.562+0000 7f8be6200640 1 -- 172.21.15.102:0/2925186755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00cfe70 msgr2=0x7f8be015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:13.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.562+0000 7f8be6200640 1 --2- 172.21.15.102:0/2925186755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00cfe70 0x7f8be015ae40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8bd40098a0 tx=0x7f8bd402b980 comp rx=0 tx=0).stop 2024-03-18T16:35:13.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.562+0000 7f8be6200640 1 -- 172.21.15.102:0/2925186755 shutdown_connections 2024-03-18T16:35:13.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.562+0000 7f8be6200640 1 --2- 172.21.15.102:0/2925186755 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00cfe70 0x7f8be015ae40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:13.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.562+0000 7f8be6200640 1 -- 172.21.15.102:0/2925186755 >> 172.21.15.102:0/2925186755 conn(0x7f8be008c3e0 msgr2=0x7f8be006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:13.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.562+0000 7f8be6200640 1 -- 172.21.15.102:0/2925186755 shutdown_connections 2024-03-18T16:35:13.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.562+0000 7f8be6200640 1 -- 172.21.15.102:0/2925186755 wait complete. 2024-03-18T16:35:13.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.564+0000 7f8be6200640 1 Processor -- start 2024-03-18T16:35:13.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.564+0000 7f8be6200640 1 -- start start 2024-03-18T16:35:13.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.565+0000 7f8be6200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8be00cc2f0 0x7f8be0150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:13.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.565+0000 7f8be6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8be00cfe70 0x7f8be01f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:13.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.565+0000 7f8be6200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00d0250 0x7f8be01f06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:13.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.565+0000 7f8be6200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8be00cbd00 con 0x7f8be00cfe70 2024-03-18T16:35:13.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.565+0000 7f8be6200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8be00cba20 con 0x7f8be00cc2f0 2024-03-18T16:35:13.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.565+0000 7f8be6200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8be00c8330 con 0x7f8be00d0250 2024-03-18T16:35:13.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.566+0000 7f8be5800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00d0250 0x7f8be01f06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:13.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.567+0000 7f8be5800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00d0250 0x7f8be01f06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57810/0 (socket says 172.21.15.102:57810) 2024-03-18T16:35:13.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.567+0000 7f8be5800640 1 -- 172.21.15.102:0/1670132765 learned_addr learned my addr 172.21.15.102:0/1670132765 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:13.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.570+0000 7f8be5800640 1 -- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8be00cc2f0 msgr2=0x7f8be0150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:13.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.570+0000 7f8be5800640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8be00cc2f0 0x7f8be0150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:13.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.570+0000 7f8be5800640 1 -- 172.21.15.102:0/1670132765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8be00cfe70 msgr2=0x7f8be01f0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:13.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.570+0000 7f8be5800640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8be00cfe70 0x7f8be01f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:13.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.570+0000 7f8be5800640 1 -- 172.21.15.102:0/1670132765 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be01f3e40 con 0x7f8be00d0250 2024-03-18T16:35:13.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.573+0000 7f8be5800640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00d0250 0x7f8be01f06a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800c7b0 tx=0x7f8bd800cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:13.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.577+0000 7f8bdcc00640 1 -- 172.21.15.102:0/1670132765 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd8017070 con 0x7f8be00d0250 2024-03-18T16:35:13.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.578+0000 7f8bdcc00640 1 -- 172.21.15.102:0/1670132765 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bd8003da0 con 0x7f8be00d0250 2024-03-18T16:35:13.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.578+0000 7f8bdcc00640 1 -- 172.21.15.102:0/1670132765 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd80052b0 con 0x7f8be00d0250 2024-03-18T16:35:13.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.579+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be014bec0 con 0x7f8be00d0250 2024-03-18T16:35:13.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.579+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be014c480 con 0x7f8be00d0250 2024-03-18T16:35:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.580+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be0074fd0 con 0x7f8be00d0250 2024-03-18T16:35:13.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.883+0000 7f8bdcc00640 1 -- 172.21.15.102:0/1670132765 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8bd801a070 con 0x7f8be00d0250 2024-03-18T16:35:13.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.885+0000 7f8bdcc00640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8bc006bc50 0x7f8bc006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:13.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.886+0000 7f8be4e00640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8bc006bc50 0x7f8bc006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:13.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.886+0000 7f8be4e00640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8bc006bc50 0x7f8bc006e110 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4036000 tx=0x7f8bd403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:13.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.888+0000 7f8bdcc00640 1 -- 172.21.15.102:0/1670132765 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8bd80569f0 con 0x7f8be00d0250 2024-03-18T16:35:13.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:13.907+0000 7f8bdcc00640 1 -- 172.21.15.102:0/1670132765 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8bd8021020 con 0x7f8be00d0250 2024-03-18T16:35:14.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.146+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8be0151630 con 0x7f8be00d0250 2024-03-18T16:35:14.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.159+0000 7f8bdcc00640 1 -- 172.21.15.102:0/1670132765 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8bd8059050 con 0x7f8be00d0250 2024-03-18T16:35:14.161 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:14.161 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:14.161 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:14.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8bc006bc50 msgr2=0x7f8bc006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8bc006bc50 0x7f8bc006e110 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4036000 tx=0x7f8bd403b040 comp rx=0 tx=0).stop 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00d0250 msgr2=0x7f8be01f06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00d0250 0x7f8be01f06a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800c7b0 tx=0x7f8bd800cc70 comp rx=0 tx=0).stop 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 shutdown_connections 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8bc006bc50 0x7f8bc006e110 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8be00d0250 0x7f8be01f06a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8be00cfe70 0x7f8be01f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 --2- 172.21.15.102:0/1670132765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8be00cc2f0 0x7f8be0150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 >> 172.21.15.102:0/1670132765 conn(0x7f8be008c3e0 msgr2=0x7f8be00c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 shutdown_connections 2024-03-18T16:35:14.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:14.162+0000 7f8be6200640 1 -- 172.21.15.102:0/1670132765 wait complete. 2024-03-18T16:35:14.784 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:14.784 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:15.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.959+0000 7fbc0f200640 1 Processor -- start 2024-03-18T16:35:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.959+0000 7fbc0f200640 1 -- start start 2024-03-18T16:35:15.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.960+0000 7fbc0f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbc1006b470 0x7fbc1015ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.960+0000 7fbc0f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc1006bf60 con 0x7fbc10154150 2024-03-18T16:35:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.960+0000 7fbc0f200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc100c94b0 con 0x7fbc1006de50 2024-03-18T16:35:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.960+0000 7fbc0f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fbc100ca0a0 con 0x7fbc1006b470 2024-03-18T16:35:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.961+0000 7fbc0d400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fbc1006de50 0x7fbc1006b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:56796/0 (socket says 172.21.15.102:56796) 2024-03-18T16:35:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.961+0000 7fbc0d400640 1 -- 172.21.15.102:0/1682757874 learned_addr learned my addr 172.21.15.102:0/1682757874 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.962+0000 7fbc0de00640 1 --2- 172.21.15.102:0/1682757874 >> v2:172.21.15.102:3300/0 conn(0x7fbc1006b470 0x7fbc1015ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:15.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.965+0000 7fbc0ca00640 1 -- 172.21.15.102:0/1682757874 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3231389382 0 0) 0x7fbc100c94b0 con 0x7fbc1006de50 2024-03-18T16:35:15.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.965+0000 7fbc0ca00640 1 -- 172.21.15.102:0/1682757874 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbf4003520 con 0x7fbc1006de50 2024-03-18T16:35:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.965+0000 7fbc0de00640 1 -- 172.21.15.102:0/1682757874 >> v1:172.21.15.154:6789/0 conn(0x7fbc1006de50 legacy=0x7fbc1006b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.965+0000 7fbc0de00640 1 -- 172.21.15.102:0/1682757874 >> v1:172.21.15.102:6789/0 conn(0x7fbc10154150 legacy=0x7fbc100c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.965+0000 7fbc0de00640 1 -- 172.21.15.102:0/1682757874 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc100c9770 con 0x7fbc1006b470 2024-03-18T16:35:15.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.971+0000 7fbc0de00640 1 --2- 172.21.15.102:0/1682757874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc1006b470 0x7fbc1015ad60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbc000097a0 tx=0x7fbc0002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5951ea443c902785 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:15.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.976+0000 7fbc0ca00640 1 -- 172.21.15.102:0/1682757874 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00034070 con 0x7fbc1006b470 2024-03-18T16:35:15.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.976+0000 7fbc0ca00640 1 -- 172.21.15.102:0/1682757874 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc00030610 con 0x7fbc1006b470 2024-03-18T16:35:15.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.976+0000 7fbc0ca00640 1 -- 172.21.15.102:0/1682757874 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc000309e0 con 0x7fbc1006b470 2024-03-18T16:35:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.977+0000 7fbc0f200640 1 -- 172.21.15.102:0/1682757874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc1006b470 msgr2=0x7fbc1015ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.977+0000 7fbc0f200640 1 --2- 172.21.15.102:0/1682757874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc1006b470 0x7fbc1015ad60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbc000097a0 tx=0x7fbc0002b960 comp rx=0 tx=0).stop 2024-03-18T16:35:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.977+0000 7fbc0f200640 1 -- 172.21.15.102:0/1682757874 shutdown_connections 2024-03-18T16:35:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.977+0000 7fbc0f200640 1 --2- 172.21.15.102:0/1682757874 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc1006b470 0x7fbc1015ad60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.977+0000 7fbc0f200640 1 -- 172.21.15.102:0/1682757874 >> 172.21.15.102:0/1682757874 conn(0x7fbc100727e0 msgr2=0x7fbc10072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.977+0000 7fbc0f200640 1 -- 172.21.15.102:0/1682757874 shutdown_connections 2024-03-18T16:35:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.977+0000 7fbc0f200640 1 -- 172.21.15.102:0/1682757874 wait complete. 2024-03-18T16:35:15.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.980+0000 7fbc0f200640 1 Processor -- start 2024-03-18T16:35:15.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.980+0000 7fbc0f200640 1 -- start start 2024-03-18T16:35:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.981+0000 7fbc0f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc1006b470 0x7fbc100c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.981+0000 7fbc0f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc1006de50 0x7fbc101491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.981+0000 7fbc0f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc10154150 0x7fbc10149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.981+0000 7fbc0f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fbc100c58a0 con 0x7fbc10154150 2024-03-18T16:35:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.981+0000 7fbc0f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc100c9d30 con 0x7fbc1006de50 2024-03-18T16:35:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.981+0000 7fbc0f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fbc100c5a20 con 0x7fbc1006b470 2024-03-18T16:35:15.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.982+0000 7fbc0de00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc1006b470 0x7fbc100c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:15.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.983+0000 7fbc0e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc10154150 0x7fbc10149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:15.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.983+0000 7fbc0de00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc1006b470 0x7fbc100c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57818/0 (socket says 172.21.15.102:57818) 2024-03-18T16:35:15.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.983+0000 7fbc0de00640 1 -- 172.21.15.102:0/2155957416 learned_addr learned my addr 172.21.15.102:0/2155957416 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:15.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.984+0000 7fbc0d400640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc1006de50 0x7fbc101491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:15.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.988+0000 7fbc0de00640 1 -- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc1006de50 msgr2=0x7fbc101491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:15.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.988+0000 7fbc0de00640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc1006de50 0x7fbc101491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:15.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.988+0000 7fbc0de00640 1 -- 172.21.15.102:0/2155957416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc10154150 msgr2=0x7fbc10149710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:15.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.988+0000 7fbc0de00640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc10154150 0x7fbc10149710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:15.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.988+0000 7fbc0de00640 1 -- 172.21.15.102:0/2155957416 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1014a140 con 0x7fbc1006b470 2024-03-18T16:35:15.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.991+0000 7fbc0de00640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc1006b470 0x7fbc100c7d50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbc000098c0 tx=0x7fbc000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:15.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.995+0000 7fbc06000640 1 -- 172.21.15.102:0/2155957416 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00034040 con 0x7fbc1006b470 2024-03-18T16:35:15.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.995+0000 7fbc06000640 1 -- 172.21.15.102:0/2155957416 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc00030410 con 0x7fbc1006b470 2024-03-18T16:35:15.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.995+0000 7fbc06000640 1 -- 172.21.15.102:0/2155957416 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00039c50 con 0x7fbc1006b470 2024-03-18T16:35:15.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.995+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc101f0520 con 0x7fbc1006b470 2024-03-18T16:35:15.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.995+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc101f07c0 con 0x7fbc1006b470 2024-03-18T16:35:15.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:15.997+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc10074fd0 con 0x7fbc1006b470 2024-03-18T16:35:16.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.016+0000 7fbc06000640 1 -- 172.21.15.102:0/2155957416 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbc00037070 con 0x7fbc1006b470 2024-03-18T16:35:16.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.017+0000 7fbc06000640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbbf006bc20 0x7fbbf006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:16.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.018+0000 7fbc0d400640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbbf006bc20 0x7fbbf006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:16.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.018+0000 7fbc06000640 1 -- 172.21.15.102:0/2155957416 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fbc000305c0 con 0x7fbc1006b470 2024-03-18T16:35:16.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.018+0000 7fbc0d400640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbbf006bc20 0x7fbbf006e0e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbc1015c3f0 tx=0x7fbbfc007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:16.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.267+0000 7fbc06000640 1 -- 172.21.15.102:0/2155957416 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbc1014a140 con 0x7fbc1006b470 2024-03-18T16:35:16.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.535+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc100cc610 con 0x7fbc1006b470 2024-03-18T16:35:16.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.547+0000 7fbc06000640 1 -- 172.21.15.102:0/2155957416 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fbc0007b830 con 0x7fbc1006b470 2024-03-18T16:35:16.549 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:16.549 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:16.550 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbbf006bc20 msgr2=0x7fbbf006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbbf006bc20 0x7fbbf006e0e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbc1015c3f0 tx=0x7fbbfc007d30 comp rx=0 tx=0).stop 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc1006b470 msgr2=0x7fbc100c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc1006b470 0x7fbc100c7d50 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbc000098c0 tx=0x7fbc000046c0 comp rx=0 tx=0).stop 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 shutdown_connections 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fbbf006bc20 0x7fbbf006e0e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc10154150 0x7fbc10149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc1006de50 0x7fbc101491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 --2- 172.21.15.102:0/2155957416 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fbc1006b470 0x7fbc100c7d50 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 >> 172.21.15.102:0/2155957416 conn(0x7fbc100727e0 msgr2=0x7fbc100c5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:16.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.550+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 shutdown_connections 2024-03-18T16:35:16.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:16.552+0000 7fbc0f200640 1 -- 172.21.15.102:0/2155957416 wait complete. 2024-03-18T16:35:17.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:17.182 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:18.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:18.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.363+0000 7f926be00640 1 Processor -- start 2024-03-18T16:35:18.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.363+0000 7f926be00640 1 -- start start 2024-03-18T16:35:18.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.364+0000 7f926be00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f926c141650 0x7f926c158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:18.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.364+0000 7f926be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f926c0c4fa0 0x7f926c0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.364+0000 7f926be00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f926c0c72d0 con 0x7f926c15c3a0 2024-03-18T16:35:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.364+0000 7f926be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f926c159370 con 0x7f926c0c4fa0 2024-03-18T16:35:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.364+0000 7f926be00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f926c158ad0 con 0x7f926c141650 2024-03-18T16:35:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.365+0000 7f926aa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f926c141650 0x7f926c158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:18.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.365+0000 7f926b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f926c0c4fa0 0x7f926c0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:18.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.365+0000 7f926b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f926c0c4fa0 0x7f926c0c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48644/0 (socket says 172.21.15.102:48644) 2024-03-18T16:35:18.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.365+0000 7f926aa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f926c141650 0x7f926c158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48234/0 (socket says 172.21.15.102:48234) 2024-03-18T16:35:18.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.365+0000 7f926b400640 1 -- 172.21.15.102:0/1043868396 learned_addr learned my addr 172.21.15.102:0/1043868396 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:18.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.368+0000 7f926b400640 1 -- 172.21.15.102:0/1043868396 >> v2:172.21.15.154:3300/0 conn(0x7f926c141650 msgr2=0x7f926c158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:18.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.368+0000 7f926b400640 1 --2- 172.21.15.102:0/1043868396 >> v2:172.21.15.154:3300/0 conn(0x7f926c141650 0x7f926c158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:18.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.368+0000 7f926b400640 1 -- 172.21.15.102:0/1043868396 >> v1:172.21.15.154:6789/0 conn(0x7f926c15c3a0 legacy=0x7f926c0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:18.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.368+0000 7f926b400640 1 -- 172.21.15.102:0/1043868396 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c158db0 con 0x7f926c0c4fa0 2024-03-18T16:35:18.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.370+0000 7f926b400640 1 --2- 172.21.15.102:0/1043868396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c0c4fa0 0x7f926c0c5410 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f926c068ac0 tx=0x7f926402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd1951471e20b7f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:18.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.375+0000 7f9269600640 1 -- 172.21.15.102:0/1043868396 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f926402cb20 con 0x7f926c0c4fa0 2024-03-18T16:35:18.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.375+0000 7f9269600640 1 -- 172.21.15.102:0/1043868396 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f926402ccc0 con 0x7f926c0c4fa0 2024-03-18T16:35:18.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.375+0000 7f9269600640 1 -- 172.21.15.102:0/1043868396 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264034610 con 0x7f926c0c4fa0 2024-03-18T16:35:18.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.376+0000 7f926be00640 1 -- 172.21.15.102:0/1043868396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c0c4fa0 msgr2=0x7f926c0c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:18.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.376+0000 7f926be00640 1 --2- 172.21.15.102:0/1043868396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c0c4fa0 0x7f926c0c5410 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f926c068ac0 tx=0x7f926402bf80 comp rx=0 tx=0).stop 2024-03-18T16:35:18.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.376+0000 7f926be00640 1 -- 172.21.15.102:0/1043868396 shutdown_connections 2024-03-18T16:35:18.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.376+0000 7f926be00640 1 --2- 172.21.15.102:0/1043868396 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c0c4fa0 0x7f926c0c5410 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:18.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.376+0000 7f926be00640 1 --2- 172.21.15.102:0/1043868396 >> v2:172.21.15.154:3300/0 conn(0x7f926c141650 0x7f926c158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:18.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.376+0000 7f926be00640 1 -- 172.21.15.102:0/1043868396 >> 172.21.15.102:0/1043868396 conn(0x7f926c0cb890 msgr2=0x7f926c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:18.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.376+0000 7f926be00640 1 -- 172.21.15.102:0/1043868396 shutdown_connections 2024-03-18T16:35:18.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.376+0000 7f926be00640 1 -- 172.21.15.102:0/1043868396 wait complete. 2024-03-18T16:35:18.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.378+0000 7f926be00640 1 Processor -- start 2024-03-18T16:35:18.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.378+0000 7f926be00640 1 -- start start 2024-03-18T16:35:18.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.379+0000 7f926be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c0c4fa0 0x7f926c154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:18.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.379+0000 7f926be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c141650 0x7f926c15b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:18.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.379+0000 7f926be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926c15c3a0 0x7f926c13c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:18.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.379+0000 7f926be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f926c0c66a0 con 0x7f926c141650 2024-03-18T16:35:18.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.379+0000 7f926be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f926c0c63c0 con 0x7f926c0c4fa0 2024-03-18T16:35:18.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.379+0000 7f926be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f926c15c7c0 con 0x7f926c15c3a0 2024-03-18T16:35:18.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.380+0000 7f926aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c0c4fa0 0x7f926c154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:18.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.380+0000 7f926a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c141650 0x7f926c15b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:18.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.380+0000 7f926b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926c15c3a0 0x7f926c13c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:18.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.380+0000 7f926a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c141650 0x7f926c15b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48646/0 (socket says 172.21.15.102:48646) 2024-03-18T16:35:18.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.380+0000 7f926a000640 1 -- 172.21.15.102:0/381354217 learned_addr learned my addr 172.21.15.102:0/381354217 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:18.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.380+0000 7f926aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c0c4fa0 0x7f926c154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48240/0 (socket says 172.21.15.102:48240) 2024-03-18T16:35:18.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.384+0000 7f926aa00640 1 -- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926c15c3a0 msgr2=0x7f926c13c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:18.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.384+0000 7f926aa00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926c15c3a0 0x7f926c13c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:18.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.384+0000 7f926aa00640 1 -- 172.21.15.102:0/381354217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c141650 msgr2=0x7f926c15b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:18.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.384+0000 7f926aa00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c141650 0x7f926c15b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:18.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.384+0000 7f926aa00640 1 -- 172.21.15.102:0/381354217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c155a20 con 0x7f926c0c4fa0 2024-03-18T16:35:18.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.386+0000 7f926aa00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c0c4fa0 0x7f926c154f30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f925c00e8b0 tx=0x7f925c00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:18.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.388+0000 7f9262a00640 1 -- 172.21.15.102:0/381354217 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c017070 con 0x7f926c0c4fa0 2024-03-18T16:35:18.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.388+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926c155d00 con 0x7f926c0c4fa0 2024-03-18T16:35:18.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.388+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926c159900 con 0x7f926c0c4fa0 2024-03-18T16:35:18.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.390+0000 7f9262a00640 1 -- 172.21.15.102:0/381354217 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f925c005990 con 0x7f926c0c4fa0 2024-03-18T16:35:18.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.390+0000 7f9262a00640 1 -- 172.21.15.102:0/381354217 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c021060 con 0x7f926c0c4fa0 2024-03-18T16:35:18.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.390+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f926c074fd0 con 0x7f926c0c4fa0 2024-03-18T16:35:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.650+0000 7f9262a00640 1 -- 172.21.15.102:0/381354217 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f925c01a070 con 0x7f926c0c4fa0 2024-03-18T16:35:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.652+0000 7f9262a00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f924c06bbd0 0x7f924c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.653+0000 7f926a000640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f924c06bbd0 0x7f924c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.653+0000 7f9262a00640 1 -- 172.21.15.102:0/381354217 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f925c057300 con 0x7f926c0c4fa0 2024-03-18T16:35:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.654+0000 7f926a000640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f924c06bbd0 0x7f924c06e090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f926c1406e0 tx=0x7f9258009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:18.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.663+0000 7f9262a00640 1 -- 172.21.15.102:0/381354217 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f925c059260 con 0x7f926c0c4fa0 2024-03-18T16:35:18.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.983+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f926c0b7f00 con 0x7f926c0c4fa0 2024-03-18T16:35:18.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:18.997+0000 7f9262a00640 1 -- 172.21.15.102:0/381354217 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f925c05d300 con 0x7f926c0c4fa0 2024-03-18T16:35:18.999 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:18.999 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:18.999 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:19.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.000+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f924c06bbd0 msgr2=0x7f924c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:19.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.000+0000 7f926be00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f924c06bbd0 0x7f924c06e090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f926c1406e0 tx=0x7f9258009040 comp rx=0 tx=0).stop 2024-03-18T16:35:19.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.000+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c0c4fa0 msgr2=0x7f926c154f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.000+0000 7f926be00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c0c4fa0 0x7f926c154f30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f925c00e8b0 tx=0x7f925c00ed70 comp rx=0 tx=0).stop 2024-03-18T16:35:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.001+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 shutdown_connections 2024-03-18T16:35:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.001+0000 7f926be00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f924c06bbd0 0x7f924c06e090 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.001+0000 7f926be00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f926c15c3a0 0x7f926c13c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.001+0000 7f926be00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f926c141650 0x7f926c15b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.001+0000 7f926be00640 1 --2- 172.21.15.102:0/381354217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f926c0c4fa0 0x7f926c154f30 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.001+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 >> 172.21.15.102:0/381354217 conn(0x7f926c0cb890 msgr2=0x7f926c140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.001+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 shutdown_connections 2024-03-18T16:35:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:19.001+0000 7f926be00640 1 -- 172.21.15.102:0/381354217 wait complete. 2024-03-18T16:35:19.630 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:19.630 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:20.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.805+0000 7fc03a600640 1 Processor -- start 2024-03-18T16:35:20.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.805+0000 7fc03a600640 1 -- start start 2024-03-18T16:35:20.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.806+0000 7fc03a600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc0340c9be0 0x7fc0340d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:20.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.806+0000 7fc03a600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0340d05a0 0x7fc0340cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:20.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.806+0000 7fc03a600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc034154ab0 con 0x7fc03415ae70 2024-03-18T16:35:20.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.806+0000 7fc03a600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc0340caba0 con 0x7fc0340d05a0 2024-03-18T16:35:20.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.806+0000 7fc03a600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc03406d2f0 con 0x7fc0340c9be0 2024-03-18T16:35:20.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.807+0000 7fc033e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc0340c9be0 0x7fc0340d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:20.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.807+0000 7fc039c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc0340d05a0 0x7fc0340cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:20.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.807+0000 7fc033e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc0340c9be0 0x7fc0340d0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48242/0 (socket says 172.21.15.102:48242) 2024-03-18T16:35:20.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.807+0000 7fc039200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc03415ae70 0x7fc0340ca490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55346/0 (socket says 172.21.15.102:55346) 2024-03-18T16:35:20.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.807+0000 7fc033e00640 1 -- 172.21.15.102:0/3192190828 learned_addr learned my addr 172.21.15.102:0/3192190828 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:20.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.810+0000 7fc033e00640 1 -- 172.21.15.102:0/3192190828 >> v2:172.21.15.102:3300/0 conn(0x7fc0340d05a0 msgr2=0x7fc0340cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:20.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.810+0000 7fc033e00640 1 --2- 172.21.15.102:0/3192190828 >> v2:172.21.15.102:3300/0 conn(0x7fc0340d05a0 0x7fc0340cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:20.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.810+0000 7fc033e00640 1 -- 172.21.15.102:0/3192190828 >> v1:172.21.15.102:6789/0 conn(0x7fc03415ae70 legacy=0x7fc0340ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:20.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.810+0000 7fc033e00640 1 -- 172.21.15.102:0/3192190828 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03415b250 con 0x7fc0340c9be0 2024-03-18T16:35:20.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.812+0000 7fc033e00640 1 --2- 172.21.15.102:0/3192190828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0340c9be0 0x7fc0340d0060 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc034157f40 tx=0x7fc02002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6a29a7b401146da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:20.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.815+0000 7fc033400640 1 -- 172.21.15.102:0/3192190828 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020034070 con 0x7fc0340c9be0 2024-03-18T16:35:20.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.816+0000 7fc033400640 1 -- 172.21.15.102:0/3192190828 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc020030860 con 0x7fc0340c9be0 2024-03-18T16:35:20.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.817+0000 7fc033400640 1 -- 172.21.15.102:0/3192190828 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020030c30 con 0x7fc0340c9be0 2024-03-18T16:35:20.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.817+0000 7fc03a600640 1 -- 172.21.15.102:0/3192190828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0340c9be0 msgr2=0x7fc0340d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:20.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.817+0000 7fc03a600640 1 --2- 172.21.15.102:0/3192190828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0340c9be0 0x7fc0340d0060 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc034157f40 tx=0x7fc02002b9a0 comp rx=0 tx=0).stop 2024-03-18T16:35:20.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.818+0000 7fc03a600640 1 -- 172.21.15.102:0/3192190828 shutdown_connections 2024-03-18T16:35:20.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.818+0000 7fc03a600640 1 --2- 172.21.15.102:0/3192190828 >> v2:172.21.15.102:3300/0 conn(0x7fc0340d05a0 0x7fc0340cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:20.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.818+0000 7fc03a600640 1 --2- 172.21.15.102:0/3192190828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0340c9be0 0x7fc0340d0060 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:20.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.818+0000 7fc03a600640 1 -- 172.21.15.102:0/3192190828 >> 172.21.15.102:0/3192190828 conn(0x7fc0340673a0 msgr2=0x7fc0340677b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:20.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.818+0000 7fc03a600640 1 -- 172.21.15.102:0/3192190828 shutdown_connections 2024-03-18T16:35:20.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.818+0000 7fc03a600640 1 -- 172.21.15.102:0/3192190828 wait complete. 2024-03-18T16:35:20.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.820+0000 7fc03a600640 1 Processor -- start 2024-03-18T16:35:20.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.820+0000 7fc03a600640 1 -- start start 2024-03-18T16:35:20.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.821+0000 7fc03a600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0340c9be0 0x7fc0340c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:20.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.821+0000 7fc03a600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0340d05a0 0x7fc0340c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:20.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.821+0000 7fc03a600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc03415ae70 0x7fc0341f0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:20.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.821+0000 7fc03a600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc03406db90 con 0x7fc0340c9be0 2024-03-18T16:35:20.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.821+0000 7fc03a600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc03406d8b0 con 0x7fc03415ae70 2024-03-18T16:35:20.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.821+0000 7fc03a600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc03406de70 con 0x7fc0340d05a0 2024-03-18T16:35:20.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.821+0000 7fc039c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc03415ae70 0x7fc0341f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:20.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.821+0000 7fc039200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0340c9be0 0x7fc0340c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:20.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.822+0000 7fc033e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0340d05a0 0x7fc0340c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:20.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.822+0000 7fc039c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc03415ae70 0x7fc0341f0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:48248/0 (socket says 172.21.15.102:48248) 2024-03-18T16:35:20.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.822+0000 7fc039c00640 1 -- 172.21.15.102:0/2167422488 learned_addr learned my addr 172.21.15.102:0/2167422488 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:20.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.826+0000 7fc039c00640 1 -- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0340d05a0 msgr2=0x7fc0340c80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:20.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.826+0000 7fc039c00640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0340d05a0 0x7fc0340c80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:20.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.826+0000 7fc039c00640 1 -- 172.21.15.102:0/2167422488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0340c9be0 msgr2=0x7fc0340c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:20.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.826+0000 7fc039c00640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0340c9be0 0x7fc0340c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:20.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.826+0000 7fc039c00640 1 -- 172.21.15.102:0/2167422488 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0341f3ba0 con 0x7fc03415ae70 2024-03-18T16:35:20.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.828+0000 7fc039c00640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc03415ae70 0x7fc0341f0670 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009d30 tx=0x7fc02c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:20.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.832+0000 7fc030c00640 1 -- 172.21.15.102:0/2167422488 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c018020 con 0x7fc03415ae70 2024-03-18T16:35:20.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.832+0000 7fc030c00640 1 -- 172.21.15.102:0/2167422488 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc02c014480 con 0x7fc03415ae70 2024-03-18T16:35:20.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.832+0000 7fc030c00640 1 -- 172.21.15.102:0/2167422488 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c014870 con 0x7fc03415ae70 2024-03-18T16:35:20.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.832+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0341f3e80 con 0x7fc03415ae70 2024-03-18T16:35:20.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.832+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0341f4440 con 0x7fc03415ae70 2024-03-18T16:35:20.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.834+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0340ca160 con 0x7fc03415ae70 2024-03-18T16:35:20.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.857+0000 7fc030c00640 1 -- 172.21.15.102:0/2167422488 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc02c005d10 con 0x7fc03415ae70 2024-03-18T16:35:20.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.859+0000 7fc030c00640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc01406bc20 0x7fc01406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:20.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.859+0000 7fc039200640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc01406bc20 0x7fc01406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:20.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:20.860+0000 7fc039200640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc01406bc20 0x7fc01406e0e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc028007fd0 tx=0x7fc028007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:21.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.101+0000 7fc030c00640 1 -- 172.21.15.102:0/2167422488 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fc02c058100 con 0x7fc03415ae70 2024-03-18T16:35:21.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.124+0000 7fc030c00640 1 -- 172.21.15.102:0/2167422488 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc0340ca160 con 0x7fc03415ae70 2024-03-18T16:35:21.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.451+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc034074fd0 con 0x7fc03415ae70 2024-03-18T16:35:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.464+0000 7fc030c00640 1 -- 172.21.15.102:0/2167422488 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fc02c022070 con 0x7fc03415ae70 2024-03-18T16:35:21.467 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:21.467 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:21.467 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:21.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.466+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc01406bc20 msgr2=0x7fc01406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:21.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.466+0000 7fc03a600640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc01406bc20 0x7fc01406e0e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc028007fd0 tx=0x7fc028007e80 comp rx=0 tx=0).stop 2024-03-18T16:35:21.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc03415ae70 msgr2=0x7fc0341f0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc03415ae70 0x7fc0341f0670 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009d30 tx=0x7fc02c00c490 comp rx=0 tx=0).stop 2024-03-18T16:35:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 shutdown_connections 2024-03-18T16:35:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc01406bc20 0x7fc01406e0e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc03415ae70 0x7fc0341f0670 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc0340d05a0 0x7fc0340c80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 --2- 172.21.15.102:0/2167422488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0340c9be0 0x7fc0340c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 >> 172.21.15.102:0/2167422488 conn(0x7fc0340673a0 msgr2=0x7fc0340c8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 shutdown_connections 2024-03-18T16:35:21.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:21.467+0000 7fc03a600640 1 -- 172.21.15.102:0/2167422488 wait complete. 2024-03-18T16:35:22.088 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:22.088 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:23.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:23.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.263+0000 7f1246200640 1 Processor -- start 2024-03-18T16:35:23.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.263+0000 7f1246200640 1 -- start start 2024-03-18T16:35:23.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.264+0000 7f1246200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1240141650 0x7f1240158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.264+0000 7f1246200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f124015c3a0 0x7f12400c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.264+0000 7f1246200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1240158ad0 con 0x7f12400c4fa0 2024-03-18T16:35:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.264+0000 7f1246200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1240159370 con 0x7f124015c3a0 2024-03-18T16:35:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.264+0000 7f1246200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1240158db0 con 0x7f1240141650 2024-03-18T16:35:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.265+0000 7f1244e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1240141650 0x7f1240158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:23.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.265+0000 7f123fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f124015c3a0 0x7f12400c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:23.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.265+0000 7f1244e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1240141650 0x7f1240158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:60538/0 (socket says 172.21.15.102:60538) 2024-03-18T16:35:23.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.265+0000 7f1244e00640 1 -- 172.21.15.102:0/2464295624 learned_addr learned my addr 172.21.15.102:0/2464295624 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:23.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.268+0000 7f1244e00640 1 -- 172.21.15.102:0/2464295624 >> v2:172.21.15.102:3300/0 conn(0x7f124015c3a0 msgr2=0x7f12400c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:23.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.268+0000 7f1244e00640 1 --2- 172.21.15.102:0/2464295624 >> v2:172.21.15.102:3300/0 conn(0x7f124015c3a0 0x7f12400c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:23.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.268+0000 7f1244e00640 1 -- 172.21.15.102:0/2464295624 >> v1:172.21.15.102:6789/0 conn(0x7f12400c4fa0 legacy=0x7f124006f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:23.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.268+0000 7f1244e00640 1 -- 172.21.15.102:0/2464295624 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1240159090 con 0x7f1240141650 2024-03-18T16:35:23.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.271+0000 7f1244e00640 1 --2- 172.21.15.102:0/2464295624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1240141650 0x7f1240158420 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1234009870 tx=0x7f123402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=472dd63787dfa227 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:23.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.276+0000 7f123f400640 1 -- 172.21.15.102:0/2464295624 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1234034070 con 0x7f1240141650 2024-03-18T16:35:23.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.276+0000 7f123f400640 1 -- 172.21.15.102:0/2464295624 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1234030840 con 0x7f1240141650 2024-03-18T16:35:23.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.276+0000 7f123f400640 1 -- 172.21.15.102:0/2464295624 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1234030c10 con 0x7f1240141650 2024-03-18T16:35:23.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.277+0000 7f1246200640 1 -- 172.21.15.102:0/2464295624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1240141650 msgr2=0x7f1240158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:23.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.277+0000 7f1246200640 1 --2- 172.21.15.102:0/2464295624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1240141650 0x7f1240158420 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1234009870 tx=0x7f123402b980 comp rx=0 tx=0).stop 2024-03-18T16:35:23.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.277+0000 7f1246200640 1 -- 172.21.15.102:0/2464295624 shutdown_connections 2024-03-18T16:35:23.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.277+0000 7f1246200640 1 --2- 172.21.15.102:0/2464295624 >> v2:172.21.15.102:3300/0 conn(0x7f124015c3a0 0x7f12400c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:23.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.277+0000 7f1246200640 1 --2- 172.21.15.102:0/2464295624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1240141650 0x7f1240158420 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:23.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.277+0000 7f1246200640 1 -- 172.21.15.102:0/2464295624 >> 172.21.15.102:0/2464295624 conn(0x7f12400cb890 msgr2=0x7f12400cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:23.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.277+0000 7f1246200640 1 -- 172.21.15.102:0/2464295624 shutdown_connections 2024-03-18T16:35:23.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.277+0000 7f1246200640 1 -- 172.21.15.102:0/2464295624 wait complete. 2024-03-18T16:35:23.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.279+0000 7f1246200640 1 Processor -- start 2024-03-18T16:35:23.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.279+0000 7f1246200640 1 -- start start 2024-03-18T16:35:23.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.280+0000 7f1246200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12400c4fa0 0x7f1240153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:23.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.280+0000 7f1246200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1240141650 0x7f124013e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:23.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.280+0000 7f1246200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f124015c3a0 0x7f12401ebed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:23.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.280+0000 7f1246200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f124015c7c0 con 0x7f1240141650 2024-03-18T16:35:23.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.280+0000 7f1246200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f12400c66a0 con 0x7f124015c3a0 2024-03-18T16:35:23.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.280+0000 7f1246200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1240141a30 con 0x7f12400c4fa0 2024-03-18T16:35:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.280+0000 7f1244e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12400c4fa0 0x7f1240153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.280+0000 7f123fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1240141650 0x7f124013e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.280+0000 7f1245800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f124015c3a0 0x7f12401ebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.281+0000 7f1244e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12400c4fa0 0x7f1240153930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:60544/0 (socket says 172.21.15.102:60544) 2024-03-18T16:35:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.281+0000 7f1244e00640 1 -- 172.21.15.102:0/1480721093 learned_addr learned my addr 172.21.15.102:0/1480721093 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:23.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.285+0000 7f1245800640 1 -- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12400c4fa0 msgr2=0x7f1240153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:23.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.285+0000 7f1245800640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12400c4fa0 0x7f1240153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:23.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.285+0000 7f1245800640 1 -- 172.21.15.102:0/1480721093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1240141650 msgr2=0x7f124013e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:23.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.285+0000 7f1245800640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1240141650 0x7f124013e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:23.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.285+0000 7f1245800640 1 -- 172.21.15.102:0/1480721093 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12401584b0 con 0x7f124015c3a0 2024-03-18T16:35:23.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.288+0000 7f1245800640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f124015c3a0 0x7f12401ebed0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f123800c6c0 tx=0x7f123800cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:23.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.292+0000 7f123cc00640 1 -- 172.21.15.102:0/1480721093 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1238003b70 con 0x7f124015c3a0 2024-03-18T16:35:23.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.292+0000 7f123cc00640 1 -- 172.21.15.102:0/1480721093 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1238003d10 con 0x7f124015c3a0 2024-03-18T16:35:23.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.292+0000 7f123cc00640 1 -- 172.21.15.102:0/1480721093 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12380052f0 con 0x7f124015c3a0 2024-03-18T16:35:23.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.292+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1240154090 con 0x7f124015c3a0 2024-03-18T16:35:23.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.292+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12401ef440 con 0x7f124015c3a0 2024-03-18T16:35:23.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.294+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1240074fd0 con 0x7f124015c3a0 2024-03-18T16:35:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.316+0000 7f123cc00640 1 -- 172.21.15.102:0/1480721093 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1238005490 con 0x7f124015c3a0 2024-03-18T16:35:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.317+0000 7f123cc00640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f122006bc20 0x7f122006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.318+0000 7f1244e00640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f122006bc20 0x7f122006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:23.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.319+0000 7f1244e00640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f122006bc20 0x7f122006e0e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1234009870 tx=0x7f1234008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:23.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.319+0000 7f123cc00640 1 -- 172.21.15.102:0/1480721093 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1238057700 con 0x7f124015c3a0 2024-03-18T16:35:23.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.583+0000 7f123cc00640 1 -- 172.21.15.102:0/1480721093 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f12401584b0 con 0x7f124015c3a0 2024-03-18T16:35:23.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.937+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12400b7f00 con 0x7f124015c3a0 2024-03-18T16:35:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.950+0000 7f123cc00640 1 -- 172.21.15.102:0/1480721093 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1238059660 con 0x7f124015c3a0 2024-03-18T16:35:23.953 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:23.953 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:23.953 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f122006bc20 msgr2=0x7f122006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f122006bc20 0x7f122006e0e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1234009870 tx=0x7f1234008030 comp rx=0 tx=0).stop 2024-03-18T16:35:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f124015c3a0 msgr2=0x7f12401ebed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f124015c3a0 0x7f12401ebed0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f123800c6c0 tx=0x7f123800cb80 comp rx=0 tx=0).stop 2024-03-18T16:35:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 shutdown_connections 2024-03-18T16:35:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f122006bc20 0x7f122006e0e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f124015c3a0 0x7f12401ebed0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:23.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1240141650 0x7f124013e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 --2- 172.21.15.102:0/1480721093 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f12400c4fa0 0x7f1240153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 >> 172.21.15.102:0/1480721093 conn(0x7f12400cb890 msgr2=0x7f12400ca1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 shutdown_connections 2024-03-18T16:35:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:23.952+0000 7f1246200640 1 -- 172.21.15.102:0/1480721093 wait complete. 2024-03-18T16:35:24.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:35:24.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:35:24.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:24.576 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:25.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:25.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.758+0000 7ff7f6800640 1 Processor -- start 2024-03-18T16:35:25.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.758+0000 7ff7f6800640 1 -- start start 2024-03-18T16:35:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.760+0000 7ff7f6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.760+0000 7ff7f6800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7f00c72d0 con 0x7ff7f00c4fa0 2024-03-18T16:35:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.760+0000 7ff7f6800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7f0158840 con 0x7ff7f0141650 2024-03-18T16:35:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.760+0000 7ff7f6800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f015c8e0 con 0x7ff7f0158420 2024-03-18T16:35:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.760+0000 7ff7f4a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.760+0000 7ff7f5400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff7f0141650 0x7ff7f00c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:36492/0 (socket says 172.21.15.102:36492) 2024-03-18T16:35:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.760+0000 7ff7f5400640 1 -- 172.21.15.102:0/2396187903 learned_addr learned my addr 172.21.15.102:0/2396187903 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.763+0000 7ff7f4a00640 1 -- 172.21.15.102:0/2396187903 >> v1:172.21.15.154:6789/0 conn(0x7ff7f0141650 legacy=0x7ff7f00c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.763+0000 7ff7f4a00640 1 -- 172.21.15.102:0/2396187903 >> v1:172.21.15.102:6789/0 conn(0x7ff7f00c4fa0 legacy=0x7ff7f0162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.763+0000 7ff7f4a00640 1 -- 172.21.15.102:0/2396187903 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f0158ad0 con 0x7ff7f0158420 2024-03-18T16:35:25.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.765+0000 7ff7f4a00640 1 --2- 172.21.15.102:0/2396187903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f015c3a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0097d0 tx=0x7ff7dc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8cc3665cae0642c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:25.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.769+0000 7ff7efe00640 1 -- 172.21.15.102:0/2396187903 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc035020 con 0x7ff7f0158420 2024-03-18T16:35:25.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.769+0000 7ff7efe00640 1 -- 172.21.15.102:0/2396187903 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7dc031800 con 0x7ff7f0158420 2024-03-18T16:35:25.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.769+0000 7ff7efe00640 1 -- 172.21.15.102:0/2396187903 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc031bd0 con 0x7ff7f0158420 2024-03-18T16:35:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.770+0000 7ff7f6800640 1 -- 172.21.15.102:0/2396187903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 msgr2=0x7ff7f015c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.770+0000 7ff7f6800640 1 --2- 172.21.15.102:0/2396187903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f015c3a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc0097d0 tx=0x7ff7dc02b8a0 comp rx=0 tx=0).stop 2024-03-18T16:35:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.770+0000 7ff7f6800640 1 -- 172.21.15.102:0/2396187903 shutdown_connections 2024-03-18T16:35:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.770+0000 7ff7f6800640 1 --2- 172.21.15.102:0/2396187903 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f015c3a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.770+0000 7ff7f6800640 1 -- 172.21.15.102:0/2396187903 >> 172.21.15.102:0/2396187903 conn(0x7ff7f00cb890 msgr2=0x7ff7f00cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.770+0000 7ff7f6800640 1 -- 172.21.15.102:0/2396187903 shutdown_connections 2024-03-18T16:35:25.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.770+0000 7ff7f6800640 1 -- 172.21.15.102:0/2396187903 wait complete. 2024-03-18T16:35:25.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.773+0000 7ff7f6800640 1 Processor -- start 2024-03-18T16:35:25.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.773+0000 7ff7f6800640 1 -- start start 2024-03-18T16:35:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.774+0000 7ff7f6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7f00c4fa0 0x7ff7f01f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.775+0000 7ff7f6800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7f0141650 0x7ff7f0152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.775+0000 7ff7f6800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f0155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.775+0000 7ff7f6800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0159090 con 0x7ff7f00c4fa0 2024-03-18T16:35:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.775+0000 7ff7f6800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0159370 con 0x7ff7f0141650 2024-03-18T16:35:25.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.775+0000 7ff7f6800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f00c63c0 con 0x7ff7f0158420 2024-03-18T16:35:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.775+0000 7ff7f4a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7f0141650 0x7ff7f0152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.775+0000 7ff7f5e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f0155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.775+0000 7ff7f5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7f00c4fa0 0x7ff7f01f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.776+0000 7ff7f5e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f0155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:60558/0 (socket says 172.21.15.102:60558) 2024-03-18T16:35:25.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.776+0000 7ff7f5e00640 1 -- 172.21.15.102:0/1162495824 learned_addr learned my addr 172.21.15.102:0/1162495824 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:25.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.779+0000 7ff7f5e00640 1 -- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7f0141650 msgr2=0x7ff7f0152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:25.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.779+0000 7ff7f5e00640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7f0141650 0x7ff7f0152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:25.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.779+0000 7ff7f5e00640 1 -- 172.21.15.102:0/1162495824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7f00c4fa0 msgr2=0x7ff7f01f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:25.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.779+0000 7ff7f5e00640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7f00c4fa0 0x7ff7f01f4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:25.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.779+0000 7ff7f5e00640 1 -- 172.21.15.102:0/1162495824 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f0153440 con 0x7ff7f0158420 2024-03-18T16:35:25.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.782+0000 7ff7f5e00640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f0155a00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800c7b0 tx=0x7ff7e800cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.786+0000 7ff7ed600640 1 -- 172.21.15.102:0/1162495824 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8004b60 con 0x7ff7f0158420 2024-03-18T16:35:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.786+0000 7ff7ed600640 1 -- 172.21.15.102:0/1162495824 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7e8004d00 con 0x7ff7f0158420 2024-03-18T16:35:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.786+0000 7ff7ed600640 1 -- 172.21.15.102:0/1162495824 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e800aa90 con 0x7ff7f0158420 2024-03-18T16:35:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.786+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f0153720 con 0x7ff7f0158420 2024-03-18T16:35:25.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.786+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f01f3c10 con 0x7ff7f0158420 2024-03-18T16:35:25.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:25.787+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f0074fd0 con 0x7ff7f0158420 2024-03-18T16:35:26.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.045+0000 7ff7ed600640 1 -- 172.21.15.102:0/1162495824 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff7e800ac30 con 0x7ff7f0158420 2024-03-18T16:35:26.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.048+0000 7ff7ed600640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff7d006bc00 0x7ff7d006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:26.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.048+0000 7ff7f5400640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff7d006bc00 0x7ff7d006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:26.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.049+0000 7ff7ed600640 1 -- 172.21.15.102:0/1162495824 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff7e80578e0 con 0x7ff7f0158420 2024-03-18T16:35:26.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.049+0000 7ff7f5400640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff7d006bc00 0x7ff7d006e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4007fd0 tx=0x7ff7e4007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:26.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.059+0000 7ff7ed600640 1 -- 172.21.15.102:0/1162495824 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff7e8020070 con 0x7ff7f0158420 2024-03-18T16:35:26.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.320+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7f00b7f00 con 0x7ff7f0158420 2024-03-18T16:35:26.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.340+0000 7ff7ed600640 1 -- 172.21.15.102:0/1162495824 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff7e8059840 con 0x7ff7f0158420 2024-03-18T16:35:26.343 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:26.343 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:26.343 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff7d006bc00 msgr2=0x7ff7d006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff7d006bc00 0x7ff7d006e0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4007fd0 tx=0x7ff7e4007e80 comp rx=0 tx=0).stop 2024-03-18T16:35:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 msgr2=0x7ff7f0155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f0155a00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800c7b0 tx=0x7ff7e800cc70 comp rx=0 tx=0).stop 2024-03-18T16:35:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 shutdown_connections 2024-03-18T16:35:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff7d006bc00 0x7ff7d006e0c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7f0158420 0x7ff7f0155a00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7f0141650 0x7ff7f0152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:26.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 --2- 172.21.15.102:0/1162495824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7f00c4fa0 0x7ff7f01f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:26.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.344+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 >> 172.21.15.102:0/1162495824 conn(0x7ff7f00cb890 msgr2=0x7ff7f0157640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:26.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.345+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 shutdown_connections 2024-03-18T16:35:26.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:26.345+0000 7ff7f6800640 1 -- 172.21.15.102:0/1162495824 wait complete. 2024-03-18T16:35:26.975 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:26.975 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:27.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:28.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.150+0000 7fd977200640 1 Processor -- start 2024-03-18T16:35:28.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.150+0000 7fd977200640 1 -- start start 2024-03-18T16:35:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.152+0000 7fd977200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.152+0000 7fd977200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd97815c3a0 0x7fd9780c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.152+0000 7fd977200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9780c4fa0 0x7fd9780c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.152+0000 7fd977200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd978159370 con 0x7fd9780c4fa0 2024-03-18T16:35:28.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.152+0000 7fd977200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd978158ad0 con 0x7fd97815c3a0 2024-03-18T16:35:28.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.152+0000 7fd977200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd978158db0 con 0x7fd978141650 2024-03-18T16:35:28.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.152+0000 7fd975e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:28.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.152+0000 7fd975400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd97815c3a0 0x7fd9780c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:28.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.153+0000 7fd975e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:60560/0 (socket says 172.21.15.102:60560) 2024-03-18T16:35:28.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.153+0000 7fd975e00640 1 -- 172.21.15.102:0/3817091624 learned_addr learned my addr 172.21.15.102:0/3817091624 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:28.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.153+0000 7fd976800640 1 --2- 172.21.15.102:0/3817091624 >> v2:172.21.15.102:3300/0 conn(0x7fd9780c4fa0 0x7fd9780c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.155+0000 7fd975e00640 1 -- 172.21.15.102:0/3817091624 >> v2:172.21.15.154:3300/0 conn(0x7fd97815c3a0 msgr2=0x7fd9780c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.155+0000 7fd975e00640 1 --2- 172.21.15.102:0/3817091624 >> v2:172.21.15.154:3300/0 conn(0x7fd97815c3a0 0x7fd9780c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.155+0000 7fd975e00640 1 -- 172.21.15.102:0/3817091624 >> v2:172.21.15.102:3300/0 conn(0x7fd9780c4fa0 msgr2=0x7fd9780c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.155+0000 7fd975e00640 1 --2- 172.21.15.102:0/3817091624 >> v2:172.21.15.102:3300/0 conn(0x7fd9780c4fa0 0x7fd9780c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.155+0000 7fd975e00640 1 -- 172.21.15.102:0/3817091624 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd978159090 con 0x7fd978141650 2024-03-18T16:35:28.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.158+0000 7fd975e00640 1 --2- 172.21.15.102:0/3817091624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978158420 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd97806e640 tx=0x7fd96802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8631c0d440108e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.162+0000 7fd974a00640 1 -- 172.21.15.102:0/3817091624 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968035020 con 0x7fd978141650 2024-03-18T16:35:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.162+0000 7fd974a00640 1 -- 172.21.15.102:0/3817091624 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd968031870 con 0x7fd978141650 2024-03-18T16:35:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.162+0000 7fd974a00640 1 -- 172.21.15.102:0/3817091624 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968031c40 con 0x7fd978141650 2024-03-18T16:35:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.163+0000 7fd977200640 1 -- 172.21.15.102:0/3817091624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 msgr2=0x7fd978158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.163+0000 7fd977200640 1 --2- 172.21.15.102:0/3817091624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978158420 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd97806e640 tx=0x7fd96802b910 comp rx=0 tx=0).stop 2024-03-18T16:35:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.163+0000 7fd977200640 1 -- 172.21.15.102:0/3817091624 shutdown_connections 2024-03-18T16:35:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.163+0000 7fd977200640 1 --2- 172.21.15.102:0/3817091624 >> v2:172.21.15.102:3300/0 conn(0x7fd9780c4fa0 0x7fd9780c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.163+0000 7fd977200640 1 --2- 172.21.15.102:0/3817091624 >> v2:172.21.15.154:3300/0 conn(0x7fd97815c3a0 0x7fd9780c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.163+0000 7fd977200640 1 --2- 172.21.15.102:0/3817091624 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978158420 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.163+0000 7fd977200640 1 -- 172.21.15.102:0/3817091624 >> 172.21.15.102:0/3817091624 conn(0x7fd9780cb890 msgr2=0x7fd9780cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.163+0000 7fd977200640 1 -- 172.21.15.102:0/3817091624 shutdown_connections 2024-03-18T16:35:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.163+0000 7fd977200640 1 -- 172.21.15.102:0/3817091624 wait complete. 2024-03-18T16:35:28.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.166+0000 7fd977200640 1 Processor -- start 2024-03-18T16:35:28.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.166+0000 7fd977200640 1 -- start start 2024-03-18T16:35:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.167+0000 7fd977200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9780c4fa0 0x7fd9781eef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.168+0000 7fd977200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.168+0000 7fd977200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd97815c3a0 0x7fd9780ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.168+0000 7fd977200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd97815c7c0 con 0x7fd97815c3a0 2024-03-18T16:35:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.168+0000 7fd977200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9780c66a0 con 0x7fd9780c4fa0 2024-03-18T16:35:28.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.168+0000 7fd977200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd978141a30 con 0x7fd978141650 2024-03-18T16:35:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.168+0000 7fd975400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.168+0000 7fd975e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9780c4fa0 0x7fd9781eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.168+0000 7fd975400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978153000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:60566/0 (socket says 172.21.15.102:60566) 2024-03-18T16:35:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.168+0000 7fd975400640 1 -- 172.21.15.102:0/3010935496 learned_addr learned my addr 172.21.15.102:0/3010935496 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.169+0000 7fd976800640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd97815c3a0 0x7fd9780ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.172+0000 7fd975400640 1 -- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9780c4fa0 msgr2=0x7fd9781eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.172+0000 7fd975400640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9780c4fa0 0x7fd9781eef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.172+0000 7fd975400640 1 -- 172.21.15.102:0/3010935496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd97815c3a0 msgr2=0x7fd9780ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:28.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.172+0000 7fd975400640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd97815c3a0 0x7fd9780ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.172+0000 7fd975400640 1 -- 172.21.15.102:0/3010935496 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9780b8260 con 0x7fd978141650 2024-03-18T16:35:28.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.174+0000 7fd975400640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978153000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd96400e8a0 tx=0x7fd96400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.177+0000 7fd96e000640 1 -- 172.21.15.102:0/3010935496 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964017070 con 0x7fd978141650 2024-03-18T16:35:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.177+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9781539b0 con 0x7fd978141650 2024-03-18T16:35:28.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.177+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9780b8500 con 0x7fd978141650 2024-03-18T16:35:28.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.178+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd978152590 con 0x7fd978141650 2024-03-18T16:35:28.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.385+0000 7fd96e000640 1 -- 172.21.15.102:0/3010935496 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd964005990 con 0x7fd978141650 2024-03-18T16:35:28.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.443+0000 7fd96e000640 1 -- 172.21.15.102:0/3010935496 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964021060 con 0x7fd978141650 2024-03-18T16:35:28.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.447+0000 7fd96e000640 1 -- 172.21.15.102:0/3010935496 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd96401a070 con 0x7fd978141650 2024-03-18T16:35:28.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.449+0000 7fd96e000640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd95806bbd0 0x7fd95806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:28.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.449+0000 7fd96e000640 1 -- 172.21.15.102:0/3010935496 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd96408b570 con 0x7fd978141650 2024-03-18T16:35:28.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.449+0000 7fd975e00640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd95806bbd0 0x7fd95806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:28.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.450+0000 7fd975e00640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd95806bbd0 0x7fd95806e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd968036000 tx=0x7fd96803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:28.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.458+0000 7fd96e000640 1 -- 172.21.15.102:0/3010935496 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd978152590 con 0x7fd978141650 2024-03-18T16:35:28.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.725+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9780c92a0 con 0x7fd978141650 2024-03-18T16:35:28.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.756+0000 7fd96e000640 1 -- 172.21.15.102:0/3010935496 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd9780c92a0 con 0x7fd978141650 2024-03-18T16:35:28.758 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:28.759 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:28.759 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd95806bbd0 msgr2=0x7fd95806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd95806bbd0 0x7fd95806e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd968036000 tx=0x7fd96803c040 comp rx=0 tx=0).stop 2024-03-18T16:35:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 msgr2=0x7fd978153000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978153000 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd96400e8a0 tx=0x7fd96400ed60 comp rx=0 tx=0).stop 2024-03-18T16:35:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 shutdown_connections 2024-03-18T16:35:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd95806bbd0 0x7fd95806e090 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd97815c3a0 0x7fd9780ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd978141650 0x7fd978153000 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 --2- 172.21.15.102:0/3010935496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9780c4fa0 0x7fd9781eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:28.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 >> 172.21.15.102:0/3010935496 conn(0x7fd9780cb890 msgr2=0x7fd97813f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:28.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 shutdown_connections 2024-03-18T16:35:28.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:28.758+0000 7fd977200640 1 -- 172.21.15.102:0/3010935496 wait complete. 2024-03-18T16:35:29.381 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:29.381 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:30.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:30.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.556+0000 7f1ac1400640 1 Processor -- start 2024-03-18T16:35:30.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.556+0000 7f1ac1400640 1 -- start start 2024-03-18T16:35:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.557+0000 7f1ac1400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1abc0cfe70 0x7f1abc15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:30.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.557+0000 7f1ac1400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1abc0d0250 0x7f1abc0c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:30.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.557+0000 7f1ac1400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1abc0cabd0 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.557+0000 7f1ac1400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1abc0caea0 con 0x7f1abc0d0250 2024-03-18T16:35:30.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.557+0000 7f1ac1400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1abc0cb180 con 0x7f1abc0cfe70 2024-03-18T16:35:30.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.558+0000 7f1abbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1abc0cfe70 0x7f1abc15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:30.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.558+0000 7f1abb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1abc0d0250 0x7f1abc0c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.558+0000 7f1abb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1abc0d0250 0x7f1abc0c65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34454/0 (socket says 172.21.15.102:34454) 2024-03-18T16:35:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.558+0000 7f1abb400640 1 -- 172.21.15.102:0/3726107008 learned_addr learned my addr 172.21.15.102:0/3726107008 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:30.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.558+0000 7f1abbe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1abc0cfe70 0x7f1abc15ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42638/0 (socket says 172.21.15.102:42638) 2024-03-18T16:35:30.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.561+0000 7f1abbe00640 1 -- 172.21.15.102:0/3726107008 >> v2:172.21.15.102:3300/0 conn(0x7f1abc0d0250 msgr2=0x7f1abc0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:30.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.561+0000 7f1abbe00640 1 --2- 172.21.15.102:0/3726107008 >> v2:172.21.15.102:3300/0 conn(0x7f1abc0d0250 0x7f1abc0c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:30.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.561+0000 7f1abbe00640 1 -- 172.21.15.102:0/3726107008 >> v1:172.21.15.154:6789/0 conn(0x7f1abc0cc2f0 legacy=0x7f1abc0cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:30.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.561+0000 7f1abbe00640 1 -- 172.21.15.102:0/3726107008 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1abc0cb460 con 0x7f1abc0cfe70 2024-03-18T16:35:30.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.566+0000 7f1abbe00640 1 --2- 172.21.15.102:0/3726107008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cfe70 0x7f1abc15ae40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0cd800 tx=0x7f1aac02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4b55914890403fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:30.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.568+0000 7f1abaa00640 1 -- 172.21.15.102:0/3726107008 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac034070 con 0x7f1abc0cfe70 2024-03-18T16:35:30.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.570+0000 7f1abaa00640 1 -- 172.21.15.102:0/3726107008 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aac030610 con 0x7f1abc0cfe70 2024-03-18T16:35:30.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.570+0000 7f1abaa00640 1 -- 172.21.15.102:0/3726107008 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac0309e0 con 0x7f1abc0cfe70 2024-03-18T16:35:30.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.571+0000 7f1ac1400640 1 -- 172.21.15.102:0/3726107008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cfe70 msgr2=0x7f1abc15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:30.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.571+0000 7f1ac1400640 1 --2- 172.21.15.102:0/3726107008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cfe70 0x7f1abc15ae40 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0cd800 tx=0x7f1aac02b960 comp rx=0 tx=0).stop 2024-03-18T16:35:30.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.571+0000 7f1ac1400640 1 -- 172.21.15.102:0/3726107008 shutdown_connections 2024-03-18T16:35:30.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.571+0000 7f1ac1400640 1 --2- 172.21.15.102:0/3726107008 >> v2:172.21.15.102:3300/0 conn(0x7f1abc0d0250 0x7f1abc0c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:30.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.571+0000 7f1ac1400640 1 --2- 172.21.15.102:0/3726107008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cfe70 0x7f1abc15ae40 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:30.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.571+0000 7f1ac1400640 1 -- 172.21.15.102:0/3726107008 >> 172.21.15.102:0/3726107008 conn(0x7f1abc08c3e0 msgr2=0x7f1abc06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:30.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.571+0000 7f1ac1400640 1 -- 172.21.15.102:0/3726107008 shutdown_connections 2024-03-18T16:35:30.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.572+0000 7f1ac1400640 1 -- 172.21.15.102:0/3726107008 wait complete. 2024-03-18T16:35:30.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.573+0000 7f1ac1400640 1 Processor -- start 2024-03-18T16:35:30.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.573+0000 7f1ac1400640 1 -- start start 2024-03-18T16:35:30.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.574+0000 7f1ac1400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cc2f0 0x7f1abc1f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:30.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.574+0000 7f1ac1400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1abc0cfe70 0x7f1abc1f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:30.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.574+0000 7f1ac1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc0d0250 0x7f1abc1e0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:30.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.574+0000 7f1ac1400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc0cbd00 con 0x7f1abc0d0250 2024-03-18T16:35:30.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.574+0000 7f1ac1400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1abc0cba20 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.574+0000 7f1ac1400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1abc0c6b20 con 0x7f1abc0cfe70 2024-03-18T16:35:30.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.575+0000 7f1abbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cc2f0 0x7f1abc1f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:30.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.575+0000 7f1ac0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc0d0250 0x7f1abc1e0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:30.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.575+0000 7f1abb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1abc0cfe70 0x7f1abc1f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:30.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.575+0000 7f1abbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cc2f0 0x7f1abc1f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:42646/0 (socket says 172.21.15.102:42646) 2024-03-18T16:35:30.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.575+0000 7f1abbe00640 1 -- 172.21.15.102:0/2723371494 learned_addr learned my addr 172.21.15.102:0/2723371494 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:30.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.579+0000 7f1abbe00640 1 -- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1abc0cfe70 msgr2=0x7f1abc1f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:30.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.579+0000 7f1abbe00640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1abc0cfe70 0x7f1abc1f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:30.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.579+0000 7f1abbe00640 1 -- 172.21.15.102:0/2723371494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc0d0250 msgr2=0x7f1abc1e0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:30.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.579+0000 7f1abbe00640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc0d0250 0x7f1abc1e0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:30.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.579+0000 7f1abbe00640 1 -- 172.21.15.102:0/2723371494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1abc153170 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.582+0000 7f1abbe00640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cc2f0 0x7f1abc1f1550 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009890 tx=0x7f1aac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:30.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.586+0000 7f1ab3e00640 1 -- 172.21.15.102:0/2723371494 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac034040 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.586+0000 7f1ab3e00640 1 -- 172.21.15.102:0/2723371494 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aac030410 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.586+0000 7f1ab3e00640 1 -- 172.21.15.102:0/2723371494 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aac039c50 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.586+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1abc153450 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.586+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1abc1f0d50 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.587+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1abc074fd0 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.606+0000 7f1ab3e00640 1 -- 172.21.15.102:0/2723371494 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1aac037070 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.607+0000 7f1ab3e00640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a9c06bc20 0x7f1a9c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:30.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.608+0000 7f1abb400640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a9c06bc20 0x7f1a9c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:30.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.609+0000 7f1abb400640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a9c06bc20 0x7f1a9c06e0e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0d0660 tx=0x7f1aa8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:30.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.610+0000 7f1ab3e00640 1 -- 172.21.15.102:0/2723371494 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1aac0305c0 con 0x7f1abc0cc2f0 2024-03-18T16:35:30.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:30.869+0000 7f1ab3e00640 1 -- 172.21.15.102:0/2723371494 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1aac07b850 con 0x7f1abc0cc2f0 2024-03-18T16:35:31.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.129+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1abc0cdda0 con 0x7f1abc0cc2f0 2024-03-18T16:35:31.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.143+0000 7f1ab3e00640 1 -- 172.21.15.102:0/2723371494 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1aac07f8f0 con 0x7f1abc0cc2f0 2024-03-18T16:35:31.144 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:31.145 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:31.145 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a9c06bc20 msgr2=0x7f1a9c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a9c06bc20 0x7f1a9c06e0e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0d0660 tx=0x7f1aa8009240 comp rx=0 tx=0).stop 2024-03-18T16:35:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cc2f0 msgr2=0x7f1abc1f1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cc2f0 0x7f1abc1f1550 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009890 tx=0x7f1aac0046c0 comp rx=0 tx=0).stop 2024-03-18T16:35:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 shutdown_connections 2024-03-18T16:35:31.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1a9c06bc20 0x7f1a9c06e0e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1abc0d0250 0x7f1abc1e0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1abc0cfe70 0x7f1abc1f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 --2- 172.21.15.102:0/2723371494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1abc0cc2f0 0x7f1abc1f1550 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 >> 172.21.15.102:0/2723371494 conn(0x7f1abc08c3e0 msgr2=0x7f1abc0c81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.145+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 shutdown_connections 2024-03-18T16:35:31.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:31.146+0000 7f1ac1400640 1 -- 172.21.15.102:0/2723371494 wait complete. 2024-03-18T16:35:31.767 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:31.767 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:32.768 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:32.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.936+0000 7fb16ce00640 1 Processor -- start 2024-03-18T16:35:32.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.936+0000 7fb16ce00640 1 -- start start 2024-03-18T16:35:32.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.938+0000 7fb16ce00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb168158c00 0x7fb16806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:32.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.938+0000 7fb16ce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb16806fd80 0x7fb168072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:32.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.938+0000 7fb16ce00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1680c4190 con 0x7fb168071700 2024-03-18T16:35:32.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.938+0000 7fb16ce00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb168158fe0 con 0x7fb168158c00 2024-03-18T16:35:32.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.938+0000 7fb16ce00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb16806f3b0 con 0x7fb16806fd80 2024-03-18T16:35:32.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.938+0000 7fb166a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb16806fd80 0x7fb168072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:32.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.939+0000 7fb167400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb168158c00 0x7fb16806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:32.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.939+0000 7fb166a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb16806fd80 0x7fb168072a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48506/0 (socket says 172.21.15.102:48506) 2024-03-18T16:35:32.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.939+0000 7fb166a00640 1 -- 172.21.15.102:0/572705523 learned_addr learned my addr 172.21.15.102:0/572705523 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:32.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.942+0000 7fb166a00640 1 -- 172.21.15.102:0/572705523 >> v2:172.21.15.154:3300/0 conn(0x7fb168158c00 msgr2=0x7fb16806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:32.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.942+0000 7fb166a00640 1 --2- 172.21.15.102:0/572705523 >> v2:172.21.15.154:3300/0 conn(0x7fb168158c00 0x7fb16806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:32.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.942+0000 7fb166a00640 1 -- 172.21.15.102:0/572705523 >> v1:172.21.15.154:6789/0 conn(0x7fb168071700 legacy=0x7fb16815e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:32.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.942+0000 7fb166a00640 1 -- 172.21.15.102:0/572705523 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16806f060 con 0x7fb16806fd80 2024-03-18T16:35:32.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.944+0000 7fb166a00640 1 --2- 172.21.15.102:0/572705523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb16806fd80 0x7fb168072a70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb154009870 tx=0x7fb15402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=805c296743d5fbfb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:32.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.948+0000 7fb166000640 1 -- 172.21.15.102:0/572705523 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb154034070 con 0x7fb16806fd80 2024-03-18T16:35:32.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.948+0000 7fb166000640 1 -- 172.21.15.102:0/572705523 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb154030840 con 0x7fb16806fd80 2024-03-18T16:35:32.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.948+0000 7fb166000640 1 -- 172.21.15.102:0/572705523 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb154030c10 con 0x7fb16806fd80 2024-03-18T16:35:32.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.949+0000 7fb16ce00640 1 -- 172.21.15.102:0/572705523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb16806fd80 msgr2=0x7fb168072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:32.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.949+0000 7fb16ce00640 1 --2- 172.21.15.102:0/572705523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb16806fd80 0x7fb168072a70 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb154009870 tx=0x7fb15402b980 comp rx=0 tx=0).stop 2024-03-18T16:35:32.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.949+0000 7fb16ce00640 1 -- 172.21.15.102:0/572705523 shutdown_connections 2024-03-18T16:35:32.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.949+0000 7fb16ce00640 1 --2- 172.21.15.102:0/572705523 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb16806fd80 0x7fb168072a70 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:32.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.949+0000 7fb16ce00640 1 --2- 172.21.15.102:0/572705523 >> v2:172.21.15.154:3300/0 conn(0x7fb168158c00 0x7fb16806f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:32.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.949+0000 7fb16ce00640 1 -- 172.21.15.102:0/572705523 >> 172.21.15.102:0/572705523 conn(0x7fb1680cb890 msgr2=0x7fb1680cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:32.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.949+0000 7fb16ce00640 1 -- 172.21.15.102:0/572705523 shutdown_connections 2024-03-18T16:35:32.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.949+0000 7fb16ce00640 1 -- 172.21.15.102:0/572705523 wait complete. 2024-03-18T16:35:32.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.951+0000 7fb16ce00640 1 Processor -- start 2024-03-18T16:35:32.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.952+0000 7fb16ce00640 1 -- start start 2024-03-18T16:35:32.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.953+0000 7fb16ce00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16806fd80 0x7fb1681f8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:32.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.953+0000 7fb16ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb168071700 0x7fb168153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:32.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.953+0000 7fb16ce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb168158c00 0x7fb16815e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:32.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.953+0000 7fb16ce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb168072270 con 0x7fb168071700 2024-03-18T16:35:32.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.953+0000 7fb16ce00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb168071f90 con 0x7fb16806fd80 2024-03-18T16:35:32.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.953+0000 7fb16ce00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb168072550 con 0x7fb168158c00 2024-03-18T16:35:32.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.954+0000 7fb167e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb168158c00 0x7fb16815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:32.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.954+0000 7fb167400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16806fd80 0x7fb1681f8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:32.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.954+0000 7fb166a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb168071700 0x7fb168153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:32.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.954+0000 7fb167e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb168158c00 0x7fb16815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48508/0 (socket says 172.21.15.102:48508) 2024-03-18T16:35:32.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.954+0000 7fb167e00640 1 -- 172.21.15.102:0/3545099308 learned_addr learned my addr 172.21.15.102:0/3545099308 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:32.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.958+0000 7fb167e00640 1 -- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16806fd80 msgr2=0x7fb1681f8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:32.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.958+0000 7fb167e00640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16806fd80 0x7fb1681f8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:32.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.958+0000 7fb167e00640 1 -- 172.21.15.102:0/3545099308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb168071700 msgr2=0x7fb168153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:32.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.958+0000 7fb167e00640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb168071700 0x7fb168153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:32.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.958+0000 7fb167e00640 1 -- 172.21.15.102:0/3545099308 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1681621c0 con 0x7fb168158c00 2024-03-18T16:35:32.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.961+0000 7fb167e00640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb168158c00 0x7fb16815e430 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb16000c7d0 tx=0x7fb16000cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:32.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.965+0000 7fb15f400640 1 -- 172.21.15.102:0/3545099308 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160003b70 con 0x7fb168158c00 2024-03-18T16:35:32.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.965+0000 7fb15f400640 1 -- 172.21.15.102:0/3545099308 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb160003d30 con 0x7fb168158c00 2024-03-18T16:35:32.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.965+0000 7fb15f400640 1 -- 172.21.15.102:0/3545099308 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16000aa90 con 0x7fb168158c00 2024-03-18T16:35:32.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.965+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1681624a0 con 0x7fb168158c00 2024-03-18T16:35:32.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.965+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16815dc30 con 0x7fb168158c00 2024-03-18T16:35:32.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:32.967+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb168074fd0 con 0x7fb168158c00 2024-03-18T16:35:33.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.232+0000 7fb15f400640 1 -- 172.21.15.102:0/3545099308 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb16000ac30 con 0x7fb168158c00 2024-03-18T16:35:33.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.234+0000 7fb15f400640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb14806bc20 0x7fb14806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:33.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.235+0000 7fb167400640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb14806bc20 0x7fb14806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:33.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.235+0000 7fb15f400640 1 -- 172.21.15.102:0/3545099308 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb160057830 con 0x7fb168158c00 2024-03-18T16:35:33.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.236+0000 7fb167400640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb14806bc20 0x7fb14806e0e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb158009b40 tx=0x7fb158009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:33.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.254+0000 7fb15f400640 1 -- 172.21.15.102:0/3545099308 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb160059790 con 0x7fb168158c00 2024-03-18T16:35:33.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.490+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1680c5c00 con 0x7fb168158c00 2024-03-18T16:35:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.579+0000 7fb15f400640 1 -- 172.21.15.102:0/3545099308 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb16005d830 con 0x7fb168158c00 2024-03-18T16:35:33.581 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:33.581 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:33.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.582+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb14806bc20 msgr2=0x7fb14806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:33.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.582+0000 7fb16ce00640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb14806bc20 0x7fb14806e0e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb158009b40 tx=0x7fb158009190 comp rx=0 tx=0).stop 2024-03-18T16:35:33.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.582+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb168158c00 msgr2=0x7fb16815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.582+0000 7fb16ce00640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb168158c00 0x7fb16815e430 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb16000c7d0 tx=0x7fb16000cc90 comp rx=0 tx=0).stop 2024-03-18T16:35:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.583+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 shutdown_connections 2024-03-18T16:35:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.583+0000 7fb16ce00640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb14806bc20 0x7fb14806e0e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.583+0000 7fb16ce00640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb168158c00 0x7fb16815e430 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.583+0000 7fb16ce00640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb168071700 0x7fb168153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.583+0000 7fb16ce00640 1 --2- 172.21.15.102:0/3545099308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16806fd80 0x7fb1681f8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.583+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 >> 172.21.15.102:0/3545099308 conn(0x7fb1680cb890 msgr2=0x7fb1680cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.583+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 shutdown_connections 2024-03-18T16:35:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:33.583+0000 7fb16ce00640 1 -- 172.21.15.102:0/3545099308 wait complete. 2024-03-18T16:35:34.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:34.212 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:35.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:35.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.382+0000 7ff26d600640 1 Processor -- start 2024-03-18T16:35:35.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.382+0000 7ff26d600640 1 -- start start 2024-03-18T16:35:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.383+0000 7ff26d600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff2680cca00 0x7ff26815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.383+0000 7ff26d600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2680cabd0 con 0x7ff2680cfe70 2024-03-18T16:35:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.383+0000 7ff26d600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2680caea0 con 0x7ff2680d0250 2024-03-18T16:35:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.383+0000 7ff26d600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff2680cb180 con 0x7ff2680cca00 2024-03-18T16:35:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.384+0000 7ff267e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff2680cfe70 0x7ff2680c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51318/0 (socket says 172.21.15.102:51318) 2024-03-18T16:35:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.384+0000 7ff267e00640 1 -- 172.21.15.102:0/2072743794 learned_addr learned my addr 172.21.15.102:0/2072743794 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.385+0000 7ff26cc00640 1 --2- 172.21.15.102:0/2072743794 >> v2:172.21.15.154:3300/0 conn(0x7ff2680cca00 0x7ff26815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.388+0000 7ff266a00640 1 -- 172.21.15.102:0/2072743794 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2297587557 0 0) 0x7ff2680cabd0 con 0x7ff2680cfe70 2024-03-18T16:35:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.388+0000 7ff266a00640 1 -- 172.21.15.102:0/2072743794 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff24c003520 con 0x7ff2680cfe70 2024-03-18T16:35:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.390+0000 7ff26cc00640 1 -- 172.21.15.102:0/2072743794 >> v1:172.21.15.154:6789/0 conn(0x7ff2680d0250 legacy=0x7ff2680cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.390+0000 7ff26cc00640 1 -- 172.21.15.102:0/2072743794 >> v1:172.21.15.102:6789/0 conn(0x7ff2680cfe70 legacy=0x7ff2680c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:35.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.390+0000 7ff26cc00640 1 -- 172.21.15.102:0/2072743794 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2680cb460 con 0x7ff2680cca00 2024-03-18T16:35:35.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.393+0000 7ff26cc00640 1 --2- 172.21.15.102:0/2072743794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2680cca00 0x7ff26815ae40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff260009d90 tx=0x7ff26002bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac7feffdd4cd178e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:35.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.397+0000 7ff266a00640 1 -- 172.21.15.102:0/2072743794 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260039070 con 0x7ff2680cca00 2024-03-18T16:35:35.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.397+0000 7ff266a00640 1 -- 172.21.15.102:0/2072743794 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff26002cd80 con 0x7ff2680cca00 2024-03-18T16:35:35.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.397+0000 7ff266a00640 1 -- 172.21.15.102:0/2072743794 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260034660 con 0x7ff2680cca00 2024-03-18T16:35:35.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.398+0000 7ff26d600640 1 -- 172.21.15.102:0/2072743794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2680cca00 msgr2=0x7ff26815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.398+0000 7ff26d600640 1 --2- 172.21.15.102:0/2072743794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2680cca00 0x7ff26815ae40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff260009d90 tx=0x7ff26002bfd0 comp rx=0 tx=0).stop 2024-03-18T16:35:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.398+0000 7ff26d600640 1 -- 172.21.15.102:0/2072743794 shutdown_connections 2024-03-18T16:35:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.398+0000 7ff26d600640 1 --2- 172.21.15.102:0/2072743794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2680cca00 0x7ff26815ae40 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.398+0000 7ff26d600640 1 -- 172.21.15.102:0/2072743794 >> 172.21.15.102:0/2072743794 conn(0x7ff26808c3e0 msgr2=0x7ff26806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.398+0000 7ff26d600640 1 -- 172.21.15.102:0/2072743794 shutdown_connections 2024-03-18T16:35:35.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.398+0000 7ff26d600640 1 -- 172.21.15.102:0/2072743794 wait complete. 2024-03-18T16:35:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.401+0000 7ff26d600640 1 Processor -- start 2024-03-18T16:35:35.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.401+0000 7ff26d600640 1 -- start start 2024-03-18T16:35:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.402+0000 7ff26d600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2680cca00 0x7ff2681e5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.402+0000 7ff26d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2680cfe70 0x7ff268159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.402+0000 7ff26d600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2680d0250 0x7ff26814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.402+0000 7ff26d600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2680cbd00 con 0x7ff2680cfe70 2024-03-18T16:35:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.402+0000 7ff26d600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2680cba20 con 0x7ff2680d0250 2024-03-18T16:35:35.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.402+0000 7ff26d600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff2680c8330 con 0x7ff2680cca00 2024-03-18T16:35:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.403+0000 7ff267e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2680cca00 0x7ff2681e5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.403+0000 7ff26cc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2680d0250 0x7ff26814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.403+0000 7ff267400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2680cfe70 0x7ff268159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.403+0000 7ff267e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2680cca00 0x7ff2681e5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48510/0 (socket says 172.21.15.102:48510) 2024-03-18T16:35:35.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.403+0000 7ff267e00640 1 -- 172.21.15.102:0/1679149542 learned_addr learned my addr 172.21.15.102:0/1679149542 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.407+0000 7ff267e00640 1 -- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2680d0250 msgr2=0x7ff26814f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.407+0000 7ff267e00640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2680d0250 0x7ff26814f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.407+0000 7ff267e00640 1 -- 172.21.15.102:0/1679149542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2680cfe70 msgr2=0x7ff268159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:35.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.407+0000 7ff267e00640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2680cfe70 0x7ff268159e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:35.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.407+0000 7ff267e00640 1 -- 172.21.15.102:0/1679149542 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2681e1100 con 0x7ff2680cca00 2024-03-18T16:35:35.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.409+0000 7ff267e00640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2680cca00 0x7ff2681e5510 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff25800c530 tx=0x7ff25800c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:35.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.413+0000 7ff25fe00640 1 -- 172.21.15.102:0/1679149542 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258017070 con 0x7ff2680cca00 2024-03-18T16:35:35.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.413+0000 7ff25fe00640 1 -- 172.21.15.102:0/1679149542 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff258005ce0 con 0x7ff2680cca00 2024-03-18T16:35:35.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.413+0000 7ff25fe00640 1 -- 172.21.15.102:0/1679149542 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2580072b0 con 0x7ff2680cca00 2024-03-18T16:35:35.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.413+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff268159070 con 0x7ff2680cca00 2024-03-18T16:35:35.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.413+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff268159630 con 0x7ff2680cca00 2024-03-18T16:35:35.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.415+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff268074fd0 con 0x7ff2680cca00 2024-03-18T16:35:35.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.712+0000 7ff25fe00640 1 -- 172.21.15.102:0/1679149542 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff25801a070 con 0x7ff2680cca00 2024-03-18T16:35:35.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.713+0000 7ff25fe00640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24806bc20 0x7ff24806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:35.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.714+0000 7ff267400640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24806bc20 0x7ff24806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:35.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.715+0000 7ff267400640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24806bc20 0x7ff24806e0e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff2680c7720 tx=0x7ff254009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:35.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.716+0000 7ff25fe00640 1 -- 172.21.15.102:0/1679149542 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff258056900 con 0x7ff2680cca00 2024-03-18T16:35:35.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.730+0000 7ff25fe00640 1 -- 172.21.15.102:0/1679149542 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff258021020 con 0x7ff2680cca00 2024-03-18T16:35:35.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.966+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2680c87a0 con 0x7ff2680cca00 2024-03-18T16:35:35.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.979+0000 7ff25fe00640 1 -- 172.21.15.102:0/1679149542 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff258058860 con 0x7ff2680cca00 2024-03-18T16:35:35.981 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:35.981 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24806bc20 msgr2=0x7ff24806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24806bc20 0x7ff24806e0e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff2680c7720 tx=0x7ff254009040 comp rx=0 tx=0).stop 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2680cca00 msgr2=0x7ff2681e5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2680cca00 0x7ff2681e5510 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff25800c530 tx=0x7ff25800c9f0 comp rx=0 tx=0).stop 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 shutdown_connections 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff24806bc20 0x7ff24806e0e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2680d0250 0x7ff26814f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2680cfe70 0x7ff268159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 --2- 172.21.15.102:0/1679149542 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2680cca00 0x7ff2681e5510 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 >> 172.21.15.102:0/1679149542 conn(0x7ff26808c3e0 msgr2=0x7ff2680c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 shutdown_connections 2024-03-18T16:35:35.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:35.983+0000 7ff26d600640 1 -- 172.21.15.102:0/1679149542 wait complete. 2024-03-18T16:35:36.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:36.615 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:37.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.806+0000 7ff866400640 1 Processor -- start 2024-03-18T16:35:37.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.807+0000 7ff866400640 1 -- start start 2024-03-18T16:35:37.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.809+0000 7ff866400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff860154920 0x7ff8600cd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:37.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.809+0000 7ff866400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff86015af20 0x7ff8600cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:37.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.809+0000 7ff866400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8600c78a0 con 0x7ff860070f70 2024-03-18T16:35:37.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.809+0000 7ff866400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff8600c7b80 con 0x7ff86015af20 2024-03-18T16:35:37.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.809+0000 7ff866400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff8600c7e60 con 0x7ff860154920 2024-03-18T16:35:37.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.810+0000 7ff85f400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff860154920 0x7ff8600cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:37.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.810+0000 7ff85f400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff860154920 0x7ff8600cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:44862/0 (socket says 172.21.15.102:44862) 2024-03-18T16:35:37.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.810+0000 7ff85f400640 1 -- 172.21.15.102:0/1841182935 learned_addr learned my addr 172.21.15.102:0/1841182935 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:37.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.813+0000 7ff85f400640 1 -- 172.21.15.102:0/1841182935 >> v2:172.21.15.102:3300/0 conn(0x7ff86015af20 msgr2=0x7ff8600cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:37.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.813+0000 7ff85f400640 1 --2- 172.21.15.102:0/1841182935 >> v2:172.21.15.102:3300/0 conn(0x7ff86015af20 0x7ff8600cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:37.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.813+0000 7ff85f400640 1 -- 172.21.15.102:0/1841182935 >> v1:172.21.15.102:6789/0 conn(0x7ff860070f70 legacy=0x7ff8600c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:37.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.813+0000 7ff85f400640 1 -- 172.21.15.102:0/1841182935 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8600c8140 con 0x7ff860154920 2024-03-18T16:35:37.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.815+0000 7ff85f400640 1 --2- 172.21.15.102:0/1841182935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff860154920 0x7ff8600cd280 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff8600cddc0 tx=0x7ff85002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=899c094e1fdff33e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:37.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.819+0000 7ff85e000640 1 -- 172.21.15.102:0/1841182935 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850035020 con 0x7ff860154920 2024-03-18T16:35:37.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.819+0000 7ff85e000640 1 -- 172.21.15.102:0/1841182935 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff850031870 con 0x7ff860154920 2024-03-18T16:35:37.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.819+0000 7ff85e000640 1 -- 172.21.15.102:0/1841182935 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850031c40 con 0x7ff860154920 2024-03-18T16:35:37.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.820+0000 7ff866400640 1 -- 172.21.15.102:0/1841182935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff860154920 msgr2=0x7ff8600cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:37.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.820+0000 7ff866400640 1 --2- 172.21.15.102:0/1841182935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff860154920 0x7ff8600cd280 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff8600cddc0 tx=0x7ff85002b910 comp rx=0 tx=0).stop 2024-03-18T16:35:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.821+0000 7ff866400640 1 -- 172.21.15.102:0/1841182935 shutdown_connections 2024-03-18T16:35:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.821+0000 7ff866400640 1 --2- 172.21.15.102:0/1841182935 >> v2:172.21.15.102:3300/0 conn(0x7ff86015af20 0x7ff8600cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.821+0000 7ff866400640 1 --2- 172.21.15.102:0/1841182935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff860154920 0x7ff8600cd280 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.821+0000 7ff866400640 1 -- 172.21.15.102:0/1841182935 >> 172.21.15.102:0/1841182935 conn(0x7ff860072e30 msgr2=0x7ff860073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.821+0000 7ff866400640 1 -- 172.21.15.102:0/1841182935 shutdown_connections 2024-03-18T16:35:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.821+0000 7ff866400640 1 -- 172.21.15.102:0/1841182935 wait complete. 2024-03-18T16:35:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.823+0000 7ff866400640 1 Processor -- start 2024-03-18T16:35:37.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.824+0000 7ff866400640 1 -- start start 2024-03-18T16:35:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.825+0000 7ff866400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff860070f70 0x7ff86014a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.825+0000 7ff866400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff860154920 0x7ff860158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.825+0000 7ff866400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff86015af20 0x7ff8600cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.825+0000 7ff866400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff8600c9ed0 con 0x7ff860154920 2024-03-18T16:35:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.825+0000 7ff866400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff86015b3c0 con 0x7ff86015af20 2024-03-18T16:35:37.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.825+0000 7ff866400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff8600ca050 con 0x7ff860070f70 2024-03-18T16:35:37.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.826+0000 7ff85f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff860070f70 0x7ff86014a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:37.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.826+0000 7ff85fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff86015af20 0x7ff8600cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:37.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.826+0000 7ff85f400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff860070f70 0x7ff86014a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48524/0 (socket says 172.21.15.102:48524) 2024-03-18T16:35:37.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.826+0000 7ff85f400640 1 -- 172.21.15.102:0/616413678 learned_addr learned my addr 172.21.15.102:0/616413678 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:37.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.830+0000 7ff85f400640 1 -- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff86015af20 msgr2=0x7ff8600cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:37.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.830+0000 7ff85f400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff86015af20 0x7ff8600cb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:37.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.830+0000 7ff85f400640 1 -- 172.21.15.102:0/616413678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff860154920 msgr2=0x7ff860158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:37.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.830+0000 7ff85f400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff860154920 0x7ff860158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:37.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.830+0000 7ff85f400640 1 -- 172.21.15.102:0/616413678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86014b3c0 con 0x7ff860070f70 2024-03-18T16:35:37.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.830+0000 7ff85fe00640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff86015af20 0x7ff8600cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:35:37.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.833+0000 7ff85f400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff860070f70 0x7ff86014a510 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff85002b950 tx=0x7ff8500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:37.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.837+0000 7ff857400640 1 -- 172.21.15.102:0/616413678 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85003ad60 con 0x7ff860070f70 2024-03-18T16:35:37.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.837+0000 7ff857400640 1 -- 172.21.15.102:0/616413678 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8500443f0 con 0x7ff860070f70 2024-03-18T16:35:37.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.837+0000 7ff857400640 1 -- 172.21.15.102:0/616413678 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8500447e0 con 0x7ff860070f70 2024-03-18T16:35:37.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.837+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8601597d0 con 0x7ff860070f70 2024-03-18T16:35:37.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.837+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff86014ae00 con 0x7ff860070f70 2024-03-18T16:35:37.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.838+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff860074fd0 con 0x7ff860070f70 2024-03-18T16:35:37.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.869+0000 7ff857400640 1 -- 172.21.15.102:0/616413678 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff850038020 con 0x7ff860070f70 2024-03-18T16:35:37.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.871+0000 7ff857400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff84006bc20 0x7ff84006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:37.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.871+0000 7ff85ea00640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff84006bc20 0x7ff84006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:37.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:37.872+0000 7ff85ea00640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff84006bc20 0x7ff84006e0e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff86006f040 tx=0x7ff84c006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:38.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.113+0000 7ff857400640 1 -- 172.21.15.102:0/616413678 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff850079940 con 0x7ff860070f70 2024-03-18T16:35:38.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.123+0000 7ff857400640 1 -- 172.21.15.102:0/616413678 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff85003f070 con 0x7ff860070f70 2024-03-18T16:35:38.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.366+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff860070640 con 0x7ff860070f70 2024-03-18T16:35:38.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.380+0000 7ff857400640 1 -- 172.21.15.102:0/616413678 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff85007b8a0 con 0x7ff860070f70 2024-03-18T16:35:38.381 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:38.381 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:38.382 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff84006bc20 msgr2=0x7ff84006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff84006bc20 0x7ff84006e0e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff86006f040 tx=0x7ff84c006c90 comp rx=0 tx=0).stop 2024-03-18T16:35:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff860070f70 msgr2=0x7ff86014a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff860070f70 0x7ff86014a510 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff85002b950 tx=0x7ff8500046c0 comp rx=0 tx=0).stop 2024-03-18T16:35:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 shutdown_connections 2024-03-18T16:35:38.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff84006bc20 0x7ff84006e0e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff86015af20 0x7ff8600cb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff860154920 0x7ff860158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 --2- 172.21.15.102:0/616413678 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff860070f70 0x7ff86014a510 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.382+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 >> 172.21.15.102:0/616413678 conn(0x7ff860072e30 msgr2=0x7ff8600cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.383+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 shutdown_connections 2024-03-18T16:35:38.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:38.383+0000 7ff866400640 1 -- 172.21.15.102:0/616413678 wait complete. 2024-03-18T16:35:39.004 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:39.004 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:40.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:40.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.180+0000 7f796ca00640 1 Processor -- start 2024-03-18T16:35:40.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.180+0000 7f796ca00640 1 -- start start 2024-03-18T16:35:40.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.181+0000 7f796ca00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f796815a310 0x7f796806d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:40.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.181+0000 7f796ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f796815a6f0 0x7f796815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:40.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.181+0000 7f796ca00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79680ca500 con 0x7f796815baa0 2024-03-18T16:35:40.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.181+0000 7f796ca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f79680cd410 con 0x7f796815a6f0 2024-03-18T16:35:40.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.181+0000 7f796ca00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f796815b080 con 0x7f796815a310 2024-03-18T16:35:40.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.182+0000 7f7966800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f796815a310 0x7f796806d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:40.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.182+0000 7f7965e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f796815a6f0 0x7f796815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:40.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.182+0000 7f7966800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f796815a310 0x7f796806d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:44894/0 (socket says 172.21.15.102:44894) 2024-03-18T16:35:40.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.182+0000 7f7966800640 1 -- 172.21.15.102:0/698183204 learned_addr learned my addr 172.21.15.102:0/698183204 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:40.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.185+0000 7f7966800640 1 -- 172.21.15.102:0/698183204 >> v2:172.21.15.102:3300/0 conn(0x7f796815a6f0 msgr2=0x7f796815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:40.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.185+0000 7f7966800640 1 --2- 172.21.15.102:0/698183204 >> v2:172.21.15.102:3300/0 conn(0x7f796815a6f0 0x7f796815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.185+0000 7f7966800640 1 -- 172.21.15.102:0/698183204 >> v1:172.21.15.102:6789/0 conn(0x7f796815baa0 legacy=0x7f796815bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:40.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.185+0000 7f7966800640 1 -- 172.21.15.102:0/698183204 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79680ca7a0 con 0x7f796815a310 2024-03-18T16:35:40.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.187+0000 7f7966800640 1 --2- 172.21.15.102:0/698183204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f796815a310 0x7f796806d3e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7968067db0 tx=0x7f795802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c958d3ea28215ea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:40.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.191+0000 7f7965400640 1 -- 172.21.15.102:0/698183204 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958034070 con 0x7f796815a310 2024-03-18T16:35:40.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.191+0000 7f7965400640 1 -- 172.21.15.102:0/698183204 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7958030860 con 0x7f796815a310 2024-03-18T16:35:40.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.191+0000 7f7965400640 1 -- 172.21.15.102:0/698183204 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958030c30 con 0x7f796815a310 2024-03-18T16:35:40.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.192+0000 7f796ca00640 1 -- 172.21.15.102:0/698183204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f796815a310 msgr2=0x7f796806d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:40.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.192+0000 7f796ca00640 1 --2- 172.21.15.102:0/698183204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f796815a310 0x7f796806d3e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7968067db0 tx=0x7f795802b9a0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.192+0000 7f796ca00640 1 -- 172.21.15.102:0/698183204 shutdown_connections 2024-03-18T16:35:40.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.192+0000 7f796ca00640 1 --2- 172.21.15.102:0/698183204 >> v2:172.21.15.102:3300/0 conn(0x7f796815a6f0 0x7f796815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.192+0000 7f796ca00640 1 --2- 172.21.15.102:0/698183204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f796815a310 0x7f796806d3e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.192+0000 7f796ca00640 1 -- 172.21.15.102:0/698183204 >> 172.21.15.102:0/698183204 conn(0x7f796808c1a0 msgr2=0x7f7968066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:40.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.192+0000 7f796ca00640 1 -- 172.21.15.102:0/698183204 shutdown_connections 2024-03-18T16:35:40.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.193+0000 7f796ca00640 1 -- 172.21.15.102:0/698183204 wait complete. 2024-03-18T16:35:40.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.195+0000 7f796ca00640 1 Processor -- start 2024-03-18T16:35:40.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.196+0000 7f796ca00640 1 -- start start 2024-03-18T16:35:40.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.197+0000 7f796ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f796815a310 0x7f79681f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.197+0000 7f796ca00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f796815a6f0 0x7f796814dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.197+0000 7f796ca00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f796815baa0 0x7f796814e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.197+0000 7f796ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f79680cd130 con 0x7f796815a310 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.197+0000 7f796ca00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79680cce50 con 0x7f796815baa0 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.197+0000 7f796ca00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f796806d920 con 0x7f796815a6f0 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.198+0000 7f7965e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f796815a6f0 0x7f796814dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.198+0000 7f7967200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f796815baa0 0x7f796814e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.198+0000 7f7966800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f796815a310 0x7f79681f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.198+0000 7f7965e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f796815a6f0 0x7f796814dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:48526/0 (socket says 172.21.15.102:48526) 2024-03-18T16:35:40.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.198+0000 7f7965e00640 1 -- 172.21.15.102:0/2215245361 learned_addr learned my addr 172.21.15.102:0/2215245361 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:40.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.201+0000 7f7965e00640 1 -- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f796815baa0 msgr2=0x7f796814e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:40.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.201+0000 7f7965e00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f796815baa0 0x7f796814e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.202+0000 7f7965e00640 1 -- 172.21.15.102:0/2215245361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f796815a310 msgr2=0x7f79681f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:40.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.202+0000 7f7965e00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f796815a310 0x7f79681f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.202+0000 7f7965e00640 1 -- 172.21.15.102:0/2215245361 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79681f43c0 con 0x7f796815a6f0 2024-03-18T16:35:40.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.204+0000 7f7965e00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f796815a6f0 0x7f796814dde0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f795400e9e0 tx=0x7f795400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:40.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.206+0000 7f795ea00640 1 -- 172.21.15.102:0/2215245361 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954017070 con 0x7f796815a6f0 2024-03-18T16:35:40.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.206+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79681582b0 con 0x7f796815a6f0 2024-03-18T16:35:40.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.206+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7968158ef0 con 0x7f796815a6f0 2024-03-18T16:35:40.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.207+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7968074fd0 con 0x7f796815a6f0 2024-03-18T16:35:40.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.465+0000 7f795ea00640 1 -- 172.21.15.102:0/2215245361 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7954005990 con 0x7f796815a6f0 2024-03-18T16:35:40.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.465+0000 7f795ea00640 1 -- 172.21.15.102:0/2215245361 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954021060 con 0x7f796815a6f0 2024-03-18T16:35:40.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.467+0000 7f795ea00640 1 -- 172.21.15.102:0/2215245361 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f795401a070 con 0x7f796815a6f0 2024-03-18T16:35:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.469+0000 7f795ea00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f794806bc00 0x7f794806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:40.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.469+0000 7f7966800640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f794806bc00 0x7f794806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:40.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.470+0000 7f7966800640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f794806bc00 0x7f794806e0c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7958036000 tx=0x7f7958008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:40.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.474+0000 7f795ea00640 1 -- 172.21.15.102:0/2215245361 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f79540576d0 con 0x7f796815a6f0 2024-03-18T16:35:40.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.493+0000 7f795ea00640 1 -- 172.21.15.102:0/2215245361 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7954059630 con 0x7f796815a6f0 2024-03-18T16:35:40.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.735+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f796806df20 con 0x7f796815a6f0 2024-03-18T16:35:40.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.750+0000 7f795ea00640 1 -- 172.21.15.102:0/2215245361 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f795405d6d0 con 0x7f796815a6f0 2024-03-18T16:35:40.751 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:40.751 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:40.752 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:40.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f794806bc00 msgr2=0x7f794806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:40.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f794806bc00 0x7f794806e0c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7958036000 tx=0x7f7958008030 comp rx=0 tx=0).stop 2024-03-18T16:35:40.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f796815a6f0 msgr2=0x7f796814dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:40.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f796815a6f0 0x7f796814dde0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f795400e9e0 tx=0x7f795400eea0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 shutdown_connections 2024-03-18T16:35:40.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f794806bc00 0x7f794806e0c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f796815baa0 0x7f796814e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f796815a6f0 0x7f796814dde0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 --2- 172.21.15.102:0/2215245361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f796815a310 0x7f79681f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:40.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.753+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 >> 172.21.15.102:0/2215245361 conn(0x7f796808c1a0 msgr2=0x7f796814ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:40.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.754+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 shutdown_connections 2024-03-18T16:35:40.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:40.754+0000 7f796ca00640 1 -- 172.21.15.102:0/2215245361 wait complete. 2024-03-18T16:35:41.383 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:41.383 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:42.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:42.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.562+0000 7fd6d2200640 1 Processor -- start 2024-03-18T16:35:42.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.562+0000 7fd6d2200640 1 -- start start 2024-03-18T16:35:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.564+0000 7fd6d2200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd6cc0ca960 0x7fd6cc06c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.564+0000 7fd6d2200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd6cc15a1a0 0x7fd6cc0705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.564+0000 7fd6d2200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6cc06c9c0 con 0x7fd6cc0cad40 2024-03-18T16:35:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.564+0000 7fd6d2200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd6cc0c8680 con 0x7fd6cc0ca960 2024-03-18T16:35:42.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.564+0000 7fd6d2200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd6cc06cca0 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.565+0000 7fd6d1800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd6cc15a1a0 0x7fd6cc0705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:42.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.565+0000 7fd6cbe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd6cc0cad40 0x7fd6cc15e130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51032/0 (socket says 172.21.15.102:51032) 2024-03-18T16:35:42.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.565+0000 7fd6cbe00640 1 -- 172.21.15.102:0/369700542 learned_addr learned my addr 172.21.15.102:0/369700542 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:42.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.565+0000 7fd6d0e00640 1 --2- 172.21.15.102:0/369700542 >> v2:172.21.15.102:3300/0 conn(0x7fd6cc0ca960 0x7fd6cc06c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:42.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.569+0000 7fd6d1800640 1 -- 172.21.15.102:0/369700542 >> v2:172.21.15.102:3300/0 conn(0x7fd6cc0ca960 msgr2=0x7fd6cc06c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:42.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.569+0000 7fd6d1800640 1 --2- 172.21.15.102:0/369700542 >> v2:172.21.15.102:3300/0 conn(0x7fd6cc0ca960 0x7fd6cc06c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:42.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.569+0000 7fd6d1800640 1 -- 172.21.15.102:0/369700542 >> v1:172.21.15.102:6789/0 conn(0x7fd6cc0cad40 legacy=0x7fd6cc15e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:42.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.569+0000 7fd6d1800640 1 -- 172.21.15.102:0/369700542 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc06cf80 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.572+0000 7fd6d1800640 1 --2- 172.21.15.102:0/369700542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6cc15a1a0 0x7fd6cc0705a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0b2f30 tx=0x7fd6c402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=273a1fbe950c541a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:42.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.576+0000 7fd6cb400640 1 -- 172.21.15.102:0/369700542 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4039070 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.576+0000 7fd6cb400640 1 -- 172.21.15.102:0/369700542 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c402cd10 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.576+0000 7fd6cb400640 1 -- 172.21.15.102:0/369700542 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4034660 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.577+0000 7fd6d2200640 1 -- 172.21.15.102:0/369700542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6cc15a1a0 msgr2=0x7fd6cc0705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.577+0000 7fd6d2200640 1 --2- 172.21.15.102:0/369700542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6cc15a1a0 0x7fd6cc0705a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0b2f30 tx=0x7fd6c402bf60 comp rx=0 tx=0).stop 2024-03-18T16:35:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.577+0000 7fd6d2200640 1 -- 172.21.15.102:0/369700542 shutdown_connections 2024-03-18T16:35:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.577+0000 7fd6d2200640 1 --2- 172.21.15.102:0/369700542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6cc15a1a0 0x7fd6cc0705a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.577+0000 7fd6d2200640 1 --2- 172.21.15.102:0/369700542 >> v2:172.21.15.102:3300/0 conn(0x7fd6cc0ca960 0x7fd6cc06c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.577+0000 7fd6d2200640 1 -- 172.21.15.102:0/369700542 >> 172.21.15.102:0/369700542 conn(0x7fd6cc067190 msgr2=0x7fd6cc0675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.577+0000 7fd6d2200640 1 -- 172.21.15.102:0/369700542 shutdown_connections 2024-03-18T16:35:42.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.577+0000 7fd6d2200640 1 -- 172.21.15.102:0/369700542 wait complete. 2024-03-18T16:35:42.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.580+0000 7fd6d2200640 1 Processor -- start 2024-03-18T16:35:42.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.580+0000 7fd6d2200640 1 -- start start 2024-03-18T16:35:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.581+0000 7fd6d2200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6cc0ca960 0x7fd6cc1e4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.581+0000 7fd6d2200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6cc0cad40 0x7fd6cc1421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.581+0000 7fd6d2200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6cc15a1a0 0x7fd6cc142730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.581+0000 7fd6d2200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc070ae0 con 0x7fd6cc0cad40 2024-03-18T16:35:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.581+0000 7fd6d2200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc15dc10 con 0x7fd6cc0ca960 2024-03-18T16:35:42.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.581+0000 7fd6d2200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd6cc070c60 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.582+0000 7fd6d1800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6cc15a1a0 0x7fd6cc142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.582+0000 7fd6d0e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6cc0ca960 0x7fd6cc1e4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.582+0000 7fd6cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6cc0cad40 0x7fd6cc1421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.582+0000 7fd6d1800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6cc15a1a0 0x7fd6cc142730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53908/0 (socket says 172.21.15.102:53908) 2024-03-18T16:35:42.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.582+0000 7fd6d1800640 1 -- 172.21.15.102:0/3907364937 learned_addr learned my addr 172.21.15.102:0/3907364937 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.586+0000 7fd6d1800640 1 -- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6cc0ca960 msgr2=0x7fd6cc1e4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.586+0000 7fd6d1800640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6cc0ca960 0x7fd6cc1e4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.586+0000 7fd6d1800640 1 -- 172.21.15.102:0/3907364937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6cc0cad40 msgr2=0x7fd6cc1421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.586+0000 7fd6d1800640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6cc0cad40 0x7fd6cc1421f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:42.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.586+0000 7fd6d1800640 1 -- 172.21.15.102:0/3907364937 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc155470 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.588+0000 7fd6d1800640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6cc15a1a0 0x7fd6cc142730 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009e50 tx=0x7fd6c4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:42.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.591+0000 7fd6c8c00640 1 -- 172.21.15.102:0/3907364937 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4044020 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.591+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6cc155750 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.591+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6cc155d10 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.592+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6cc074fd0 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.802+0000 7fd6c8c00640 1 -- 172.21.15.102:0/3907364937 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c402cd10 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.843+0000 7fd6c8c00640 1 -- 172.21.15.102:0/3907364937 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4039040 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.850+0000 7fd6c8c00640 1 -- 172.21.15.102:0/3907364937 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd6c4047020 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.851+0000 7fd6c8c00640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd6ac06bbd0 0x7fd6ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:42.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.851+0000 7fd6c8c00640 1 -- 172.21.15.102:0/3907364937 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd6c40adc80 con 0x7fd6cc15a1a0 2024-03-18T16:35:42.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.852+0000 7fd6d0e00640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd6ac06bbd0 0x7fd6ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:42.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.852+0000 7fd6d0e00640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd6ac06bbd0 0x7fd6ac06e090 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009b40 tx=0x7fd6c0009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:42.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:42.862+0000 7fd6c8c00640 1 -- 172.21.15.102:0/3907364937 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd6c404e070 con 0x7fd6cc15a1a0 2024-03-18T16:35:43.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.216+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6cc1e2e50 con 0x7fd6cc15a1a0 2024-03-18T16:35:43.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.229+0000 7fd6c8c00640 1 -- 172.21.15.102:0/3907364937 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd6c407bc30 con 0x7fd6cc15a1a0 2024-03-18T16:35:43.230 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:43.230 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:43.231 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd6ac06bbd0 msgr2=0x7fd6ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd6ac06bbd0 0x7fd6ac06e090 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009b40 tx=0x7fd6c0009190 comp rx=0 tx=0).stop 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6cc15a1a0 msgr2=0x7fd6cc142730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6cc15a1a0 0x7fd6cc142730 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009e50 tx=0x7fd6c4004a40 comp rx=0 tx=0).stop 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 shutdown_connections 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd6ac06bbd0 0x7fd6ac06e090 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd6cc15a1a0 0x7fd6cc142730 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6cc0cad40 0x7fd6cc1421f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 --2- 172.21.15.102:0/3907364937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6cc0ca960 0x7fd6cc1e4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 >> 172.21.15.102:0/3907364937 conn(0x7fd6cc067190 msgr2=0x7fd6cc158a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:43.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 shutdown_connections 2024-03-18T16:35:43.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:43.231+0000 7fd6d2200640 1 -- 172.21.15.102:0/3907364937 wait complete. 2024-03-18T16:35:43.854 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:43.854 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:44.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.033+0000 7f8cc8a00640 1 Processor -- start 2024-03-18T16:35:45.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.033+0000 7f8cc8a00640 1 -- start start 2024-03-18T16:35:45.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.035+0000 7f8cc8a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8cc40c9a20 0x7f8cc40cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.035+0000 7f8cc8a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 0x7f8cc40d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.035+0000 7f8cc8a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cc4154a90 con 0x7f8cc406d450 2024-03-18T16:35:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.035+0000 7f8cc8a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f8cc415e270 con 0x7f8cc40c9a20 2024-03-18T16:35:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.035+0000 7f8cc8a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc415e530 con 0x7f8cc415aff0 2024-03-18T16:35:45.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.036+0000 7f8cc3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 0x7f8cc40d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:45.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.038+0000 7f8cc2a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f8cc406d450 0x7f8cc415a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:35830/0 (socket says 172.21.15.102:35830) 2024-03-18T16:35:45.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.038+0000 7f8cc2a00640 1 -- 172.21.15.102:0/143881543 learned_addr learned my addr 172.21.15.102:0/143881543 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:45.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.082+0000 7f8cc3400640 1 --2- 172.21.15.102:0/143881543 >> v2:172.21.15.102:3300/0 conn(0x7f8cc40c9a20 0x7f8cc40cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.086+0000 7f8cc3400640 1 -- 172.21.15.102:0/143881543 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 msgr2=0x7f8cc40d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.086+0000 7f8cc3400640 1 --2- 172.21.15.102:0/143881543 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 0x7f8cc40d0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.086+0000 7f8cc3400640 1 -- 172.21.15.102:0/143881543 >> v1:172.21.15.154:6789/0 conn(0x7f8cc406d450 legacy=0x7f8cc415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:45.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.086+0000 7f8cc3400640 1 -- 172.21.15.102:0/143881543 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc40ca610 con 0x7f8cc40c9a20 2024-03-18T16:35:45.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.092+0000 7f8cc3400640 1 --2- 172.21.15.102:0/143881543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40c9a20 0x7f8cc40cfe00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40097a0 tx=0x7f8cb402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c3cedf49773591 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:45.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.100+0000 7f8cc2000640 1 -- 172.21.15.102:0/143881543 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4034070 con 0x7f8cc40c9a20 2024-03-18T16:35:45.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.100+0000 7f8cc2000640 1 -- 172.21.15.102:0/143881543 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cb4030610 con 0x7f8cc40c9a20 2024-03-18T16:35:45.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.100+0000 7f8cc2000640 1 -- 172.21.15.102:0/143881543 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb40309e0 con 0x7f8cc40c9a20 2024-03-18T16:35:45.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.101+0000 7f8cc8a00640 1 -- 172.21.15.102:0/143881543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40c9a20 msgr2=0x7f8cc40cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:45.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.101+0000 7f8cc8a00640 1 --2- 172.21.15.102:0/143881543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40c9a20 0x7f8cc40cfe00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8cb40097a0 tx=0x7f8cb402b960 comp rx=0 tx=0).stop 2024-03-18T16:35:45.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.102+0000 7f8cc8a00640 1 -- 172.21.15.102:0/143881543 shutdown_connections 2024-03-18T16:35:45.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.102+0000 7f8cc8a00640 1 --2- 172.21.15.102:0/143881543 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 0x7f8cc40d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.102+0000 7f8cc8a00640 1 --2- 172.21.15.102:0/143881543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40c9a20 0x7f8cc40cfe00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.102+0000 7f8cc8a00640 1 -- 172.21.15.102:0/143881543 >> 172.21.15.102:0/143881543 conn(0x7f8cc4067160 msgr2=0x7f8cc4067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:45.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.102+0000 7f8cc8a00640 1 -- 172.21.15.102:0/143881543 shutdown_connections 2024-03-18T16:35:45.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.102+0000 7f8cc8a00640 1 -- 172.21.15.102:0/143881543 wait complete. 2024-03-18T16:35:45.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.104+0000 7f8cc8a00640 1 Processor -- start 2024-03-18T16:35:45.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.105+0000 7f8cc8a00640 1 -- start start 2024-03-18T16:35:45.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.106+0000 7f8cc8a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc406d450 0x7f8cc41e6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:45.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.106+0000 7f8cc8a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40c9a20 0x7f8cc41e40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.106+0000 7f8cc8a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 0x7f8cc41f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.106+0000 7f8cc8a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc40cad50 con 0x7f8cc40c9a20 2024-03-18T16:35:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.106+0000 7f8cc8a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc40cabd0 con 0x7f8cc406d450 2024-03-18T16:35:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.106+0000 7f8cc8a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc406e1d0 con 0x7f8cc415aff0 2024-03-18T16:35:45.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.107+0000 7f8cc3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 0x7f8cc41f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.107+0000 7f8cc3400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc406d450 0x7f8cc41e6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.107+0000 7f8cc3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 0x7f8cc41f05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53918/0 (socket says 172.21.15.102:53918) 2024-03-18T16:35:45.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.107+0000 7f8cc3e00640 1 -- 172.21.15.102:0/2459347425 learned_addr learned my addr 172.21.15.102:0/2459347425 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.111+0000 7f8cc3400640 1 -- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 msgr2=0x7f8cc41f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.111+0000 7f8cc3400640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 0x7f8cc41f05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.111+0000 7f8cc3400640 1 -- 172.21.15.102:0/2459347425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40c9a20 msgr2=0x7f8cc41e40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.111+0000 7f8cc3400640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40c9a20 0x7f8cc41e40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.111+0000 7f8cc3400640 1 -- 172.21.15.102:0/2459347425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc414c040 con 0x7f8cc406d450 2024-03-18T16:35:45.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.114+0000 7f8cc3400640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc406d450 0x7f8cc41e6000 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4033000 tx=0x7f8cb40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.118+0000 7f8cbb400640 1 -- 172.21.15.102:0/2459347425 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4047070 con 0x7f8cc406d450 2024-03-18T16:35:45.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.118+0000 7f8cbb400640 1 -- 172.21.15.102:0/2459347425 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cb4039a10 con 0x7f8cc406d450 2024-03-18T16:35:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.118+0000 7f8cbb400640 1 -- 172.21.15.102:0/2459347425 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4034040 con 0x7f8cc406d450 2024-03-18T16:35:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.118+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc41f0b30 con 0x7f8cc406d450 2024-03-18T16:35:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.118+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc414c5c0 con 0x7f8cc406d450 2024-03-18T16:35:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.120+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc414c800 con 0x7f8cc406d450 2024-03-18T16:35:45.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.147+0000 7f8cbb400640 1 -- 172.21.15.102:0/2459347425 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8cb404a070 con 0x7f8cc406d450 2024-03-18T16:35:45.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.147+0000 7f8cbb400640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca406bc20 0x7f8ca406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:45.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.148+0000 7f8cc2a00640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca406bc20 0x7f8ca406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:45.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.149+0000 7f8cc2a00640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca406bc20 0x7f8ca406e0e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0000f60 tx=0x7f8cb0007a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:45.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.389+0000 7f8cbb400640 1 -- 172.21.15.102:0/2459347425 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f8cb4078ef0 con 0x7f8cc406d450 2024-03-18T16:35:45.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.400+0000 7f8cbb400640 1 -- 172.21.15.102:0/2459347425 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8cc414c800 con 0x7f8cc406d450 2024-03-18T16:35:45.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.706+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cc4074fd0 con 0x7f8cc406d450 2024-03-18T16:35:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.719+0000 7f8cbb400640 1 -- 172.21.15.102:0/2459347425 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f8cb4051020 con 0x7f8cc406d450 2024-03-18T16:35:45.721 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:45.721 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca406bc20 msgr2=0x7f8ca406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca406bc20 0x7f8ca406e0e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0000f60 tx=0x7f8cb0007a80 comp rx=0 tx=0).stop 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc406d450 msgr2=0x7f8cc41e6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc406d450 0x7f8cc41e6000 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4033000 tx=0x7f8cb40046c0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 shutdown_connections 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f8ca406bc20 0x7f8ca406e0e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f8cc415aff0 0x7f8cc41f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8cc40c9a20 0x7f8cc41e40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 --2- 172.21.15.102:0/2459347425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8cc406d450 0x7f8cc41e6000 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 >> 172.21.15.102:0/2459347425 conn(0x7f8cc4067160 msgr2=0x7f8cc4153e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:45.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 shutdown_connections 2024-03-18T16:35:45.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:45.722+0000 7f8cc8a00640 1 -- 172.21.15.102:0/2459347425 wait complete. 2024-03-18T16:35:46.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:46.344 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:47.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:47.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.515+0000 7ff2d1600640 1 Processor -- start 2024-03-18T16:35:47.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.515+0000 7ff2d1600640 1 -- start start 2024-03-18T16:35:47.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.517+0000 7ff2d1600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff2cc141650 0x7ff2cc158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:47.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.517+0000 7ff2d1600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2cc0c72d0 con 0x7ff2cc15c3a0 2024-03-18T16:35:47.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.517+0000 7ff2d1600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2cc158ad0 con 0x7ff2cc0c4fa0 2024-03-18T16:35:47.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.517+0000 7ff2d1600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff2cc159370 con 0x7ff2cc141650 2024-03-18T16:35:47.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.518+0000 7ff2cb400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff2cc15c3a0 0x7ff2cc0c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:51040/0 (socket says 172.21.15.102:51040) 2024-03-18T16:35:47.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.518+0000 7ff2cb400640 1 -- 172.21.15.102:0/508421063 learned_addr learned my addr 172.21.15.102:0/508421063 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:35:47.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.521+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 381163452 0 0) 0x7ff2cc0c72d0 con 0x7ff2cc15c3a0 2024-03-18T16:35:47.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.521+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2b0003520 con 0x7ff2cc15c3a0 2024-03-18T16:35:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 782549955 0 0) 0x7ff2b0003520 con 0x7ff2cc15c3a0 2024-03-18T16:35:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 >> v2:172.21.15.154:3300/0 conn(0x7ff2cc141650 msgr2=0x7ff2cc158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 --2- 172.21.15.102:0/508421063 >> v2:172.21.15.154:3300/0 conn(0x7ff2cc141650 0x7ff2cc158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 >> v1:172.21.15.154:6789/0 conn(0x7ff2cc0c4fa0 legacy=0x7ff2cc162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc158db0 con 0x7ff2cc15c3a0 2024-03-18T16:35:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2344234600 0 0) 0x7ff2b8001670 con 0x7ff2cc15c3a0 2024-03-18T16:35:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 >> v1:172.21.15.102:6789/0 conn(0x7ff2cc15c3a0 legacy=0x7ff2cc0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2b0003a00 0x7ff2b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0004800 0x7ff2b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.525+0000 7ff2caa00640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2b00052e0 0x7ff2b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.526+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc159370 con 0x7ff2b00052e0 2024-03-18T16:35:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.526+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b000c230 con 0x7ff2b0004800 2024-03-18T16:35:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.526+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b000c430 con 0x7ff2b0003a00 2024-03-18T16:35:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.526+0000 7ff2cbe00640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2b0003a00 0x7ff2b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:47.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.530+0000 7ff2d0c00640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2b00052e0 0x7ff2b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:47.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.530+0000 7ff2cbe00640 1 -- 172.21.15.102:0/508421063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0004800 msgr2=0x7ff2b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:47.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.530+0000 7ff2cbe00640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0004800 0x7ff2b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:47.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.530+0000 7ff2cbe00640 1 -- 172.21.15.102:0/508421063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2b00052e0 msgr2=0x7ff2b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:47.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.530+0000 7ff2cbe00640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2b00052e0 0x7ff2b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:47.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.530+0000 7ff2cbe00640 1 -- 172.21.15.102:0/508421063 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b0003520 con 0x7ff2b0003a00 2024-03-18T16:35:47.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.530+0000 7ff2d0c00640 1 -- 172.21.15.102:0/508421063 reap_dead start 2024-03-18T16:35:47.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.533+0000 7ff2cbe00640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2b0003a00 0x7ff2b0003e00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc006fd0 tx=0x7ff2b0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c2c387405ad7525 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:47.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.536+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc037070 con 0x7ff2b0003a00 2024-03-18T16:35:47.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.537+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2bc032620 con 0x7ff2b0003a00 2024-03-18T16:35:47.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.537+0000 7ff2caa00640 1 -- 172.21.15.102:0/508421063 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc009460 con 0x7ff2b0003a00 2024-03-18T16:35:47.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.538+0000 7ff2d1600640 1 -- 172.21.15.102:0/508421063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2b0003a00 msgr2=0x7ff2b0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:47.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.538+0000 7ff2d1600640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2b0003a00 0x7ff2b0003e00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc006fd0 tx=0x7ff2b0005830 comp rx=0 tx=0).stop 2024-03-18T16:35:47.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.538+0000 7ff2d1600640 1 -- 172.21.15.102:0/508421063 shutdown_connections 2024-03-18T16:35:47.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.538+0000 7ff2d1600640 1 --2- 172.21.15.102:0/508421063 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2b0003a00 0x7ff2b0003e00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:47.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.538+0000 7ff2d1600640 1 -- 172.21.15.102:0/508421063 >> 172.21.15.102:0/508421063 conn(0x7ff2cc0cb890 msgr2=0x7ff2cc0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:47.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.538+0000 7ff2d1600640 1 -- 172.21.15.102:0/508421063 shutdown_connections 2024-03-18T16:35:47.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.538+0000 7ff2d1600640 1 -- 172.21.15.102:0/508421063 wait complete. 2024-03-18T16:35:47.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.540+0000 7ff2d1600640 1 Processor -- start 2024-03-18T16:35:47.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.540+0000 7ff2d1600640 1 -- start start 2024-03-18T16:35:47.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.541+0000 7ff2d1600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0003a00 0x7ff2cc1eca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:47.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.541+0000 7ff2d1600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2cc13e340 0x7ff2cc0b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:47.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.541+0000 7ff2d1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2cc13e780 0x7ff2cc13d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:47.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.541+0000 7ff2d1600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc0c66a0 con 0x7ff2cc13e780 2024-03-18T16:35:47.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.541+0000 7ff2d1600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc0c63c0 con 0x7ff2b0003a00 2024-03-18T16:35:47.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.541+0000 7ff2d1600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff2cc15c7c0 con 0x7ff2cc13e340 2024-03-18T16:35:47.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.541+0000 7ff2cbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0003a00 0x7ff2cc1eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:47.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.541+0000 7ff2cb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2cc13e340 0x7ff2cc0b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:47.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.542+0000 7ff2cbe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0003a00 0x7ff2cc1eca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57454/0 (socket says 172.21.15.102:57454) 2024-03-18T16:35:47.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.542+0000 7ff2cb400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2cc13e340 0x7ff2cc0b8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53946/0 (socket says 172.21.15.102:53946) 2024-03-18T16:35:47.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.542+0000 7ff2cbe00640 1 -- 172.21.15.102:0/3307944684 learned_addr learned my addr 172.21.15.102:0/3307944684 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:47.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.545+0000 7ff2cbe00640 1 -- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2cc13e340 msgr2=0x7ff2cc0b8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:47.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.545+0000 7ff2cbe00640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2cc13e340 0x7ff2cc0b8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:47.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.545+0000 7ff2cbe00640 1 -- 172.21.15.102:0/3307944684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2cc13e780 msgr2=0x7ff2cc13d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:47.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.545+0000 7ff2cbe00640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2cc13e780 0x7ff2cc13d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:47.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.545+0000 7ff2cbe00640 1 -- 172.21.15.102:0/3307944684 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc0b95a0 con 0x7ff2b0003a00 2024-03-18T16:35:47.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.549+0000 7ff2cbe00640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0003a00 0x7ff2cc1eca40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc000c00 tx=0x7ff2bc0047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:47.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.553+0000 7ff2c3e00640 1 -- 172.21.15.102:0/3307944684 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc042070 con 0x7ff2b0003a00 2024-03-18T16:35:47.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.553+0000 7ff2c3e00640 1 -- 172.21.15.102:0/3307944684 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2bc03e680 con 0x7ff2b0003a00 2024-03-18T16:35:47.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.553+0000 7ff2c3e00640 1 -- 172.21.15.102:0/3307944684 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc037040 con 0x7ff2b0003a00 2024-03-18T16:35:47.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.553+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2cc1ed540 con 0x7ff2b0003a00 2024-03-18T16:35:47.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.553+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2cc1edb00 con 0x7ff2b0003a00 2024-03-18T16:35:47.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.554+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2cc072cc0 con 0x7ff2b0003a00 2024-03-18T16:35:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.574+0000 7ff2c3e00640 1 -- 172.21.15.102:0/3307944684 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff2bc045070 con 0x7ff2b0003a00 2024-03-18T16:35:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.576+0000 7ff2c3e00640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ac06bc50 0x7ff2ac06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.576+0000 7ff2cb400640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ac06bc50 0x7ff2ac06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:47.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.577+0000 7ff2cb400640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ac06bc50 0x7ff2ac06e110 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc1e9bf0 tx=0x7ff2b80019f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:47.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.816+0000 7ff2c3e00640 1 -- 172.21.15.102:0/3307944684 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff2bc079110 con 0x7ff2b0003a00 2024-03-18T16:35:47.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:47.829+0000 7ff2c3e00640 1 -- 172.21.15.102:0/3307944684 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff2cc072cc0 con 0x7ff2b0003a00 2024-03-18T16:35:48.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.062+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2cc0c5f80 con 0x7ff2b0003a00 2024-03-18T16:35:48.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.078+0000 7ff2c3e00640 1 -- 172.21.15.102:0/3307944684 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff2cc0c5f80 con 0x7ff2b0003a00 2024-03-18T16:35:48.080 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:48.080 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:48.080 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:48.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.080+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ac06bc50 msgr2=0x7ff2ac06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:48.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.080+0000 7ff2d1600640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ac06bc50 0x7ff2ac06e110 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc1e9bf0 tx=0x7ff2b80019f0 comp rx=0 tx=0).stop 2024-03-18T16:35:48.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.080+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0003a00 msgr2=0x7ff2cc1eca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:48.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.080+0000 7ff2d1600640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0003a00 0x7ff2cc1eca40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc000c00 tx=0x7ff2bc0047d0 comp rx=0 tx=0).stop 2024-03-18T16:35:48.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.080+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 shutdown_connections 2024-03-18T16:35:48.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.080+0000 7ff2d1600640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff2ac06bc50 0x7ff2ac06e110 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:48.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.080+0000 7ff2d1600640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2cc13e780 0x7ff2cc13d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:48.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.080+0000 7ff2d1600640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff2cc13e340 0x7ff2cc0b8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:48.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.080+0000 7ff2d1600640 1 --2- 172.21.15.102:0/3307944684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2b0003a00 0x7ff2cc1eca40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:48.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.081+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 >> 172.21.15.102:0/3307944684 conn(0x7ff2cc0cb890 msgr2=0x7ff2cc13ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:48.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.081+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 shutdown_connections 2024-03-18T16:35:48.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:48.081+0000 7ff2d1600640 1 -- 172.21.15.102:0/3307944684 wait complete. 2024-03-18T16:35:48.709 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:48.709 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:49.710 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:49.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.884+0000 7f3c17000640 1 Processor -- start 2024-03-18T16:35:49.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.884+0000 7f3c17000640 1 -- start start 2024-03-18T16:35:49.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.885+0000 7f3c17000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c10158420 0x7f3c1015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:49.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.885+0000 7f3c17000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c100c4fa0 0x7f3c100c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:49.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.885+0000 7f3c17000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c100c72d0 con 0x7f3c10141650 2024-03-18T16:35:49.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.885+0000 7f3c17000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3c1015c8e0 con 0x7f3c10158420 2024-03-18T16:35:49.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.885+0000 7f3c17000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3c10159370 con 0x7f3c100c4fa0 2024-03-18T16:35:49.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.886+0000 7f3c15200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c10158420 0x7f3c1015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:49.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.886+0000 7f3c16600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c100c4fa0 0x7f3c100c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:49.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.886+0000 7f3c16600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c100c4fa0 0x7f3c100c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53958/0 (socket says 172.21.15.102:53958) 2024-03-18T16:35:49.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.886+0000 7f3c15200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c10158420 0x7f3c1015c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35574/0 (socket says 172.21.15.102:35574) 2024-03-18T16:35:49.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.886+0000 7f3c16600640 1 -- 172.21.15.102:0/967849547 learned_addr learned my addr 172.21.15.102:0/967849547 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.889+0000 7f3c16600640 1 -- 172.21.15.102:0/967849547 >> v2:172.21.15.102:3300/0 conn(0x7f3c10158420 msgr2=0x7f3c1015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.889+0000 7f3c16600640 1 --2- 172.21.15.102:0/967849547 >> v2:172.21.15.102:3300/0 conn(0x7f3c10158420 0x7f3c1015c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.889+0000 7f3c16600640 1 -- 172.21.15.102:0/967849547 >> v1:172.21.15.154:6789/0 conn(0x7f3c10141650 legacy=0x7f3c100c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.889+0000 7f3c16600640 1 -- 172.21.15.102:0/967849547 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c10158840 con 0x7f3c100c4fa0 2024-03-18T16:35:49.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.891+0000 7f3c16600640 1 --2- 172.21.15.102:0/967849547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c100c4fa0 0x7f3c100c5430 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009e50 tx=0x7f3c0802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=579499765b7d6030 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.895+0000 7f3c0fe00640 1 -- 172.21.15.102:0/967849547 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c0802cb00 con 0x7f3c100c4fa0 2024-03-18T16:35:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.895+0000 7f3c0fe00640 1 -- 172.21.15.102:0/967849547 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c0802cca0 con 0x7f3c100c4fa0 2024-03-18T16:35:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.895+0000 7f3c0fe00640 1 -- 172.21.15.102:0/967849547 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08034610 con 0x7f3c100c4fa0 2024-03-18T16:35:49.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.895+0000 7f3c17000640 1 -- 172.21.15.102:0/967849547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c100c4fa0 msgr2=0x7f3c100c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:49.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.895+0000 7f3c17000640 1 --2- 172.21.15.102:0/967849547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c100c4fa0 0x7f3c100c5430 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009e50 tx=0x7f3c0802bf60 comp rx=0 tx=0).stop 2024-03-18T16:35:49.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.896+0000 7f3c17000640 1 -- 172.21.15.102:0/967849547 shutdown_connections 2024-03-18T16:35:49.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.896+0000 7f3c17000640 1 --2- 172.21.15.102:0/967849547 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c100c4fa0 0x7f3c100c5430 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:49.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.896+0000 7f3c17000640 1 --2- 172.21.15.102:0/967849547 >> v2:172.21.15.102:3300/0 conn(0x7f3c10158420 0x7f3c1015c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:49.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.896+0000 7f3c17000640 1 -- 172.21.15.102:0/967849547 >> 172.21.15.102:0/967849547 conn(0x7f3c100cb890 msgr2=0x7f3c100cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:49.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.896+0000 7f3c17000640 1 -- 172.21.15.102:0/967849547 shutdown_connections 2024-03-18T16:35:49.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.896+0000 7f3c17000640 1 -- 172.21.15.102:0/967849547 wait complete. 2024-03-18T16:35:49.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.897+0000 7f3c17000640 1 Processor -- start 2024-03-18T16:35:49.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.898+0000 7f3c17000640 1 -- start start 2024-03-18T16:35:49.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.898+0000 7f3c17000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c100c4fa0 0x7f3c101e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:49.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.898+0000 7f3c17000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c10141650 0x7f3c100ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:49.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.899+0000 7f3c17000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c10158420 0x7f3c100caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:49.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.899+0000 7f3c17000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3c10159090 con 0x7f3c100c4fa0 2024-03-18T16:35:49.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.899+0000 7f3c17000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3c10158db0 con 0x7f3c10141650 2024-03-18T16:35:49.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.899+0000 7f3c17000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3c100c63c0 con 0x7f3c10158420 2024-03-18T16:35:49.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.899+0000 7f3c16600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c10158420 0x7f3c100caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:49.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.899+0000 7f3c15200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c10141650 0x7f3c100ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:49.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.899+0000 7f3c15c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c100c4fa0 0x7f3c101e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:49.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.900+0000 7f3c16600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c10158420 0x7f3c100caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:53964/0 (socket says 172.21.15.102:53964) 2024-03-18T16:35:49.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.900+0000 7f3c16600640 1 -- 172.21.15.102:0/2713603346 learned_addr learned my addr 172.21.15.102:0/2713603346 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:49.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.903+0000 7f3c16600640 1 -- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c10141650 msgr2=0x7f3c100ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:49.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.903+0000 7f3c16600640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c10141650 0x7f3c100ca9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:49.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.903+0000 7f3c16600640 1 -- 172.21.15.102:0/2713603346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c100c4fa0 msgr2=0x7f3c101e5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:49.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.903+0000 7f3c16600640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c100c4fa0 0x7f3c101e5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:49.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.903+0000 7f3c16600640 1 -- 172.21.15.102:0/2713603346 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c100ca1b0 con 0x7f3c10158420 2024-03-18T16:35:49.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.906+0000 7f3c16600640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c10158420 0x7f3c100caf00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3c08036040 tx=0x7f3c08004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:49.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.910+0000 7f3c0d600640 1 -- 172.21.15.102:0/2713603346 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08043020 con 0x7f3c10158420 2024-03-18T16:35:49.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.910+0000 7f3c0d600640 1 -- 172.21.15.102:0/2713603346 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c0802ce50 con 0x7f3c10158420 2024-03-18T16:35:49.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.910+0000 7f3c0d600640 1 -- 172.21.15.102:0/2713603346 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08009510 con 0x7f3c10158420 2024-03-18T16:35:49.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.910+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c100ca750 con 0x7f3c10158420 2024-03-18T16:35:49.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.910+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c101f1c40 con 0x7f3c10158420 2024-03-18T16:35:49.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.911+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c10157410 con 0x7f3c10158420 2024-03-18T16:35:49.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.931+0000 7f3c0d600640 1 -- 172.21.15.102:0/2713603346 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3c08046020 con 0x7f3c10158420 2024-03-18T16:35:49.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.932+0000 7f3c0d600640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bf006bc20 0x7f3bf006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:49.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.932+0000 7f3c15c00640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bf006bc20 0x7f3bf006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:49.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:49.933+0000 7f3c15c00640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bf006bc20 0x7f3bf006e0e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3c04001780 tx=0x7f3c04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:50.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.031+0000 7f3c0d600640 1 -- 172.21.15.102:0/2713603346 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3c08079830 con 0x7f3c10158420 2024-03-18T16:35:50.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.185+0000 7f3c0d600640 1 -- 172.21.15.102:0/2713603346 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3c10157410 con 0x7f3c10158420 2024-03-18T16:35:50.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.430+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c10157e50 con 0x7f3c10158420 2024-03-18T16:35:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.445+0000 7f3c0d600640 1 -- 172.21.15.102:0/2713603346 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3c0804d030 con 0x7f3c10158420 2024-03-18T16:35:50.447 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:50.447 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:50.447 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:50.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.447+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bf006bc20 msgr2=0x7f3bf006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.447+0000 7f3c17000640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bf006bc20 0x7f3bf006e0e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3c04001780 tx=0x7f3c04009040 comp rx=0 tx=0).stop 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.447+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c10158420 msgr2=0x7f3c100caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.447+0000 7f3c17000640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c10158420 0x7f3c100caf00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3c08036040 tx=0x7f3c08004a40 comp rx=0 tx=0).stop 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.448+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 shutdown_connections 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.448+0000 7f3c17000640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3bf006bc20 0x7f3bf006e0e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.448+0000 7f3c17000640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c10158420 0x7f3c100caf00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.448+0000 7f3c17000640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c10141650 0x7f3c100ca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.448+0000 7f3c17000640 1 --2- 172.21.15.102:0/2713603346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c100c4fa0 0x7f3c101e5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.448+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 >> 172.21.15.102:0/2713603346 conn(0x7f3c100cb890 msgr2=0x7f3c1013f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.448+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 shutdown_connections 2024-03-18T16:35:50.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:50.448+0000 7f3c17000640 1 -- 172.21.15.102:0/2713603346 wait complete. 2024-03-18T16:35:51.070 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:51.070 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:52.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:52.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.243+0000 7f5ddaa00640 1 Processor -- start 2024-03-18T16:35:52.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.243+0000 7f5ddaa00640 1 -- start start 2024-03-18T16:35:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.244+0000 7f5ddaa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5dd4141650 0x7f5dd4158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.244+0000 7f5ddaa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5dd40c4fa0 0x7f5dd40c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:52.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.244+0000 7f5ddaa00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dd40c72d0 con 0x7f5dd415c3a0 2024-03-18T16:35:52.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.244+0000 7f5ddaa00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd4159370 con 0x7f5dd40c4fa0 2024-03-18T16:35:52.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.244+0000 7f5ddaa00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd4158ad0 con 0x7f5dd4141650 2024-03-18T16:35:52.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.245+0000 7f5dd9600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5dd4141650 0x7f5dd4158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:52.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.245+0000 7f5dda000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5dd40c4fa0 0x7f5dd40c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:52.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.245+0000 7f5dda000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5dd40c4fa0 0x7f5dd40c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56556/0 (socket says 172.21.15.102:56556) 2024-03-18T16:35:52.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.245+0000 7f5dda000640 1 -- 172.21.15.102:0/1885176778 learned_addr learned my addr 172.21.15.102:0/1885176778 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:52.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.248+0000 7f5dda000640 1 -- 172.21.15.102:0/1885176778 >> v2:172.21.15.154:3300/0 conn(0x7f5dd4141650 msgr2=0x7f5dd4158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:52.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.248+0000 7f5dda000640 1 --2- 172.21.15.102:0/1885176778 >> v2:172.21.15.154:3300/0 conn(0x7f5dd4141650 0x7f5dd4158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:52.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.248+0000 7f5dda000640 1 -- 172.21.15.102:0/1885176778 >> v1:172.21.15.154:6789/0 conn(0x7f5dd415c3a0 legacy=0x7f5dd40c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:52.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.248+0000 7f5dda000640 1 -- 172.21.15.102:0/1885176778 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd4158db0 con 0x7f5dd40c4fa0 2024-03-18T16:35:52.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.250+0000 7f5dda000640 1 --2- 172.21.15.102:0/1885176778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c4fa0 0x7f5dd40c5410 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4068ac0 tx=0x7f5dcc02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d01242137d52a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:52.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.255+0000 7f5dd3e00640 1 -- 172.21.15.102:0/1885176778 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc02cb10 con 0x7f5dd40c4fa0 2024-03-18T16:35:52.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.255+0000 7f5dd3e00640 1 -- 172.21.15.102:0/1885176778 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dcc02ccb0 con 0x7f5dd40c4fa0 2024-03-18T16:35:52.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.255+0000 7f5dd3e00640 1 -- 172.21.15.102:0/1885176778 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc02ce60 con 0x7f5dd40c4fa0 2024-03-18T16:35:52.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.256+0000 7f5ddaa00640 1 -- 172.21.15.102:0/1885176778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c4fa0 msgr2=0x7f5dd40c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:52.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.256+0000 7f5ddaa00640 1 --2- 172.21.15.102:0/1885176778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c4fa0 0x7f5dd40c5410 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4068ac0 tx=0x7f5dcc02bf70 comp rx=0 tx=0).stop 2024-03-18T16:35:52.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.257+0000 7f5ddaa00640 1 -- 172.21.15.102:0/1885176778 shutdown_connections 2024-03-18T16:35:52.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.257+0000 7f5ddaa00640 1 --2- 172.21.15.102:0/1885176778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd40c4fa0 0x7f5dd40c5410 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:52.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.257+0000 7f5ddaa00640 1 --2- 172.21.15.102:0/1885176778 >> v2:172.21.15.154:3300/0 conn(0x7f5dd4141650 0x7f5dd4158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:52.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.257+0000 7f5ddaa00640 1 -- 172.21.15.102:0/1885176778 >> 172.21.15.102:0/1885176778 conn(0x7f5dd40cb890 msgr2=0x7f5dd40cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:52.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.257+0000 7f5ddaa00640 1 -- 172.21.15.102:0/1885176778 shutdown_connections 2024-03-18T16:35:52.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.257+0000 7f5ddaa00640 1 -- 172.21.15.102:0/1885176778 wait complete. 2024-03-18T16:35:52.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.258+0000 7f5ddaa00640 1 Processor -- start 2024-03-18T16:35:52.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.259+0000 7f5ddaa00640 1 -- start start 2024-03-18T16:35:52.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.259+0000 7f5ddaa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dd40c4fa0 0x7f5dd4159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:52.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.259+0000 7f5ddaa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dd4141650 0x7f5dd413d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:52.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.259+0000 7f5ddaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd415c3a0 0x7f5dd4153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:52.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.260+0000 7f5ddaa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd40c66a0 con 0x7f5dd415c3a0 2024-03-18T16:35:52.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.260+0000 7f5ddaa00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd40c63c0 con 0x7f5dd40c4fa0 2024-03-18T16:35:52.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.260+0000 7f5ddaa00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd415c7c0 con 0x7f5dd4141650 2024-03-18T16:35:52.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.260+0000 7f5dd9600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dd40c4fa0 0x7f5dd4159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:52.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.260+0000 7f5dda000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd415c3a0 0x7f5dd4153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:52.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.260+0000 7f5dd8c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dd4141650 0x7f5dd413d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:52.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.261+0000 7f5dd8c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dd4141650 0x7f5dd413d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57422/0 (socket says 172.21.15.102:57422) 2024-03-18T16:35:52.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.261+0000 7f5dd9600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dd40c4fa0 0x7f5dd4159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49614/0 (socket says 172.21.15.102:49614) 2024-03-18T16:35:52.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.261+0000 7f5dd8c00640 1 -- 172.21.15.102:0/433045876 learned_addr learned my addr 172.21.15.102:0/433045876 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:52.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.264+0000 7f5dda000640 1 -- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dd4141650 msgr2=0x7f5dd413d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:52.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.264+0000 7f5dda000640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dd4141650 0x7f5dd413d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:52.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.265+0000 7f5dda000640 1 -- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dd40c4fa0 msgr2=0x7f5dd4159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:52.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.265+0000 7f5dda000640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dd40c4fa0 0x7f5dd4159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:52.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.265+0000 7f5dda000640 1 -- 172.21.15.102:0/433045876 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd4159f80 con 0x7f5dd415c3a0 2024-03-18T16:35:52.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.267+0000 7f5dda000640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd415c3a0 0x7f5dd4153490 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4070160 tx=0x7f5dcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:52.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.270+0000 7f5dd1600640 1 -- 172.21.15.102:0/433045876 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc043020 con 0x7f5dd415c3a0 2024-03-18T16:35:52.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.270+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd41531b0 con 0x7f5dd415c3a0 2024-03-18T16:35:52.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.271+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd4155970 con 0x7f5dd415c3a0 2024-03-18T16:35:52.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.272+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd4074fd0 con 0x7f5dd415c3a0 2024-03-18T16:35:52.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.477+0000 7f5dd1600640 1 -- 172.21.15.102:0/433045876 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dcc02cb20 con 0x7f5dd415c3a0 2024-03-18T16:35:52.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.560+0000 7f5dd1600640 1 -- 172.21.15.102:0/433045876 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc02ce60 con 0x7f5dd415c3a0 2024-03-18T16:35:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.562+0000 7f5dd1600640 1 -- 172.21.15.102:0/433045876 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5dcc046020 con 0x7f5dd415c3a0 2024-03-18T16:35:52.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.563+0000 7f5dd1600640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5db406bc00 0x7f5db406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:52.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.563+0000 7f5dd9600640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5db406bc00 0x7f5db406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:52.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.564+0000 7f5dd9600640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5db406bc00 0x7f5db406e0c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009b40 tx=0x7f5dc8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:52.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.587+0000 7f5dd1600640 1 -- 172.21.15.102:0/433045876 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5dcc0799a0 con 0x7f5dd415c3a0 2024-03-18T16:35:52.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.588+0000 7f5dd1600640 1 -- 172.21.15.102:0/433045876 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5dcc04d030 con 0x7f5dd415c3a0 2024-03-18T16:35:52.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.859+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5dd40b7f00 con 0x7f5dd415c3a0 2024-03-18T16:35:52.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.887+0000 7f5dd1600640 1 -- 172.21.15.102:0/433045876 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f5dcc07b900 con 0x7f5dd415c3a0 2024-03-18T16:35:52.889 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:52.889 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:52.889 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:52.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5db406bc00 msgr2=0x7f5db406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5db406bc00 0x7f5db406e0c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8009b40 tx=0x7f5dc8009190 comp rx=0 tx=0).stop 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd415c3a0 msgr2=0x7f5dd4153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd415c3a0 0x7f5dd4153490 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4070160 tx=0x7f5dcc004a40 comp rx=0 tx=0).stop 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 shutdown_connections 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5db406bc00 0x7f5db406e0c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5dd415c3a0 0x7f5dd4153490 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5dd4141650 0x7f5dd413d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 --2- 172.21.15.102:0/433045876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5dd40c4fa0 0x7f5dd4159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 >> 172.21.15.102:0/433045876 conn(0x7f5dd40cb890 msgr2=0x7f5dd40c9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 shutdown_connections 2024-03-18T16:35:52.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:52.891+0000 7f5ddaa00640 1 -- 172.21.15.102:0/433045876 wait complete. 2024-03-18T16:35:53.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:53.520 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:54.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:35:54.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:35:54.522 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.689+0000 7f9fd6600640 1 Processor -- start 2024-03-18T16:35:54.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.689+0000 7f9fd6600640 1 -- start start 2024-03-18T16:35:54.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.690+0000 7f9fd6600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd006c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.690+0000 7f9fd6600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9fd015a1a0 0x7f9fd00705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.690+0000 7f9fd6600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd006c9c0 con 0x7f9fd00cad40 2024-03-18T16:35:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.690+0000 7f9fd6600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f9fd00c8680 con 0x7f9fd015a1a0 2024-03-18T16:35:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.690+0000 7f9fd6600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd006cca0 con 0x7f9fd00ca960 2024-03-18T16:35:54.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.691+0000 7f9fd5200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd006c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.691+0000 7f9fd5200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd006c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57436/0 (socket says 172.21.15.102:57436) 2024-03-18T16:35:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.691+0000 7f9fd5200640 1 -- 172.21.15.102:0/1629389177 learned_addr learned my addr 172.21.15.102:0/1629389177 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:54.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.691+0000 7f9fd5c00640 1 --2- 172.21.15.102:0/1629389177 >> v2:172.21.15.102:3300/0 conn(0x7f9fd015a1a0 0x7f9fd00705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.695+0000 7f9fd5200640 1 -- 172.21.15.102:0/1629389177 >> v2:172.21.15.102:3300/0 conn(0x7f9fd015a1a0 msgr2=0x7f9fd00705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:54.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.695+0000 7f9fd5200640 1 --2- 172.21.15.102:0/1629389177 >> v2:172.21.15.102:3300/0 conn(0x7f9fd015a1a0 0x7f9fd00705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.695+0000 7f9fd5200640 1 -- 172.21.15.102:0/1629389177 >> v1:172.21.15.102:6789/0 conn(0x7f9fd00cad40 legacy=0x7f9fd015e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:54.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.695+0000 7f9fd5200640 1 -- 172.21.15.102:0/1629389177 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd006cf80 con 0x7f9fd00ca960 2024-03-18T16:35:54.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.700+0000 7f9fd5200640 1 --2- 172.21.15.102:0/1629389177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd006c310 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4009870 tx=0x7f9fc402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59ce95c82c5e482d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.705+0000 7f9fcf400640 1 -- 172.21.15.102:0/1629389177 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc402f830 con 0x7f9fd00ca960 2024-03-18T16:35:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.705+0000 7f9fcf400640 1 -- 172.21.15.102:0/1629389177 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc402f9d0 con 0x7f9fd00ca960 2024-03-18T16:35:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.705+0000 7f9fcf400640 1 -- 172.21.15.102:0/1629389177 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc402fda0 con 0x7f9fd00ca960 2024-03-18T16:35:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.706+0000 7f9fd6600640 1 -- 172.21.15.102:0/1629389177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 msgr2=0x7f9fd006c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.706+0000 7f9fd6600640 1 --2- 172.21.15.102:0/1629389177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd006c310 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4009870 tx=0x7f9fc402ba10 comp rx=0 tx=0).stop 2024-03-18T16:35:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.706+0000 7f9fd6600640 1 -- 172.21.15.102:0/1629389177 shutdown_connections 2024-03-18T16:35:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.706+0000 7f9fd6600640 1 --2- 172.21.15.102:0/1629389177 >> v2:172.21.15.102:3300/0 conn(0x7f9fd015a1a0 0x7f9fd00705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.706+0000 7f9fd6600640 1 --2- 172.21.15.102:0/1629389177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd006c310 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.706+0000 7f9fd6600640 1 -- 172.21.15.102:0/1629389177 >> 172.21.15.102:0/1629389177 conn(0x7f9fd0067190 msgr2=0x7f9fd00675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.706+0000 7f9fd6600640 1 -- 172.21.15.102:0/1629389177 shutdown_connections 2024-03-18T16:35:54.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.706+0000 7f9fd6600640 1 -- 172.21.15.102:0/1629389177 wait complete. 2024-03-18T16:35:54.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.708+0000 7f9fd6600640 1 Processor -- start 2024-03-18T16:35:54.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.708+0000 7f9fd6600640 1 -- start start 2024-03-18T16:35:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.709+0000 7f9fd6600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd00c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.709+0000 7f9fd6600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fd00cad40 0x7f9fd0141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:54.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.709+0000 7f9fd6600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fd015a1a0 0x7f9fd0156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.709+0000 7f9fd6600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd0070ae0 con 0x7f9fd00cad40 2024-03-18T16:35:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.709+0000 7f9fd6600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd015dc10 con 0x7f9fd015a1a0 2024-03-18T16:35:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.709+0000 7f9fd6600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd0070c60 con 0x7f9fd00ca960 2024-03-18T16:35:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.709+0000 7f9fd5200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd00c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.709+0000 7f9fcfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fd00cad40 0x7f9fd0141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:54.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.710+0000 7f9fd5c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fd015a1a0 0x7f9fd0156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.710+0000 7f9fcfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fd00cad40 0x7f9fd0141ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:56586/0 (socket says 172.21.15.102:56586) 2024-03-18T16:35:54.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.710+0000 7f9fcfe00640 1 -- 172.21.15.102:0/3199128981 learned_addr learned my addr 172.21.15.102:0/3199128981 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.714+0000 7f9fcfe00640 1 -- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 msgr2=0x7f9fd00c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.714+0000 7f9fcfe00640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd00c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.714+0000 7f9fcfe00640 1 -- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fd015a1a0 msgr2=0x7f9fd0156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.714+0000 7f9fcfe00640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fd015a1a0 0x7f9fd0156e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.714+0000 7f9fcfe00640 1 -- 172.21.15.102:0/3199128981 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd01560f0 con 0x7f9fd00cad40 2024-03-18T16:35:54.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.714+0000 7f9fd5c00640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fd015a1a0 0x7f9fd0156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:35:54.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.717+0000 7f9fcfe00640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fd00cad40 0x7f9fd0141ca0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00c430 tx=0x7f9fbc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.721+0000 7f9fccc00640 1 -- 172.21.15.102:0/3199128981 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc0052d0 con 0x7f9fd00cad40 2024-03-18T16:35:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.721+0000 7f9fccc00640 1 -- 172.21.15.102:0/3199128981 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fbc005490 con 0x7f9fd00cad40 2024-03-18T16:35:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.721+0000 7f9fccc00640 1 -- 172.21.15.102:0/3199128981 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc007260 con 0x7f9fd00cad40 2024-03-18T16:35:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.721+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd01f0580 con 0x7f9fd00cad40 2024-03-18T16:35:54.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.722+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd0156670 con 0x7f9fd00cad40 2024-03-18T16:35:54.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.723+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fd0074fd0 con 0x7f9fd00cad40 2024-03-18T16:35:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.743+0000 7f9fccc00640 1 -- 172.21.15.102:0/3199128981 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9fbc007400 con 0x7f9fd00cad40 2024-03-18T16:35:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.744+0000 7f9fccc00640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9fb006bc50 0x7f9fb006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.745+0000 7f9fd5200640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9fb006bc50 0x7f9fb006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:54.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:54.746+0000 7f9fd5200640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9fb006bc50 0x7f9fb006e110 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4036000 tx=0x7f9fc4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:55.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.012+0000 7f9fccc00640 1 -- 172.21.15.102:0/3199128981 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9fbc056980 con 0x7f9fd00cad40 2024-03-18T16:35:55.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.013+0000 7f9fccc00640 1 -- 172.21.15.102:0/3199128981 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9fbc0588e0 con 0x7f9fd00cad40 2024-03-18T16:35:55.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.336+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fd0067940 con 0x7f9fd00cad40 2024-03-18T16:35:55.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.358+0000 7f9fccc00640 1 -- 172.21.15.102:0/3199128981 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f9fbc05c980 con 0x7f9fd00cad40 2024-03-18T16:35:55.360 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:55.360 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:55.360 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.361+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9fb006bc50 msgr2=0x7f9fb006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9fb006bc50 0x7f9fb006e110 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4036000 tx=0x7f9fc4008030 comp rx=0 tx=0).stop 2024-03-18T16:35:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fd00cad40 msgr2=0x7f9fd0141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:55.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fd00cad40 0x7f9fd0141ca0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00c430 tx=0x7f9fbc00c8f0 comp rx=0 tx=0).stop 2024-03-18T16:35:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 shutdown_connections 2024-03-18T16:35:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9fb006bc50 0x7f9fb006e110 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fd015a1a0 0x7f9fd0156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fd00cad40 0x7f9fd0141ca0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 --2- 172.21.15.102:0/3199128981 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9fd00ca960 0x7f9fd00c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 >> 172.21.15.102:0/3199128981 conn(0x7f9fd0067190 msgr2=0x7f9fd015d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 shutdown_connections 2024-03-18T16:35:55.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:55.362+0000 7f9fd6600640 1 -- 172.21.15.102:0/3199128981 wait complete. 2024-03-18T16:35:55.984 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:55.984 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:56.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.161+0000 7f1ebee00640 1 Processor -- start 2024-03-18T16:35:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.161+0000 7f1ebee00640 1 -- start start 2024-03-18T16:35:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.162+0000 7f1ebee00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1eb80cfe70 0x7f1eb815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.162+0000 7f1ebee00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1eb80cc2f0 0x7f1eb80cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.162+0000 7f1ebee00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eb80ccc80 con 0x7f1eb80d0250 2024-03-18T16:35:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.162+0000 7f1ebee00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb80cabd0 con 0x7f1eb80cfe70 2024-03-18T16:35:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.162+0000 7f1ebee00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb80caea0 con 0x7f1eb80cc2f0 2024-03-18T16:35:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.163+0000 7f1ebe400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1eb80cc2f0 0x7f1eb80cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.163+0000 7f1ebe400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1eb80cc2f0 0x7f1eb80cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49630/0 (socket says 172.21.15.102:49630) 2024-03-18T16:35:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.163+0000 7f1ebe400640 1 -- 172.21.15.102:0/563981926 learned_addr learned my addr 172.21.15.102:0/563981926 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.164+0000 7f1ebda00640 1 --2- 172.21.15.102:0/563981926 >> v2:172.21.15.102:3300/0 conn(0x7f1eb80cfe70 0x7f1eb815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.166+0000 7f1ebe400640 1 -- 172.21.15.102:0/563981926 >> v2:172.21.15.102:3300/0 conn(0x7f1eb80cfe70 msgr2=0x7f1eb815ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.166+0000 7f1ebe400640 1 --2- 172.21.15.102:0/563981926 >> v2:172.21.15.102:3300/0 conn(0x7f1eb80cfe70 0x7f1eb815ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.166+0000 7f1ebe400640 1 -- 172.21.15.102:0/563981926 >> v1:172.21.15.102:6789/0 conn(0x7f1eb80d0250 legacy=0x7f1eb80c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.166+0000 7f1ebe400640 1 -- 172.21.15.102:0/563981926 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb80cb180 con 0x7f1eb80cc2f0 2024-03-18T16:35:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.168+0000 7f1ebe400640 1 --2- 172.21.15.102:0/563981926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 0x7f1eb80cc740 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1eb80ca160 tx=0x7f1eb002c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ff4ecb8b1f091ea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:57.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.170+0000 7f1eb7e00640 1 -- 172.21.15.102:0/563981926 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb002cbb0 con 0x7f1eb80cc2f0 2024-03-18T16:35:57.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.172+0000 7f1eb7e00640 1 -- 172.21.15.102:0/563981926 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb002cd50 con 0x7f1eb80cc2f0 2024-03-18T16:35:57.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.172+0000 7f1eb7e00640 1 -- 172.21.15.102:0/563981926 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0034610 con 0x7f1eb80cc2f0 2024-03-18T16:35:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.173+0000 7f1ebee00640 1 -- 172.21.15.102:0/563981926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 msgr2=0x7f1eb80cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.173+0000 7f1ebee00640 1 --2- 172.21.15.102:0/563981926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 0x7f1eb80cc740 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f1eb80ca160 tx=0x7f1eb002c010 comp rx=0 tx=0).stop 2024-03-18T16:35:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.173+0000 7f1ebee00640 1 -- 172.21.15.102:0/563981926 shutdown_connections 2024-03-18T16:35:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.173+0000 7f1ebee00640 1 --2- 172.21.15.102:0/563981926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 0x7f1eb80cc740 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.173+0000 7f1ebee00640 1 --2- 172.21.15.102:0/563981926 >> v2:172.21.15.102:3300/0 conn(0x7f1eb80cfe70 0x7f1eb815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.173+0000 7f1ebee00640 1 -- 172.21.15.102:0/563981926 >> 172.21.15.102:0/563981926 conn(0x7f1eb808c3e0 msgr2=0x7f1eb806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.173+0000 7f1ebee00640 1 -- 172.21.15.102:0/563981926 shutdown_connections 2024-03-18T16:35:57.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.174+0000 7f1ebee00640 1 -- 172.21.15.102:0/563981926 wait complete. 2024-03-18T16:35:57.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.176+0000 7f1ebee00640 1 Processor -- start 2024-03-18T16:35:57.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.176+0000 7f1ebee00640 1 -- start start 2024-03-18T16:35:57.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.178+0000 7f1ebee00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 0x7f1eb80c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:57.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.178+0000 7f1ebee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1eb80cfe70 0x7f1eb81e03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:57.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.178+0000 7f1ebee00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eb80d0250 0x7f1eb8156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:57.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.178+0000 7f1ebee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb80cba20 con 0x7f1eb80cfe70 2024-03-18T16:35:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.178+0000 7f1ebee00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb80cb740 con 0x7f1eb80cc2f0 2024-03-18T16:35:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.178+0000 7f1ebee00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb80cbd00 con 0x7f1eb80d0250 2024-03-18T16:35:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.179+0000 7f1ebda00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 0x7f1eb80c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.179+0000 7f1ebd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1eb80cfe70 0x7f1eb81e03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.179+0000 7f1ebe400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eb80d0250 0x7f1eb8156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.179+0000 7f1ebda00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 0x7f1eb80c8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49646/0 (socket says 172.21.15.102:49646) 2024-03-18T16:35:57.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.179+0000 7f1ebda00640 1 -- 172.21.15.102:0/443827174 learned_addr learned my addr 172.21.15.102:0/443827174 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:57.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.183+0000 7f1ebd000640 1 -- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eb80d0250 msgr2=0x7f1eb8156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:57.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.183+0000 7f1ebd000640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eb80d0250 0x7f1eb8156e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:57.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.183+0000 7f1ebd000640 1 -- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 msgr2=0x7f1eb80c8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:57.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.183+0000 7f1ebd000640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 0x7f1eb80c8270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:57.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.183+0000 7f1ebd000640 1 -- 172.21.15.102:0/443827174 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb81f3010 con 0x7f1eb80cfe70 2024-03-18T16:35:57.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.187+0000 7f1ebd000640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1eb80cfe70 0x7f1eb81e03b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1eb80d0660 tx=0x7f1ea400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:57.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.192+0000 7f1eb5600640 1 -- 172.21.15.102:0/443827174 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4017070 con 0x7f1eb80cfe70 2024-03-18T16:35:57.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.192+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb81e1d40 con 0x7f1eb80cfe70 2024-03-18T16:35:57.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.192+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb81f3850 con 0x7f1eb80cfe70 2024-03-18T16:35:57.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.192+0000 7f1eb5600640 1 -- 172.21.15.102:0/443827174 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea40052d0 con 0x7f1eb80cfe70 2024-03-18T16:35:57.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.192+0000 7f1eb5600640 1 -- 172.21.15.102:0/443827174 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4007220 con 0x7f1eb80cfe70 2024-03-18T16:35:57.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.193+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e94005290 con 0x7f1eb80cfe70 2024-03-18T16:35:57.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.223+0000 7f1eb5600640 1 -- 172.21.15.102:0/443827174 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1ea401a070 con 0x7f1eb80cfe70 2024-03-18T16:35:57.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.224+0000 7f1eb5600640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e9806bc00 0x7f1e9806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:57.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.224+0000 7f1ebda00640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e9806bc00 0x7f1e9806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:57.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.225+0000 7f1ebda00640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e9806bc00 0x7f1e9806e0c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1eac006fd0 tx=0x7f1eac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:57.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.225+0000 7f1eb5600640 1 -- 172.21.15.102:0/443827174 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1ea40569a0 con 0x7f1eb80cfe70 2024-03-18T16:35:57.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.500+0000 7f1eb5600640 1 -- 172.21.15.102:0/443827174 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1ea4021070 con 0x7f1eb80cfe70 2024-03-18T16:35:57.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.840+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e94005580 con 0x7f1eb80cfe70 2024-03-18T16:35:57.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.864+0000 7f1eb5600640 1 -- 172.21.15.102:0/443827174 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1ea4058900 con 0x7f1eb80cfe70 2024-03-18T16:35:57.866 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:35:57.866 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:35:57.867 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:35:57.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e9806bc00 msgr2=0x7f1e9806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e9806bc00 0x7f1e9806e0c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1eac006fd0 tx=0x7f1eac008040 comp rx=0 tx=0).stop 2024-03-18T16:35:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1eb80cfe70 msgr2=0x7f1eb81e03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1eb80cfe70 0x7f1eb81e03b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1eb80d0660 tx=0x7f1ea400c820 comp rx=0 tx=0).stop 2024-03-18T16:35:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 shutdown_connections 2024-03-18T16:35:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1e9806bc00 0x7f1e9806e0c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1eb80d0250 0x7f1eb8156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1eb80cfe70 0x7f1eb81e03b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 --2- 172.21.15.102:0/443827174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1eb80cc2f0 0x7f1eb80c8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:57.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.867+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 >> 172.21.15.102:0/443827174 conn(0x7f1eb808c3e0 msgr2=0x7f1eb80cce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:57.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.868+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 shutdown_connections 2024-03-18T16:35:57.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:57.868+0000 7f1ebee00640 1 -- 172.21.15.102:0/443827174 wait complete. 2024-03-18T16:35:58.489 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:35:58.489 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:35:59.490 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:35:59.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.698+0000 7f4502400640 1 Processor -- start 2024-03-18T16:35:59.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.698+0000 7f4502400640 1 -- start start 2024-03-18T16:35:59.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.700+0000 7f4502400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44fc1524c0 0x7f44fc072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:59.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.700+0000 7f4502400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44fc158ee0 0x7f44fc06e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:59.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.700+0000 7f4502400640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44fc15c200 con 0x7f44fc15b8e0 2024-03-18T16:35:59.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.700+0000 7f4502400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f44fc159300 con 0x7f44fc1524c0 2024-03-18T16:35:59.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.700+0000 7f4502400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f44fc15c4e0 con 0x7f44fc158ee0 2024-03-18T16:35:59.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.706+0000 7f44fb400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44fc1524c0 0x7f44fc072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:59.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.706+0000 7f44fb400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44fc1524c0 0x7f44fc072700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49650/0 (socket says 172.21.15.102:49650) 2024-03-18T16:35:59.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.706+0000 7f44fb400640 1 -- 172.21.15.102:0/2767294759 learned_addr learned my addr 172.21.15.102:0/2767294759 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:59.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.711+0000 7f44fb400640 1 -- 172.21.15.102:0/2767294759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44fc158ee0 msgr2=0x7f44fc06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:35:59.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.711+0000 7f44fb400640 1 --2- 172.21.15.102:0/2767294759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44fc158ee0 0x7f44fc06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:59.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.711+0000 7f44fb400640 1 -- 172.21.15.102:0/2767294759 >> v1:172.21.15.154:6789/0 conn(0x7f44fc15b8e0 legacy=0x7f44fc0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:59.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.711+0000 7f44fb400640 1 -- 172.21.15.102:0/2767294759 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc15c7c0 con 0x7f44fc1524c0 2024-03-18T16:35:59.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.717+0000 7f44fb400640 1 --2- 172.21.15.102:0/2767294759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc1524c0 0x7f44fc072700 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009870 tx=0x7f44ec02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75ae2e812627353e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:59.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.722+0000 7f44fa000640 1 -- 172.21.15.102:0/2767294759 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec034070 con 0x7f44fc1524c0 2024-03-18T16:35:59.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.722+0000 7f44fa000640 1 -- 172.21.15.102:0/2767294759 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44ec030840 con 0x7f44fc1524c0 2024-03-18T16:35:59.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.722+0000 7f44fa000640 1 -- 172.21.15.102:0/2767294759 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44ec030c10 con 0x7f44fc1524c0 2024-03-18T16:35:59.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.723+0000 7f4502400640 1 -- 172.21.15.102:0/2767294759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc1524c0 msgr2=0x7f44fc072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:59.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.723+0000 7f4502400640 1 --2- 172.21.15.102:0/2767294759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc1524c0 0x7f44fc072700 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009870 tx=0x7f44ec02b980 comp rx=0 tx=0).stop 2024-03-18T16:35:59.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.723+0000 7f4502400640 1 -- 172.21.15.102:0/2767294759 shutdown_connections 2024-03-18T16:35:59.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.723+0000 7f4502400640 1 --2- 172.21.15.102:0/2767294759 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44fc158ee0 0x7f44fc06e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:59.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.723+0000 7f4502400640 1 --2- 172.21.15.102:0/2767294759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc1524c0 0x7f44fc072700 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:59.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.723+0000 7f4502400640 1 -- 172.21.15.102:0/2767294759 >> 172.21.15.102:0/2767294759 conn(0x7f44fc0700d0 msgr2=0x7f44fc06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:35:59.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.723+0000 7f4502400640 1 -- 172.21.15.102:0/2767294759 shutdown_connections 2024-03-18T16:35:59.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.723+0000 7f4502400640 1 -- 172.21.15.102:0/2767294759 wait complete. 2024-03-18T16:35:59.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.725+0000 7f4502400640 1 Processor -- start 2024-03-18T16:35:59.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.725+0000 7f4502400640 1 -- start start 2024-03-18T16:35:59.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.726+0000 7f4502400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44fc1524c0 0x7f44fc1f08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:59.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.726+0000 7f4502400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc158ee0 0x7f44fc152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:59.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.726+0000 7f4502400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc15b8e0 0x7f44fc0c4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:35:59.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.726+0000 7f4502400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1589c0 con 0x7f44fc15b8e0 2024-03-18T16:35:59.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.726+0000 7f4502400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44fc1586e0 con 0x7f44fc158ee0 2024-03-18T16:35:59.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.726+0000 7f4502400640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f44fc06e880 con 0x7f44fc1524c0 2024-03-18T16:35:59.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.727+0000 7f44faa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc158ee0 0x7f44fc152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:59.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.728+0000 7f44faa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc158ee0 0x7f44fc152e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:49658/0 (socket says 172.21.15.102:49658) 2024-03-18T16:35:59.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.728+0000 7f44faa00640 1 -- 172.21.15.102:0/3588795826 learned_addr learned my addr 172.21.15.102:0/3588795826 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:35:59.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.732+0000 7f44fbe00640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc15b8e0 0x7f44fc0c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:35:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.732+0000 7f44faa00640 1 -- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44fc1524c0 msgr2=0x7f44fc1f08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.732+0000 7f44faa00640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44fc1524c0 0x7f44fc1f08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.732+0000 7f44faa00640 1 -- 172.21.15.102:0/3588795826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc15b8e0 msgr2=0x7f44fc0c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:35:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.732+0000 7f44faa00640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc15b8e0 0x7f44fc0c4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:35:59.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.733+0000 7f44faa00640 1 -- 172.21.15.102:0/3588795826 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44fc0c0ea0 con 0x7f44fc158ee0 2024-03-18T16:35:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.736+0000 7f44faa00640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc158ee0 0x7f44fc152e50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f44e800b4a0 tx=0x7f44e800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:35:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.742+0000 7f44f3400640 1 -- 172.21.15.102:0/3588795826 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8014070 con 0x7f44fc158ee0 2024-03-18T16:35:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.742+0000 7f44f3400640 1 -- 172.21.15.102:0/3588795826 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44e8010610 con 0x7f44fc158ee0 2024-03-18T16:35:59.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.742+0000 7f44f3400640 1 -- 172.21.15.102:0/3588795826 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44e8010a00 con 0x7f44fc158ee0 2024-03-18T16:35:59.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.742+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44fc0c7da0 con 0x7f44fc158ee0 2024-03-18T16:35:59.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.742+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44fc0c8340 con 0x7f44fc158ee0 2024-03-18T16:35:59.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:35:59.743+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44fc074fd0 con 0x7f44fc158ee0 2024-03-18T16:36:00.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.018+0000 7f44f3400640 1 -- 172.21.15.102:0/3588795826 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f44e8017070 con 0x7f44fc158ee0 2024-03-18T16:36:00.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.020+0000 7f44f3400640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44dc06bc20 0x7f44dc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:00.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.021+0000 7f44fb400640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44dc06bc20 0x7f44dc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:00.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.021+0000 7f44fb400640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44dc06bc20 0x7f44dc06e0e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f44ec036000 tx=0x7f44ec03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:00.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.024+0000 7f44f3400640 1 -- 172.21.15.102:0/3588795826 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f44e8056d60 con 0x7f44fc158ee0 2024-03-18T16:36:00.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.045+0000 7f44f3400640 1 -- 172.21.15.102:0/3588795826 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f44e80593c0 con 0x7f44fc158ee0 2024-03-18T16:36:00.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.290+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44fc072f20 con 0x7f44fc158ee0 2024-03-18T16:36:00.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.314+0000 7f44f3400640 1 -- 172.21.15.102:0/3588795826 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f44e8058cc0 con 0x7f44fc158ee0 2024-03-18T16:36:00.315 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:00.315 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:00.316 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:00.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.317+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44dc06bc20 msgr2=0x7f44dc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.317+0000 7f4502400640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44dc06bc20 0x7f44dc06e0e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f44ec036000 tx=0x7f44ec03b040 comp rx=0 tx=0).stop 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.317+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc158ee0 msgr2=0x7f44fc152e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.317+0000 7f4502400640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc158ee0 0x7f44fc152e50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f44e800b4a0 tx=0x7f44e800b960 comp rx=0 tx=0).stop 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.318+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 shutdown_connections 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.318+0000 7f4502400640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f44dc06bc20 0x7f44dc06e0e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.318+0000 7f4502400640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc15b8e0 0x7f44fc0c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.318+0000 7f4502400640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44fc158ee0 0x7f44fc152e50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.318+0000 7f4502400640 1 --2- 172.21.15.102:0/3588795826 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f44fc1524c0 0x7f44fc1f08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.318+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 >> 172.21.15.102:0/3588795826 conn(0x7f44fc0700d0 msgr2=0x7f44fc073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.318+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 shutdown_connections 2024-03-18T16:36:00.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:00.318+0000 7f4502400640 1 -- 172.21.15.102:0/3588795826 wait complete. 2024-03-18T16:36:00.941 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:00.941 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:01.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:02.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.117+0000 7f1d23e00640 1 Processor -- start 2024-03-18T16:36:02.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.117+0000 7f1d23e00640 1 -- start start 2024-03-18T16:36:02.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.118+0000 7f1d23e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1d2415a6f0 0x7f1d2415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:02.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.118+0000 7f1d23e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415baa0 0x7f1d2415bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:02.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.118+0000 7f1d23e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d2415c470 con 0x7f1d2415a310 2024-03-18T16:36:02.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.118+0000 7f1d23e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f1d240cd410 con 0x7f1d2415a6f0 2024-03-18T16:36:02.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.118+0000 7f1d23e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1d240ca500 con 0x7f1d2415baa0 2024-03-18T16:36:02.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.119+0000 7f1d22a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f1d2415a310 0x7f1d2406d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:50532/0 (socket says 172.21.15.102:50532) 2024-03-18T16:36:02.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.119+0000 7f1d22a00640 1 -- 172.21.15.102:0/1150231168 learned_addr learned my addr 172.21.15.102:0/1150231168 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:02.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.119+0000 7f1d23400640 1 --2- 172.21.15.102:0/1150231168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415baa0 0x7f1d2415bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:02.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.119+0000 7f1d22000640 1 --2- 172.21.15.102:0/1150231168 >> v2:172.21.15.102:3300/0 conn(0x7f1d2415a6f0 0x7f1d2415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:02.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.122+0000 7f1d21600640 1 -- 172.21.15.102:0/1150231168 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3925391484 0 0) 0x7f1d2415c470 con 0x7f1d2415a310 2024-03-18T16:36:02.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.122+0000 7f1d21600640 1 -- 172.21.15.102:0/1150231168 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d08003520 con 0x7f1d2415a310 2024-03-18T16:36:02.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.122+0000 7f1d23400640 1 -- 172.21.15.102:0/1150231168 >> v2:172.21.15.102:3300/0 conn(0x7f1d2415a6f0 msgr2=0x7f1d2415ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:02.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.122+0000 7f1d23400640 1 --2- 172.21.15.102:0/1150231168 >> v2:172.21.15.102:3300/0 conn(0x7f1d2415a6f0 0x7f1d2415ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.122+0000 7f1d23400640 1 -- 172.21.15.102:0/1150231168 >> v1:172.21.15.154:6789/0 conn(0x7f1d2415a310 legacy=0x7f1d2406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:02.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.122+0000 7f1d23400640 1 -- 172.21.15.102:0/1150231168 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2415b080 con 0x7f1d2415baa0 2024-03-18T16:36:02.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.124+0000 7f1d23400640 1 --2- 172.21.15.102:0/1150231168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415baa0 0x7f1d2415bf30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009d90 tx=0x7f1d1c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3175cafa12eb504c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:02.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.129+0000 7f1d21600640 1 -- 172.21.15.102:0/1150231168 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c039070 con 0x7f1d2415baa0 2024-03-18T16:36:02.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.129+0000 7f1d21600640 1 -- 172.21.15.102:0/1150231168 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d1c02cc60 con 0x7f1d2415baa0 2024-03-18T16:36:02.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.129+0000 7f1d21600640 1 -- 172.21.15.102:0/1150231168 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c034660 con 0x7f1d2415baa0 2024-03-18T16:36:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.130+0000 7f1d23e00640 1 -- 172.21.15.102:0/1150231168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415baa0 msgr2=0x7f1d2415bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.130+0000 7f1d23e00640 1 --2- 172.21.15.102:0/1150231168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415baa0 0x7f1d2415bf30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c009d90 tx=0x7f1d1c02beb0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.130+0000 7f1d23e00640 1 -- 172.21.15.102:0/1150231168 shutdown_connections 2024-03-18T16:36:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.130+0000 7f1d23e00640 1 --2- 172.21.15.102:0/1150231168 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415baa0 0x7f1d2415bf30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.130+0000 7f1d23e00640 1 --2- 172.21.15.102:0/1150231168 >> v2:172.21.15.102:3300/0 conn(0x7f1d2415a6f0 0x7f1d2415ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.130+0000 7f1d23e00640 1 -- 172.21.15.102:0/1150231168 >> 172.21.15.102:0/1150231168 conn(0x7f1d2408c1a0 msgr2=0x7f1d24066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.130+0000 7f1d23e00640 1 -- 172.21.15.102:0/1150231168 shutdown_connections 2024-03-18T16:36:02.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.130+0000 7f1d23e00640 1 -- 172.21.15.102:0/1150231168 wait complete. 2024-03-18T16:36:02.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.133+0000 7f1d23e00640 1 Processor -- start 2024-03-18T16:36:02.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.133+0000 7f1d23e00640 1 -- start start 2024-03-18T16:36:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.135+0000 7f1d23e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d2415a310 0x7f1d241f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.135+0000 7f1d23e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415a6f0 0x7f1d240c6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.135+0000 7f1d23e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d2415baa0 0x7f1d240cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.135+0000 7f1d23e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f1d240cce50 con 0x7f1d2415a310 2024-03-18T16:36:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.135+0000 7f1d23e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1d240caa80 con 0x7f1d2415baa0 2024-03-18T16:36:02.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.135+0000 7f1d23e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1d240cd130 con 0x7f1d2415a6f0 2024-03-18T16:36:02.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.135+0000 7f1d22000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415a6f0 0x7f1d240c6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:02.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.135+0000 7f1d23400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d2415baa0 0x7f1d240cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:02.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.135+0000 7f1d22a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d2415a310 0x7f1d241f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:02.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.136+0000 7f1d22000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415a6f0 0x7f1d240c6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:49384/0 (socket says 172.21.15.102:49384) 2024-03-18T16:36:02.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.136+0000 7f1d22000640 1 -- 172.21.15.102:0/2208887577 learned_addr learned my addr 172.21.15.102:0/2208887577 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.139+0000 7f1d22000640 1 -- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d2415baa0 msgr2=0x7f1d240cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.139+0000 7f1d22000640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d2415baa0 0x7f1d240cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.139+0000 7f1d22000640 1 -- 172.21.15.102:0/2208887577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d2415a310 msgr2=0x7f1d241f47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.139+0000 7f1d22000640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d2415a310 0x7f1d241f47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.139+0000 7f1d22000640 1 -- 172.21.15.102:0/2208887577 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d240cff20 con 0x7f1d2415a6f0 2024-03-18T16:36:02.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.142+0000 7f1d22000640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415a6f0 0x7f1d240c6e80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000e8a0 tx=0x7f1d1000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:02.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.145+0000 7f1d1aa00640 1 -- 172.21.15.102:0/2208887577 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d10017070 con 0x7f1d2415a6f0 2024-03-18T16:36:02.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.145+0000 7f1d23e00640 1 -- 172.21.15.102:0/2208887577 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d241f4ce0 con 0x7f1d2415a6f0 2024-03-18T16:36:02.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.145+0000 7f1d23e00640 1 -- 172.21.15.102:0/2208887577 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d240cf960 con 0x7f1d2415a6f0 2024-03-18T16:36:02.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.147+0000 7f1d0fe00640 1 -- 172.21.15.102:0/2208887577 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d24074fd0 con 0x7f1d2415a6f0 2024-03-18T16:36:02.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.171+0000 7f1d1aa00640 1 -- 172.21.15.102:0/2208887577 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d10005990 con 0x7f1d2415a6f0 2024-03-18T16:36:02.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.171+0000 7f1d1aa00640 1 -- 172.21.15.102:0/2208887577 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d10021060 con 0x7f1d2415a6f0 2024-03-18T16:36:02.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.173+0000 7f1d1aa00640 1 -- 172.21.15.102:0/2208887577 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1d1001a070 con 0x7f1d2415a6f0 2024-03-18T16:36:02.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.176+0000 7f1d1aa00640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1d0406bbd0 0x7f1d0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:02.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.176+0000 7f1d22a00640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1d0406bbd0 0x7f1d0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:02.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.177+0000 7f1d22a00640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1d0406bbd0 0x7f1d0406e090 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1d14005880 tx=0x7f1d14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.420+0000 7f1d1aa00640 1 -- 172.21.15.102:0/2208887577 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f1d10057580 con 0x7f1d2415a6f0 2024-03-18T16:36:02.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.438+0000 7f1d1aa00640 1 -- 172.21.15.102:0/2208887577 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1d100594e0 con 0x7f1d2415a6f0 2024-03-18T16:36:02.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.711+0000 7f1d0fe00640 1 -- 172.21.15.102:0/2208887577 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d240d00f0 con 0x7f1d2415a6f0 2024-03-18T16:36:02.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.789+0000 7f1d1aa00640 1 -- 172.21.15.102:0/2208887577 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f1d1005d580 con 0x7f1d2415a6f0 2024-03-18T16:36:02.791 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:02.791 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:02.791 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 -- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1d0406bbd0 msgr2=0x7f1d0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1d0406bbd0 0x7f1d0406e090 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1d14005880 tx=0x7f1d14009040 comp rx=0 tx=0).stop 2024-03-18T16:36:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 -- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415a6f0 msgr2=0x7f1d240c6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415a6f0 0x7f1d240c6e80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f1d1000e8a0 tx=0x7f1d1000ed60 comp rx=0 tx=0).stop 2024-03-18T16:36:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 -- 172.21.15.102:0/2208887577 shutdown_connections 2024-03-18T16:36:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f1d0406bbd0 0x7f1d0406e090 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d2415baa0 0x7f1d240cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f1d2415a6f0 0x7f1d240c6e80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 --2- 172.21.15.102:0/2208887577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d2415a310 0x7f1d241f47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 -- 172.21.15.102:0/2208887577 >> 172.21.15.102:0/2208887577 conn(0x7f1d2408c1a0 msgr2=0x7f1d2414ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 -- 172.21.15.102:0/2208887577 shutdown_connections 2024-03-18T16:36:02.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:02.791+0000 7f1d0fe00640 1 -- 172.21.15.102:0/2208887577 wait complete. 2024-03-18T16:36:03.421 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:03.421 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:04.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:04.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.595+0000 7fa080a00640 1 Processor -- start 2024-03-18T16:36:04.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.595+0000 7fa080a00640 1 -- start start 2024-03-18T16:36:04.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.596+0000 7fa080a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa07c06de50 0x7fa07c15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:04.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.596+0000 7fa080a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa07c154150 0x7fa07c154580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:04.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.596+0000 7fa080a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa07c06bf60 con 0x7fa07c06b470 2024-03-18T16:36:04.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.596+0000 7fa080a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa07c0ca0a0 con 0x7fa07c154150 2024-03-18T16:36:04.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.596+0000 7fa080a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa07c0c94b0 con 0x7fa07c06de50 2024-03-18T16:36:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.597+0000 7fa079e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa07c06de50 0x7fa07c15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.597+0000 7fa07b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa07c154150 0x7fa07c154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.598+0000 7fa079e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa07c06de50 0x7fa07c15ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41444/0 (socket says 172.21.15.102:41444) 2024-03-18T16:36:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.598+0000 7fa079e00640 1 -- 172.21.15.102:0/897537519 learned_addr learned my addr 172.21.15.102:0/897537519 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.601+0000 7fa079e00640 1 -- 172.21.15.102:0/897537519 >> v2:172.21.15.102:3300/0 conn(0x7fa07c154150 msgr2=0x7fa07c154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.601+0000 7fa079e00640 1 --2- 172.21.15.102:0/897537519 >> v2:172.21.15.102:3300/0 conn(0x7fa07c154150 0x7fa07c154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.601+0000 7fa079e00640 1 -- 172.21.15.102:0/897537519 >> v1:172.21.15.102:6789/0 conn(0x7fa07c06b470 legacy=0x7fa07c06b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.601+0000 7fa079e00640 1 -- 172.21.15.102:0/897537519 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07c0c9770 con 0x7fa07c06de50 2024-03-18T16:36:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.604+0000 7fa079e00640 1 --2- 172.21.15.102:0/897537519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa07c06de50 0x7fa07c15ad60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa07c06c1e0 tx=0x7fa06802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9beda2afad270dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.608+0000 7fa079400640 1 -- 172.21.15.102:0/897537519 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068034070 con 0x7fa07c06de50 2024-03-18T16:36:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.608+0000 7fa079400640 1 -- 172.21.15.102:0/897537519 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa068030860 con 0x7fa07c06de50 2024-03-18T16:36:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.608+0000 7fa079400640 1 -- 172.21.15.102:0/897537519 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068030c30 con 0x7fa07c06de50 2024-03-18T16:36:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.609+0000 7fa080a00640 1 -- 172.21.15.102:0/897537519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa07c06de50 msgr2=0x7fa07c15ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.609+0000 7fa080a00640 1 --2- 172.21.15.102:0/897537519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa07c06de50 0x7fa07c15ad60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa07c06c1e0 tx=0x7fa06802b9a0 comp rx=0 tx=0).stop 2024-03-18T16:36:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.609+0000 7fa080a00640 1 -- 172.21.15.102:0/897537519 shutdown_connections 2024-03-18T16:36:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.609+0000 7fa080a00640 1 --2- 172.21.15.102:0/897537519 >> v2:172.21.15.102:3300/0 conn(0x7fa07c154150 0x7fa07c154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.609+0000 7fa080a00640 1 --2- 172.21.15.102:0/897537519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa07c06de50 0x7fa07c15ad60 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.609+0000 7fa080a00640 1 -- 172.21.15.102:0/897537519 >> 172.21.15.102:0/897537519 conn(0x7fa07c0727e0 msgr2=0x7fa07c072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.609+0000 7fa080a00640 1 -- 172.21.15.102:0/897537519 shutdown_connections 2024-03-18T16:36:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.609+0000 7fa080a00640 1 -- 172.21.15.102:0/897537519 wait complete. 2024-03-18T16:36:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.611+0000 7fa080a00640 1 Processor -- start 2024-03-18T16:36:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.611+0000 7fa080a00640 1 -- start start 2024-03-18T16:36:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.612+0000 7fa080a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c06b470 0x7fa07c1f0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.612+0000 7fa080a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa07c06de50 0x7fa07c1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.612+0000 7fa080a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa07c154150 0x7fa07c156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.612+0000 7fa080a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa07c0c5190 con 0x7fa07c154150 2024-03-18T16:36:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.612+0000 7fa080a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa07c0c9d30 con 0x7fa07c06de50 2024-03-18T16:36:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.612+0000 7fa080a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa07c0c5310 con 0x7fa07c06b470 2024-03-18T16:36:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.613+0000 7fa079e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa07c06de50 0x7fa07c1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:04.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.613+0000 7fa07a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c06b470 0x7fa07c1f0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.613+0000 7fa07a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c06b470 0x7fa07c1f0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:49386/0 (socket says 172.21.15.102:49386) 2024-03-18T16:36:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.613+0000 7fa07a800640 1 -- 172.21.15.102:0/1487119177 learned_addr learned my addr 172.21.15.102:0/1487119177 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:04.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.613+0000 7fa07b200640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa07c154150 0x7fa07c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.617+0000 7fa07a800640 1 -- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa07c06de50 msgr2=0x7fa07c1526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.617+0000 7fa07a800640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa07c06de50 0x7fa07c1526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.617+0000 7fa07a800640 1 -- 172.21.15.102:0/1487119177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa07c154150 msgr2=0x7fa07c156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.617+0000 7fa07a800640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa07c154150 0x7fa07c156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.617+0000 7fa07a800640 1 -- 172.21.15.102:0/1487119177 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07c155260 con 0x7fa07c06b470 2024-03-18T16:36:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.619+0000 7fa07a800640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c06b470 0x7fa07c1f0680 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00c430 tx=0x7fa06c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:04.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.623+0000 7fa072a00640 1 -- 172.21.15.102:0/1487119177 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06c0052d0 con 0x7fa07c06b470 2024-03-18T16:36:04.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.623+0000 7fa072a00640 1 -- 172.21.15.102:0/1487119177 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa06c005490 con 0x7fa07c06b470 2024-03-18T16:36:04.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.623+0000 7fa072a00640 1 -- 172.21.15.102:0/1487119177 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06c007260 con 0x7fa07c06b470 2024-03-18T16:36:04.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.623+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa07c155500 con 0x7fa07c06b470 2024-03-18T16:36:04.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.624+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa07c156150 con 0x7fa07c06b470 2024-03-18T16:36:04.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.625+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa07c074fd0 con 0x7fa07c06b470 2024-03-18T16:36:04.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.645+0000 7fa072a00640 1 -- 172.21.15.102:0/1487119177 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa06c007400 con 0x7fa07c06b470 2024-03-18T16:36:04.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.646+0000 7fa072a00640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa05c06bc20 0x7fa05c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:04.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.647+0000 7fa079e00640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa05c06bc20 0x7fa05c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:04.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.647+0000 7fa079e00640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa05c06bc20 0x7fa05c06e0e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa068036000 tx=0x7fa068008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:04.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.648+0000 7fa072a00640 1 -- 172.21.15.102:0/1487119177 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa06c056cb0 con 0x7fa07c06b470 2024-03-18T16:36:04.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:04.909+0000 7fa072a00640 1 -- 172.21.15.102:0/1487119177 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa06c058c10 con 0x7fa07c06b470 2024-03-18T16:36:05.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.195+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa07c0c7c70 con 0x7fa07c06b470 2024-03-18T16:36:05.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.209+0000 7fa072a00640 1 -- 172.21.15.102:0/1487119177 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa06c05ccb0 con 0x7fa07c06b470 2024-03-18T16:36:05.210 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:05.210 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:05.211 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:05.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.211+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa05c06bc20 msgr2=0x7fa05c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:05.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.211+0000 7fa080a00640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa05c06bc20 0x7fa05c06e0e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa068036000 tx=0x7fa068008030 comp rx=0 tx=0).stop 2024-03-18T16:36:05.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.211+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c06b470 msgr2=0x7fa07c1f0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:05.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.211+0000 7fa080a00640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c06b470 0x7fa07c1f0680 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00c430 tx=0x7fa06c00c8f0 comp rx=0 tx=0).stop 2024-03-18T16:36:05.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.211+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 shutdown_connections 2024-03-18T16:36:05.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.211+0000 7fa080a00640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa05c06bc20 0x7fa05c06e0e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:05.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.211+0000 7fa080a00640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa07c154150 0x7fa07c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:05.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.211+0000 7fa080a00640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa07c06de50 0x7fa07c1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:05.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.211+0000 7fa080a00640 1 --2- 172.21.15.102:0/1487119177 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa07c06b470 0x7fa07c1f0680 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:05.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.212+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 >> 172.21.15.102:0/1487119177 conn(0x7fa07c0727e0 msgr2=0x7fa07c072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:05.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.212+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 shutdown_connections 2024-03-18T16:36:05.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:05.212+0000 7fa080a00640 1 -- 172.21.15.102:0/1487119177 wait complete. 2024-03-18T16:36:05.834 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:05.834 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:06.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:07.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.013+0000 7f02abe00640 1 Processor -- start 2024-03-18T16:36:07.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.014+0000 7f02abe00640 1 -- start start 2024-03-18T16:36:07.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.015+0000 7f02abe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f02ac158c00 0x7f02ac06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:07.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.015+0000 7f02abe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f02ac06fd80 0x7f02ac072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:07.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.015+0000 7f02abe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac071700 0x7f02ac072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:07.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.015+0000 7f02abe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f02ac158fe0 con 0x7f02ac06fd80 2024-03-18T16:36:07.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.015+0000 7f02abe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f02ac06f3b0 con 0x7f02ac158c00 2024-03-18T16:36:07.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.015+0000 7f02abe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f02ac0c4190 con 0x7f02ac071700 2024-03-18T16:36:07.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.016+0000 7f02aaa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f02ac158c00 0x7f02ac06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:07.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.016+0000 7f02aaa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f02ac158c00 0x7f02ac06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41460/0 (socket says 172.21.15.102:41460) 2024-03-18T16:36:07.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.016+0000 7f02aaa00640 1 -- 172.21.15.102:0/1181917936 learned_addr learned my addr 172.21.15.102:0/1181917936 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:07.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.016+0000 7f02ab400640 1 --2- 172.21.15.102:0/1181917936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac071700 0x7f02ac072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:07.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.018+0000 7f02aaa00640 1 -- 172.21.15.102:0/1181917936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac071700 msgr2=0x7f02ac072fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:07.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.018+0000 7f02aaa00640 1 --2- 172.21.15.102:0/1181917936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac071700 0x7f02ac072fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.018+0000 7f02aaa00640 1 -- 172.21.15.102:0/1181917936 >> v2:172.21.15.102:3300/0 conn(0x7f02ac06fd80 msgr2=0x7f02ac072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:07.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.018+0000 7f02aaa00640 1 --2- 172.21.15.102:0/1181917936 >> v2:172.21.15.102:3300/0 conn(0x7f02ac06fd80 0x7f02ac072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.018+0000 7f02aaa00640 1 -- 172.21.15.102:0/1181917936 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac06f060 con 0x7f02ac158c00 2024-03-18T16:36:07.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.021+0000 7f02aaa00640 1 --2- 172.21.15.102:0/1181917936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 0x7f02ac06f840 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f02ac067fa0 tx=0x7f029c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4f9883880da1740 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:07.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.023+0000 7f02a9600640 1 -- 172.21.15.102:0/1181917936 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c035020 con 0x7f02ac158c00 2024-03-18T16:36:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.026+0000 7f02a9600640 1 -- 172.21.15.102:0/1181917936 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f029c031870 con 0x7f02ac158c00 2024-03-18T16:36:07.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.026+0000 7f02a9600640 1 -- 172.21.15.102:0/1181917936 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c031c40 con 0x7f02ac158c00 2024-03-18T16:36:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.026+0000 7f02abe00640 1 -- 172.21.15.102:0/1181917936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 msgr2=0x7f02ac06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.026+0000 7f02abe00640 1 --2- 172.21.15.102:0/1181917936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 0x7f02ac06f840 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f02ac067fa0 tx=0x7f029c02b910 comp rx=0 tx=0).stop 2024-03-18T16:36:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.026+0000 7f02abe00640 1 -- 172.21.15.102:0/1181917936 shutdown_connections 2024-03-18T16:36:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.026+0000 7f02abe00640 1 --2- 172.21.15.102:0/1181917936 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac071700 0x7f02ac072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.026+0000 7f02abe00640 1 --2- 172.21.15.102:0/1181917936 >> v2:172.21.15.102:3300/0 conn(0x7f02ac06fd80 0x7f02ac072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.026+0000 7f02abe00640 1 --2- 172.21.15.102:0/1181917936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 0x7f02ac06f840 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.026+0000 7f02abe00640 1 -- 172.21.15.102:0/1181917936 >> 172.21.15.102:0/1181917936 conn(0x7f02ac0cb890 msgr2=0x7f02ac0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.027+0000 7f02abe00640 1 -- 172.21.15.102:0/1181917936 shutdown_connections 2024-03-18T16:36:07.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.027+0000 7f02abe00640 1 -- 172.21.15.102:0/1181917936 wait complete. 2024-03-18T16:36:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.028+0000 7f02abe00640 1 Processor -- start 2024-03-18T16:36:07.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.028+0000 7f02abe00640 1 -- start start 2024-03-18T16:36:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.029+0000 7f02abe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06fd80 0x7f02ac1f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.029+0000 7f02abe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02ac071700 0x7f02ac1e95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.029+0000 7f02abe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 0x7f02ac15e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.029+0000 7f02abe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac072270 con 0x7f02ac071700 2024-03-18T16:36:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.029+0000 7f02abe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac071f90 con 0x7f02ac158c00 2024-03-18T16:36:07.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.029+0000 7f02abe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f02ac072550 con 0x7f02ac06fd80 2024-03-18T16:36:07.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.030+0000 7f02ab400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 0x7f02ac15e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:07.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.030+0000 7f02aaa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06fd80 0x7f02ac1f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:07.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.030+0000 7f02ab400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 0x7f02ac15e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41470/0 (socket says 172.21.15.102:41470) 2024-03-18T16:36:07.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.030+0000 7f02ab400640 1 -- 172.21.15.102:0/1229547552 learned_addr learned my addr 172.21.15.102:0/1229547552 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.035+0000 7f02aaa00640 1 -- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 msgr2=0x7f02ac15e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.035+0000 7f02aaa00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 0x7f02ac15e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.035+0000 7f02aaa00640 1 -- 172.21.15.102:0/1229547552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02ac071700 msgr2=0x7f02ac1e95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.035+0000 7f02aaa00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02ac071700 0x7f02ac1e95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.035+0000 7f02aaa00640 1 -- 172.21.15.102:0/1229547552 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac15ca70 con 0x7f02ac06fd80 2024-03-18T16:36:07.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.038+0000 7f02aaa00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06fd80 0x7f02ac1f54b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f029c033000 tx=0x7f029c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.042+0000 7f02a2a00640 1 -- 172.21.15.102:0/1229547552 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c035040 con 0x7f02ac06fd80 2024-03-18T16:36:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.042+0000 7f02a2a00640 1 -- 172.21.15.102:0/1229547552 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f029c030d00 con 0x7f02ac06fd80 2024-03-18T16:36:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.042+0000 7f02a2a00640 1 -- 172.21.15.102:0/1229547552 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c0456c0 con 0x7f02ac06fd80 2024-03-18T16:36:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.042+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ac15cd50 con 0x7f02ac06fd80 2024-03-18T16:36:07.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.042+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ac15d310 con 0x7f02ac06fd80 2024-03-18T16:36:07.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.044+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02ac15d550 con 0x7f02ac06fd80 2024-03-18T16:36:07.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.301+0000 7f02a2a00640 1 -- 172.21.15.102:0/1229547552 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f029c038020 con 0x7f02ac06fd80 2024-03-18T16:36:07.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.302+0000 7f02a2a00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f028c06bc20 0x7f028c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:07.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.302+0000 7f02aa000640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f028c06bc20 0x7f028c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:07.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.303+0000 7f02aa000640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f028c06bc20 0x7f028c06e0e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f02ac1f2660 tx=0x7f0298006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:07.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.304+0000 7f02a2a00640 1 -- 172.21.15.102:0/1229547552 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f029c079780 con 0x7f02ac06fd80 2024-03-18T16:36:07.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.315+0000 7f02a2a00640 1 -- 172.21.15.102:0/1229547552 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f02ac15d550 con 0x7f02ac06fd80 2024-03-18T16:36:07.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.555+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02ac0cae70 con 0x7f02ac06fd80 2024-03-18T16:36:07.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.568+0000 7f02a2a00640 1 -- 172.21.15.102:0/1229547552 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f02ac0cae70 con 0x7f02ac06fd80 2024-03-18T16:36:07.569 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:07.569 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:07.570 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f028c06bc20 msgr2=0x7f028c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f028c06bc20 0x7f028c06e0e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f02ac1f2660 tx=0x7f0298006bf0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06fd80 msgr2=0x7f02ac1f54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06fd80 0x7f02ac1f54b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f029c033000 tx=0x7f029c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 shutdown_connections 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f028c06bc20 0x7f028c06e0e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02ac158c00 0x7f02ac15e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f02ac071700 0x7f02ac1e95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 --2- 172.21.15.102:0/1229547552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f02ac06fd80 0x7f02ac1f54b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 >> 172.21.15.102:0/1229547552 conn(0x7f02ac0cb890 msgr2=0x7f02ac1531b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 shutdown_connections 2024-03-18T16:36:07.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:07.570+0000 7f02abe00640 1 -- 172.21.15.102:0/1229547552 wait complete. 2024-03-18T16:36:08.192 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:08.193 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:09.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.405+0000 7fb882000640 1 Processor -- start 2024-03-18T16:36:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.405+0000 7fb882000640 1 -- start start 2024-03-18T16:36:09.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.406+0000 7fb882000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c157d90 0x7fb87c0c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:09.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.406+0000 7fb882000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb87c073530 con 0x7fb87c072810 2024-03-18T16:36:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.406+0000 7fb882000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb87c0c57e0 con 0x7fb87c0c7f30 2024-03-18T16:36:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.406+0000 7fb882000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb87c0c5d80 con 0x7fb87c157d90 2024-03-18T16:36:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.407+0000 7fb880c00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb87c0c7f30 0x7fb87c15b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:50534/0 (socket says 172.21.15.102:50534) 2024-03-18T16:36:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.407+0000 7fb880c00640 1 -- 172.21.15.102:0/3974778664 learned_addr learned my addr 172.21.15.102:0/3974778664 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.407+0000 7fb87be00640 1 --2- 172.21.15.102:0/3974778664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c157d90 0x7fb87c0c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:09.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.410+0000 7fb87b400640 1 -- 172.21.15.102:0/3974778664 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062096723 0 0) 0x7fb87c0c57e0 con 0x7fb87c0c7f30 2024-03-18T16:36:09.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.410+0000 7fb87b400640 1 -- 172.21.15.102:0/3974778664 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb864005730 con 0x7fb87c0c7f30 2024-03-18T16:36:09.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.411+0000 7fb87be00640 1 -- 172.21.15.102:0/3974778664 >> v1:172.21.15.154:6789/0 conn(0x7fb87c0c7f30 legacy=0x7fb87c15b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:09.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.411+0000 7fb87be00640 1 -- 172.21.15.102:0/3974778664 >> v1:172.21.15.102:6789/0 conn(0x7fb87c072810 legacy=0x7fb87c161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:09.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.411+0000 7fb87be00640 1 -- 172.21.15.102:0/3974778664 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c0c5aa0 con 0x7fb87c157d90 2024-03-18T16:36:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.414+0000 7fb87be00640 1 --2- 172.21.15.102:0/3974778664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c157d90 0x7fb87c0c2d90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb8680097d0 tx=0x7fb86802b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29317da63b9af189 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.419+0000 7fb87b400640 1 -- 172.21.15.102:0/3974778664 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868034070 con 0x7fb87c157d90 2024-03-18T16:36:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.419+0000 7fb87b400640 1 -- 172.21.15.102:0/3974778664 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb868030890 con 0x7fb87c157d90 2024-03-18T16:36:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.419+0000 7fb87b400640 1 -- 172.21.15.102:0/3974778664 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868030c60 con 0x7fb87c157d90 2024-03-18T16:36:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.420+0000 7fb882000640 1 -- 172.21.15.102:0/3974778664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c157d90 msgr2=0x7fb87c0c2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.420+0000 7fb882000640 1 --2- 172.21.15.102:0/3974778664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c157d90 0x7fb87c0c2d90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb8680097d0 tx=0x7fb86802b9d0 comp rx=0 tx=0).stop 2024-03-18T16:36:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.420+0000 7fb882000640 1 -- 172.21.15.102:0/3974778664 shutdown_connections 2024-03-18T16:36:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.420+0000 7fb882000640 1 --2- 172.21.15.102:0/3974778664 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c157d90 0x7fb87c0c2d90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.420+0000 7fb882000640 1 -- 172.21.15.102:0/3974778664 >> 172.21.15.102:0/3974778664 conn(0x7fb87c0cb890 msgr2=0x7fb87c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.420+0000 7fb882000640 1 -- 172.21.15.102:0/3974778664 shutdown_connections 2024-03-18T16:36:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.420+0000 7fb882000640 1 -- 172.21.15.102:0/3974778664 wait complete. 2024-03-18T16:36:09.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.422+0000 7fb882000640 1 Processor -- start 2024-03-18T16:36:09.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.422+0000 7fb882000640 1 -- start start 2024-03-18T16:36:09.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.423+0000 7fb882000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c072810 0x7fb87c144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.423+0000 7fb882000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb87c0c7f30 0x7fb87c1eccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.423+0000 7fb882000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb87c157d90 0x7fb87c1e0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.423+0000 7fb882000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c157590 con 0x7fb87c0c7f30 2024-03-18T16:36:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.423+0000 7fb882000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c157900 con 0x7fb87c157d90 2024-03-18T16:36:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.423+0000 7fb882000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb87c0c2940 con 0x7fb87c072810 2024-03-18T16:36:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.424+0000 7fb881600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb87c157d90 0x7fb87c1e0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.424+0000 7fb87be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb87c0c7f30 0x7fb87c1eccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:09.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.424+0000 7fb881600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb87c157d90 0x7fb87c1e0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:41474/0 (socket says 172.21.15.102:41474) 2024-03-18T16:36:09.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.424+0000 7fb881600640 1 -- 172.21.15.102:0/70209929 learned_addr learned my addr 172.21.15.102:0/70209929 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:09.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.427+0000 7fb881600640 1 -- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c072810 msgr2=0x7fb87c144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:09.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.427+0000 7fb881600640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c072810 0x7fb87c144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:09.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.427+0000 7fb881600640 1 -- 172.21.15.102:0/70209929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb87c0c7f30 msgr2=0x7fb87c1eccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:09.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.427+0000 7fb881600640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb87c0c7f30 0x7fb87c1eccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:09.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.427+0000 7fb881600640 1 -- 172.21.15.102:0/70209929 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c14fe70 con 0x7fb87c157d90 2024-03-18T16:36:09.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.430+0000 7fb881600640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb87c157d90 0x7fb87c1e0530 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb87400c710 tx=0x7fb87400cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:09.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.434+0000 7fb878c00640 1 -- 172.21.15.102:0/70209929 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874016020 con 0x7fb87c157d90 2024-03-18T16:36:09.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.434+0000 7fb878c00640 1 -- 172.21.15.102:0/70209929 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb874006d50 con 0x7fb87c157d90 2024-03-18T16:36:09.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.434+0000 7fb878c00640 1 -- 172.21.15.102:0/70209929 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874003bb0 con 0x7fb87c157d90 2024-03-18T16:36:09.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.434+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87c145900 con 0x7fb87c157d90 2024-03-18T16:36:09.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.434+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87c145bc0 con 0x7fb87c157d90 2024-03-18T16:36:09.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.435+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb87c074fd0 con 0x7fb87c157d90 2024-03-18T16:36:09.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.458+0000 7fb878c00640 1 -- 172.21.15.102:0/70209929 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb87400a850 con 0x7fb87c157d90 2024-03-18T16:36:09.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.459+0000 7fb878c00640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb85c06bc20 0x7fb85c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:09.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.460+0000 7fb880c00640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb85c06bc20 0x7fb85c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.460+0000 7fb878c00640 1 -- 172.21.15.102:0/70209929 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb874057950 con 0x7fb87c157d90 2024-03-18T16:36:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.460+0000 7fb880c00640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb85c06bc20 0x7fb85c06e0e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb870001610 tx=0x7fb870007930 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:09.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.712+0000 7fb878c00640 1 -- 172.21.15.102:0/70209929 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb874020020 con 0x7fb87c157d90 2024-03-18T16:36:09.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.947+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb87c14ca50 con 0x7fb87c157d90 2024-03-18T16:36:09.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.962+0000 7fb878c00640 1 -- 172.21.15.102:0/70209929 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb874059fb0 con 0x7fb87c157d90 2024-03-18T16:36:09.963 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:09.964 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:09.964 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.964+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb85c06bc20 msgr2=0x7fb85c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.964+0000 7fb882000640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb85c06bc20 0x7fb85c06e0e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb870001610 tx=0x7fb870007930 comp rx=0 tx=0).stop 2024-03-18T16:36:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.964+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb87c157d90 msgr2=0x7fb87c1e0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.964+0000 7fb882000640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb87c157d90 0x7fb87c1e0530 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fb87400c710 tx=0x7fb87400cbd0 comp rx=0 tx=0).stop 2024-03-18T16:36:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.965+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 shutdown_connections 2024-03-18T16:36:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.965+0000 7fb882000640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb85c06bc20 0x7fb85c06e0e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.965+0000 7fb882000640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb87c157d90 0x7fb87c1e0530 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.965+0000 7fb882000640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb87c0c7f30 0x7fb87c1eccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.965+0000 7fb882000640 1 --2- 172.21.15.102:0/70209929 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb87c072810 0x7fb87c144530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:09.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.965+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 >> 172.21.15.102:0/70209929 conn(0x7fb87c0cb890 msgr2=0x7fb87c0c9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:09.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.966+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 shutdown_connections 2024-03-18T16:36:09.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:09.966+0000 7fb882000640 1 -- 172.21.15.102:0/70209929 wait complete. 2024-03-18T16:36:10.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:10.589 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:11.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:11.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.774+0000 7f541cc00640 1 Processor -- start 2024-03-18T16:36:11.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.774+0000 7f541cc00640 1 -- start start 2024-03-18T16:36:11.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.775+0000 7f541cc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5418141650 0x7f5418158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.775+0000 7f541cc00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f541815c3a0 0x7f54180c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.775+0000 7f541cc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5418158ad0 con 0x7f54180c4fa0 2024-03-18T16:36:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.775+0000 7f541cc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5418159370 con 0x7f5418141650 2024-03-18T16:36:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.775+0000 7f541cc00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5418158db0 con 0x7f541815c3a0 2024-03-18T16:36:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.776+0000 7f5417400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5418141650 0x7f5418158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:11.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.776+0000 7f5416a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f541815c3a0 0x7f54180c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:11.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.777+0000 7f5416a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f541815c3a0 0x7f54180c6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57734/0 (socket says 172.21.15.102:57734) 2024-03-18T16:36:11.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.777+0000 7f5416a00640 1 -- 172.21.15.102:0/3654567119 learned_addr learned my addr 172.21.15.102:0/3654567119 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.780+0000 7f5416a00640 1 -- 172.21.15.102:0/3654567119 >> v2:172.21.15.102:3300/0 conn(0x7f5418141650 msgr2=0x7f5418158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.780+0000 7f5416a00640 1 --2- 172.21.15.102:0/3654567119 >> v2:172.21.15.102:3300/0 conn(0x7f5418141650 0x7f5418158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.780+0000 7f5416a00640 1 -- 172.21.15.102:0/3654567119 >> v1:172.21.15.102:6789/0 conn(0x7f54180c4fa0 legacy=0x7f541806f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:11.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.780+0000 7f5416a00640 1 -- 172.21.15.102:0/3654567119 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5418159090 con 0x7f541815c3a0 2024-03-18T16:36:11.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.783+0000 7f5416a00640 1 --2- 172.21.15.102:0/3654567119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541815c3a0 0x7f54180c6bc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5418068e80 tx=0x7f540402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed41097b5caab5b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:11.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.787+0000 7f5416000640 1 -- 172.21.15.102:0/3654567119 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5404034070 con 0x7f541815c3a0 2024-03-18T16:36:11.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.787+0000 7f5416000640 1 -- 172.21.15.102:0/3654567119 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5404030860 con 0x7f541815c3a0 2024-03-18T16:36:11.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.787+0000 7f5416000640 1 -- 172.21.15.102:0/3654567119 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5404030c30 con 0x7f541815c3a0 2024-03-18T16:36:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.788+0000 7f541cc00640 1 -- 172.21.15.102:0/3654567119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541815c3a0 msgr2=0x7f54180c6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.788+0000 7f541cc00640 1 --2- 172.21.15.102:0/3654567119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541815c3a0 0x7f54180c6bc0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5418068e80 tx=0x7f540402b9a0 comp rx=0 tx=0).stop 2024-03-18T16:36:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.788+0000 7f541cc00640 1 -- 172.21.15.102:0/3654567119 shutdown_connections 2024-03-18T16:36:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.788+0000 7f541cc00640 1 --2- 172.21.15.102:0/3654567119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541815c3a0 0x7f54180c6bc0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.788+0000 7f541cc00640 1 --2- 172.21.15.102:0/3654567119 >> v2:172.21.15.102:3300/0 conn(0x7f5418141650 0x7f5418158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.788+0000 7f541cc00640 1 -- 172.21.15.102:0/3654567119 >> 172.21.15.102:0/3654567119 conn(0x7f54180cb890 msgr2=0x7f54180cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.788+0000 7f541cc00640 1 -- 172.21.15.102:0/3654567119 shutdown_connections 2024-03-18T16:36:11.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.788+0000 7f541cc00640 1 -- 172.21.15.102:0/3654567119 wait complete. 2024-03-18T16:36:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.791+0000 7f541cc00640 1 Processor -- start 2024-03-18T16:36:11.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.791+0000 7f541cc00640 1 -- start start 2024-03-18T16:36:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.793+0000 7f541cc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54180c4fa0 0x7f54181f4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.793+0000 7f541cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418141650 0x7f541813d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.793+0000 7f541cc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f541815c3a0 0x7f5418152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.793+0000 7f541cc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f541815c7c0 con 0x7f5418141650 2024-03-18T16:36:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.793+0000 7f541cc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f54180c66a0 con 0x7f54180c4fa0 2024-03-18T16:36:11.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.793+0000 7f541cc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5418141a30 con 0x7f541815c3a0 2024-03-18T16:36:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.793+0000 7f5417400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54180c4fa0 0x7f54181f4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.793+0000 7f5416a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418141650 0x7f541813d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.793+0000 7f5417e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f541815c3a0 0x7f5418152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.794+0000 7f5417400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54180c4fa0 0x7f54181f4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57746/0 (socket says 172.21.15.102:57746) 2024-03-18T16:36:11.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.794+0000 7f5417400640 1 -- 172.21.15.102:0/886799187 learned_addr learned my addr 172.21.15.102:0/886799187 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:11.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.797+0000 7f5417e00640 1 -- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54180c4fa0 msgr2=0x7f54181f4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:11.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.797+0000 7f5417e00640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54180c4fa0 0x7f54181f4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:11.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.797+0000 7f5417e00640 1 -- 172.21.15.102:0/886799187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418141650 msgr2=0x7f541813d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:11.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.797+0000 7f5417e00640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418141650 0x7f541813d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:11.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.797+0000 7f5417e00640 1 -- 172.21.15.102:0/886799187 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54181f4f40 con 0x7f541815c3a0 2024-03-18T16:36:11.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.800+0000 7f5417e00640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f541815c3a0 0x7f5418152d90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f541000c6c0 tx=0x7f541000cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:11.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.804+0000 7f540f400640 1 -- 172.21.15.102:0/886799187 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410003b70 con 0x7f541815c3a0 2024-03-18T16:36:11.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.804+0000 7f540f400640 1 -- 172.21.15.102:0/886799187 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5410003d10 con 0x7f541815c3a0 2024-03-18T16:36:11.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.804+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541813e7b0 con 0x7f541815c3a0 2024-03-18T16:36:11.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.804+0000 7f540f400640 1 -- 172.21.15.102:0/886799187 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54100052f0 con 0x7f541815c3a0 2024-03-18T16:36:11.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.804+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f54181386b0 con 0x7f541815c3a0 2024-03-18T16:36:11.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.805+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54180c9030 con 0x7f541815c3a0 2024-03-18T16:36:11.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.825+0000 7f540f400640 1 -- 172.21.15.102:0/886799187 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5410005490 con 0x7f541815c3a0 2024-03-18T16:36:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.827+0000 7f540f400640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f53f806bc50 0x7f53f806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.827+0000 7f540f400640 1 -- 172.21.15.102:0/886799187 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f541008b4b0 con 0x7f541815c3a0 2024-03-18T16:36:11.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.827+0000 7f5417400640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f53f806bc50 0x7f53f806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:11.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:11.828+0000 7f5417400640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f53f806bc50 0x7f53f806e110 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5408009c00 tx=0x7f5408009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:12.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.077+0000 7f540f400640 1 -- 172.21.15.102:0/886799187 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f54180c9030 con 0x7f541815c3a0 2024-03-18T16:36:12.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.313+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5418074fd0 con 0x7f541815c3a0 2024-03-18T16:36:12.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.326+0000 7f540f400640 1 -- 172.21.15.102:0/886799187 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f5410059660 con 0x7f541815c3a0 2024-03-18T16:36:12.328 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:12.329 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:12.329 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f53f806bc50 msgr2=0x7f53f806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f53f806bc50 0x7f53f806e110 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5408009c00 tx=0x7f5408009190 comp rx=0 tx=0).stop 2024-03-18T16:36:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f541815c3a0 msgr2=0x7f5418152d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f541815c3a0 0x7f5418152d90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f541000c6c0 tx=0x7f541000cb80 comp rx=0 tx=0).stop 2024-03-18T16:36:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 shutdown_connections 2024-03-18T16:36:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f53f806bc50 0x7f53f806e110 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:12.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f541815c3a0 0x7f5418152d90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:12.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5418141650 0x7f541813d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:12.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 --2- 172.21.15.102:0/886799187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54180c4fa0 0x7f54181f4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:12.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.329+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 >> 172.21.15.102:0/886799187 conn(0x7f54180cb890 msgr2=0x7f54180cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:12.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.330+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 shutdown_connections 2024-03-18T16:36:12.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:12.330+0000 7f541cc00640 1 -- 172.21.15.102:0/886799187 wait complete. 2024-03-18T16:36:12.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:12.954 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:13.956 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:14.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.131+0000 7fa20be00640 1 Processor -- start 2024-03-18T16:36:14.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.131+0000 7fa20be00640 1 -- start start 2024-03-18T16:36:14.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.132+0000 7fa20be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa20c0cee80 0x7fa20c0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:14.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.132+0000 7fa20be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0d0370 0x7fa20c15a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:14.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.132+0000 7fa20be00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa20c0cac20 con 0x7fa20c154630 2024-03-18T16:36:14.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.132+0000 7fa20be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa20c0cf5c0 con 0x7fa20c0cee80 2024-03-18T16:36:14.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.132+0000 7fa20be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa20c06d640 con 0x7fa20c0d0370 2024-03-18T16:36:14.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.133+0000 7fa20aa00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fa20c154630 0x7fa20c0ca510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33372/0 (socket says 172.21.15.102:33372) 2024-03-18T16:36:14.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.133+0000 7fa20aa00640 1 -- 172.21.15.102:0/921988978 learned_addr learned my addr 172.21.15.102:0/921988978 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:14.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.133+0000 7fa20b400640 1 --2- 172.21.15.102:0/921988978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0d0370 0x7fa20c15a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:14.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.133+0000 7fa20a000640 1 --2- 172.21.15.102:0/921988978 >> v2:172.21.15.102:3300/0 conn(0x7fa20c0cee80 0x7fa20c0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:14.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.136+0000 7fa20b400640 1 -- 172.21.15.102:0/921988978 >> v2:172.21.15.102:3300/0 conn(0x7fa20c0cee80 msgr2=0x7fa20c0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:14.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.136+0000 7fa20b400640 1 --2- 172.21.15.102:0/921988978 >> v2:172.21.15.102:3300/0 conn(0x7fa20c0cee80 0x7fa20c0cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:14.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.136+0000 7fa20b400640 1 -- 172.21.15.102:0/921988978 >> v1:172.21.15.154:6789/0 conn(0x7fa20c154630 legacy=0x7fa20c0ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:14.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.136+0000 7fa20b400640 1 -- 172.21.15.102:0/921988978 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20c06d920 con 0x7fa20c0d0370 2024-03-18T16:36:14.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.139+0000 7fa20b400640 1 --2- 172.21.15.102:0/921988978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0d0370 0x7fa20c15a520 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa204009d90 tx=0x7fa20402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55408accabb0644e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:14.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.142+0000 7fa209600640 1 -- 172.21.15.102:0/921988978 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa204039070 con 0x7fa20c0d0370 2024-03-18T16:36:14.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.142+0000 7fa209600640 1 -- 172.21.15.102:0/921988978 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa20402cce0 con 0x7fa20c0d0370 2024-03-18T16:36:14.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.142+0000 7fa209600640 1 -- 172.21.15.102:0/921988978 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa204034660 con 0x7fa20c0d0370 2024-03-18T16:36:14.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.143+0000 7fa20be00640 1 -- 172.21.15.102:0/921988978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0d0370 msgr2=0x7fa20c15a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:14.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.143+0000 7fa20be00640 1 --2- 172.21.15.102:0/921988978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0d0370 0x7fa20c15a520 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa204009d90 tx=0x7fa20402bf30 comp rx=0 tx=0).stop 2024-03-18T16:36:14.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.143+0000 7fa20be00640 1 -- 172.21.15.102:0/921988978 shutdown_connections 2024-03-18T16:36:14.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.143+0000 7fa20be00640 1 --2- 172.21.15.102:0/921988978 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0d0370 0x7fa20c15a520 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:14.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.143+0000 7fa20be00640 1 --2- 172.21.15.102:0/921988978 >> v2:172.21.15.102:3300/0 conn(0x7fa20c0cee80 0x7fa20c0cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:14.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.143+0000 7fa20be00640 1 -- 172.21.15.102:0/921988978 >> 172.21.15.102:0/921988978 conn(0x7fa20c067160 msgr2=0x7fa20c067570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:14.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.143+0000 7fa20be00640 1 -- 172.21.15.102:0/921988978 shutdown_connections 2024-03-18T16:36:14.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.143+0000 7fa20be00640 1 -- 172.21.15.102:0/921988978 wait complete. 2024-03-18T16:36:14.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.145+0000 7fa20be00640 1 Processor -- start 2024-03-18T16:36:14.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.145+0000 7fa20be00640 1 -- start start 2024-03-18T16:36:14.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.146+0000 7fa20be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0cee80 0x7fa20c152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:14.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.146+0000 7fa20be00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa20c0d0370 0x7fa20c152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:14.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.146+0000 7fa20be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c154630 0x7fa20c144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:14.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.146+0000 7fa20be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c0cf2d0 con 0x7fa20c154630 2024-03-18T16:36:14.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.146+0000 7fa20be00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c06dee0 con 0x7fa20c0d0370 2024-03-18T16:36:14.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.146+0000 7fa20be00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa20c154a10 con 0x7fa20c0cee80 2024-03-18T16:36:14.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.147+0000 7fa20b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c154630 0x7fa20c144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:14.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.147+0000 7fa20a000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa20c0d0370 0x7fa20c152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:14.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.147+0000 7fa20aa00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0cee80 0x7fa20c152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:14.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.147+0000 7fa20a000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa20c0d0370 0x7fa20c152480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:57758/0 (socket says 172.21.15.102:57758) 2024-03-18T16:36:14.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.147+0000 7fa20a000640 1 -- 172.21.15.102:0/2155929258 learned_addr learned my addr 172.21.15.102:0/2155929258 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.151+0000 7fa20b400640 1 -- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0cee80 msgr2=0x7fa20c152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.151+0000 7fa20b400640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0cee80 0x7fa20c152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.151+0000 7fa20b400640 1 -- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa20c0d0370 msgr2=0x7fa20c152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.151+0000 7fa20b400640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa20c0d0370 0x7fa20c152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.151+0000 7fa20b400640 1 -- 172.21.15.102:0/2155929258 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20c153500 con 0x7fa20c154630 2024-03-18T16:36:14.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.154+0000 7fa20b400640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c154630 0x7fa20c144010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0cd6a0 tx=0x7fa204004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.159+0000 7fa202a00640 1 -- 172.21.15.102:0/2155929258 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa204039050 con 0x7fa20c154630 2024-03-18T16:36:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.159+0000 7fa202a00640 1 -- 172.21.15.102:0/2155929258 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa204045070 con 0x7fa20c154630 2024-03-18T16:36:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.159+0000 7fa202a00640 1 -- 172.21.15.102:0/2155929258 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa204009450 con 0x7fa20c154630 2024-03-18T16:36:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.159+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20c1583a0 con 0x7fa20c154630 2024-03-18T16:36:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.160+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa20c0ca160 con 0x7fa20c154630 2024-03-18T16:36:14.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.161+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa20c074fd0 con 0x7fa20c154630 2024-03-18T16:36:14.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.220+0000 7fa202a00640 1 -- 172.21.15.102:0/2155929258 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa20403e850 con 0x7fa20c154630 2024-03-18T16:36:14.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.222+0000 7fa202a00640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1ec06bc50 0x7fa1ec06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:14.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.222+0000 7fa20aa00640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1ec06bc50 0x7fa1ec06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:14.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.223+0000 7fa20aa00640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1ec06bc50 0x7fa1ec06e110 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc007fd0 tx=0x7fa1fc007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:14.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.277+0000 7fa202a00640 1 -- 172.21.15.102:0/2155929258 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa20407a670 con 0x7fa20c154630 2024-03-18T16:36:14.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.496+0000 7fa202a00640 1 -- 172.21.15.102:0/2155929258 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa20407c5d0 con 0x7fa20c154630 2024-03-18T16:36:14.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.735+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa20c153e20 con 0x7fa20c154630 2024-03-18T16:36:14.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.756+0000 7fa202a00640 1 -- 172.21.15.102:0/2155929258 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fa204080670 con 0x7fa20c154630 2024-03-18T16:36:14.759 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:14.759 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:14.759 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1ec06bc50 msgr2=0x7fa1ec06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1ec06bc50 0x7fa1ec06e110 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc007fd0 tx=0x7fa1fc007e80 comp rx=0 tx=0).stop 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c154630 msgr2=0x7fa20c144010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c154630 0x7fa20c144010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0cd6a0 tx=0x7fa204004a40 comp rx=0 tx=0).stop 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 shutdown_connections 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa1ec06bc50 0x7fa1ec06e110 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa20c154630 0x7fa20c144010 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa20c0d0370 0x7fa20c152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 --2- 172.21.15.102:0/2155929258 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa20c0cee80 0x7fa20c152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 >> 172.21.15.102:0/2155929258 conn(0x7fa20c067160 msgr2=0x7fa20c14f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 shutdown_connections 2024-03-18T16:36:14.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:14.759+0000 7fa20be00640 1 -- 172.21.15.102:0/2155929258 wait complete. 2024-03-18T16:36:15.381 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:15.381 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:16.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:16.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.555+0000 7fd7eac00640 1 Processor -- start 2024-03-18T16:36:16.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.556+0000 7fd7eac00640 1 -- start start 2024-03-18T16:36:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.557+0000 7fd7eac00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd7e40cfe70 0x7fd7e415ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.557+0000 7fd7eac00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7e40cabd0 con 0x7fd7e40d0250 2024-03-18T16:36:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.557+0000 7fd7eac00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7e40caea0 con 0x7fd7e40cc2f0 2024-03-18T16:36:16.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.557+0000 7fd7eac00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd7e40cb180 con 0x7fd7e40cfe70 2024-03-18T16:36:16.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.558+0000 7fd7e9800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd7e40cfe70 0x7fd7e415ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:16.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.558+0000 7fd7e8e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd7e40d0250 0x7fd7e40c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40646/0 (socket says 172.21.15.102:40646) 2024-03-18T16:36:16.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.558+0000 7fd7e8e00640 1 -- 172.21.15.102:0/2226164153 learned_addr learned my addr 172.21.15.102:0/2226164153 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:16.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.561+0000 7fd7e3e00640 1 -- 172.21.15.102:0/2226164153 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2496703284 0 0) 0x7fd7e40cabd0 con 0x7fd7e40d0250 2024-03-18T16:36:16.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.561+0000 7fd7e3e00640 1 -- 172.21.15.102:0/2226164153 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7cc003520 con 0x7fd7e40d0250 2024-03-18T16:36:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.564+0000 7fd7e9800640 1 -- 172.21.15.102:0/2226164153 >> v1:172.21.15.154:6789/0 conn(0x7fd7e40cc2f0 legacy=0x7fd7e40cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.564+0000 7fd7e9800640 1 -- 172.21.15.102:0/2226164153 >> v1:172.21.15.102:6789/0 conn(0x7fd7e40d0250 legacy=0x7fd7e40c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:16.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.564+0000 7fd7e9800640 1 -- 172.21.15.102:0/2226164153 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e40cb460 con 0x7fd7e40cfe70 2024-03-18T16:36:16.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.568+0000 7fd7e9800640 1 --2- 172.21.15.102:0/2226164153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e40cfe70 0x7fd7e415ae40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80097a0 tx=0x7fd7d802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e0baa5ee78fa5ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:16.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.572+0000 7fd7e3e00640 1 -- 172.21.15.102:0/2226164153 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d802f810 con 0x7fd7e40cfe70 2024-03-18T16:36:16.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.573+0000 7fd7e3e00640 1 -- 172.21.15.102:0/2226164153 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d802f9b0 con 0x7fd7e40cfe70 2024-03-18T16:36:16.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.573+0000 7fd7e3e00640 1 -- 172.21.15.102:0/2226164153 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d802fd60 con 0x7fd7e40cfe70 2024-03-18T16:36:16.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.574+0000 7fd7eac00640 1 -- 172.21.15.102:0/2226164153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e40cfe70 msgr2=0x7fd7e415ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:16.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.574+0000 7fd7eac00640 1 --2- 172.21.15.102:0/2226164153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e40cfe70 0x7fd7e415ae40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80097a0 tx=0x7fd7d802b9f0 comp rx=0 tx=0).stop 2024-03-18T16:36:16.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.574+0000 7fd7eac00640 1 -- 172.21.15.102:0/2226164153 shutdown_connections 2024-03-18T16:36:16.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.574+0000 7fd7eac00640 1 --2- 172.21.15.102:0/2226164153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e40cfe70 0x7fd7e415ae40 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:16.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.574+0000 7fd7eac00640 1 -- 172.21.15.102:0/2226164153 >> 172.21.15.102:0/2226164153 conn(0x7fd7e408c3e0 msgr2=0x7fd7e406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:16.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.574+0000 7fd7eac00640 1 -- 172.21.15.102:0/2226164153 shutdown_connections 2024-03-18T16:36:16.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.574+0000 7fd7eac00640 1 -- 172.21.15.102:0/2226164153 wait complete. 2024-03-18T16:36:16.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.577+0000 7fd7eac00640 1 Processor -- start 2024-03-18T16:36:16.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.577+0000 7fd7eac00640 1 -- start start 2024-03-18T16:36:16.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.578+0000 7fd7eac00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e40cc2f0 0x7fd7e41e1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:16.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.578+0000 7fd7eac00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7e40cfe70 0x7fd7e41e0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:16.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.578+0000 7fd7eac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40d0250 0x7fd7e4159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:16.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.578+0000 7fd7eac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e40cbd00 con 0x7fd7e40d0250 2024-03-18T16:36:16.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.578+0000 7fd7eac00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e40cba20 con 0x7fd7e40cc2f0 2024-03-18T16:36:16.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.578+0000 7fd7eac00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e40c8330 con 0x7fd7e40cfe70 2024-03-18T16:36:16.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.579+0000 7fd7e8e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7e40cfe70 0x7fd7e41e0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:16.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.579+0000 7fd7e9800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e40cc2f0 0x7fd7e41e1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:16.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.579+0000 7fd7ea200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40d0250 0x7fd7e4159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:16.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.579+0000 7fd7e8e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7e40cfe70 0x7fd7e41e0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50650/0 (socket says 172.21.15.102:50650) 2024-03-18T16:36:16.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.579+0000 7fd7e8e00640 1 -- 172.21.15.102:0/3907952692 learned_addr learned my addr 172.21.15.102:0/3907952692 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:16.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.583+0000 7fd7e8e00640 1 -- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e40cc2f0 msgr2=0x7fd7e41e1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:16.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.583+0000 7fd7e8e00640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e40cc2f0 0x7fd7e41e1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:16.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.583+0000 7fd7e8e00640 1 -- 172.21.15.102:0/3907952692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40d0250 msgr2=0x7fd7e4159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:16.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.583+0000 7fd7e8e00640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40d0250 0x7fd7e4159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:16.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.583+0000 7fd7e8e00640 1 -- 172.21.15.102:0/3907952692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e41efcf0 con 0x7fd7e40cfe70 2024-03-18T16:36:16.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.585+0000 7fd7e8e00640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7e40cfe70 0x7fd7e41e0620 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd7d000c3e0 tx=0x7fd7d000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:16.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.589+0000 7fd7e1600640 1 -- 172.21.15.102:0/3907952692 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0017070 con 0x7fd7e40cfe70 2024-03-18T16:36:16.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.589+0000 7fd7e1600640 1 -- 172.21.15.102:0/3907952692 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d0005ce0 con 0x7fd7e40cfe70 2024-03-18T16:36:16.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.589+0000 7fd7e1600640 1 -- 172.21.15.102:0/3907952692 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d00072d0 con 0x7fd7e40cfe70 2024-03-18T16:36:16.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.589+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e41effb0 con 0x7fd7e40cfe70 2024-03-18T16:36:16.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.589+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e41f0850 con 0x7fd7e40cfe70 2024-03-18T16:36:16.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.591+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e4074fd0 con 0x7fd7e40cfe70 2024-03-18T16:36:16.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.615+0000 7fd7e1600640 1 -- 172.21.15.102:0/3907952692 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd7d001a070 con 0x7fd7e40cfe70 2024-03-18T16:36:16.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.617+0000 7fd7e1600640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd7c406bc00 0x7fd7c406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:16.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.617+0000 7fd7e9800640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd7c406bc00 0x7fd7c406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:16.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.617+0000 7fd7e1600640 1 -- 172.21.15.102:0/3907952692 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd7d0056770 con 0x7fd7e40cfe70 2024-03-18T16:36:16.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.618+0000 7fd7e9800640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd7c406bc00 0x7fd7c406e0c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8036000 tx=0x7fd7d8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:16.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:16.889+0000 7fd7e1600640 1 -- 172.21.15.102:0/3907952692 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd7d00586d0 con 0x7fd7e40cfe70 2024-03-18T16:36:17.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.134+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7e40c4e30 con 0x7fd7e40cfe70 2024-03-18T16:36:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.147+0000 7fd7e1600640 1 -- 172.21.15.102:0/3907952692 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fd7d005c770 con 0x7fd7e40cfe70 2024-03-18T16:36:17.150 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:17.150 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:17.150 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd7c406bc00 msgr2=0x7fd7c406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd7c406bc00 0x7fd7c406e0c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8036000 tx=0x7fd7d8008030 comp rx=0 tx=0).stop 2024-03-18T16:36:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7e40cfe70 msgr2=0x7fd7e41e0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7e40cfe70 0x7fd7e41e0620 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd7d000c3e0 tx=0x7fd7d000c8a0 comp rx=0 tx=0).stop 2024-03-18T16:36:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 shutdown_connections 2024-03-18T16:36:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd7c406bc00 0x7fd7c406e0c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40d0250 0x7fd7e4159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:17.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd7e40cfe70 0x7fd7e41e0620 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 --2- 172.21.15.102:0/3907952692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7e40cc2f0 0x7fd7e41e1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 >> 172.21.15.102:0/3907952692 conn(0x7fd7e408c3e0 msgr2=0x7fd7e40c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 shutdown_connections 2024-03-18T16:36:17.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:17.150+0000 7fd7eac00640 1 -- 172.21.15.102:0/3907952692 wait complete. 2024-03-18T16:36:17.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:17.777 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:18.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:18.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.954+0000 7ff706000640 1 Processor -- start 2024-03-18T16:36:18.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.954+0000 7ff706000640 1 -- start start 2024-03-18T16:36:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.955+0000 7ff706000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cfe70 0x7ff70015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.956+0000 7ff706000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff7000cc2f0 0x7ff7000cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.956+0000 7ff706000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7000ccca0 con 0x7ff7000d0250 2024-03-18T16:36:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.956+0000 7ff706000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7ff7000cabd0 con 0x7ff7000cc2f0 2024-03-18T16:36:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.956+0000 7ff706000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff7000caea0 con 0x7ff7000cfe70 2024-03-18T16:36:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.956+0000 7ff704c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cfe70 0x7ff70015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.956+0000 7ff6ffe00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7ff7000d0250 0x7ff7000c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:33392/0 (socket says 172.21.15.102:33392) 2024-03-18T16:36:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.957+0000 7ff6ffe00640 1 -- 172.21.15.102:0/1876260196 learned_addr learned my addr 172.21.15.102:0/1876260196 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.957+0000 7ff705600640 1 --2- 172.21.15.102:0/1876260196 >> v2:172.21.15.102:3300/0 conn(0x7ff7000cc2f0 0x7ff7000cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.959+0000 7ff704c00640 1 -- 172.21.15.102:0/1876260196 >> v2:172.21.15.102:3300/0 conn(0x7ff7000cc2f0 msgr2=0x7ff7000cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.959+0000 7ff704c00640 1 --2- 172.21.15.102:0/1876260196 >> v2:172.21.15.102:3300/0 conn(0x7ff7000cc2f0 0x7ff7000cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.959+0000 7ff704c00640 1 -- 172.21.15.102:0/1876260196 >> v1:172.21.15.154:6789/0 conn(0x7ff7000d0250 legacy=0x7ff7000c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.959+0000 7ff704c00640 1 -- 172.21.15.102:0/1876260196 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7000cb180 con 0x7ff7000cfe70 2024-03-18T16:36:18.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.962+0000 7ff704c00640 1 --2- 172.21.15.102:0/1876260196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cfe70 0x7ff70015ae40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40097a0 tx=0x7ff6f402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fd14c2c6c7b7eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:18.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.965+0000 7ff6ff400640 1 -- 172.21.15.102:0/1876260196 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4034070 con 0x7ff7000cfe70 2024-03-18T16:36:18.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.965+0000 7ff6ff400640 1 -- 172.21.15.102:0/1876260196 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6f4030800 con 0x7ff7000cfe70 2024-03-18T16:36:18.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.965+0000 7ff6ff400640 1 -- 172.21.15.102:0/1876260196 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4030bd0 con 0x7ff7000cfe70 2024-03-18T16:36:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.966+0000 7ff706000640 1 -- 172.21.15.102:0/1876260196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cfe70 msgr2=0x7ff70015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.966+0000 7ff706000640 1 --2- 172.21.15.102:0/1876260196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cfe70 0x7ff70015ae40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff6f40097a0 tx=0x7ff6f402b940 comp rx=0 tx=0).stop 2024-03-18T16:36:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.967+0000 7ff706000640 1 -- 172.21.15.102:0/1876260196 shutdown_connections 2024-03-18T16:36:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.967+0000 7ff706000640 1 --2- 172.21.15.102:0/1876260196 >> v2:172.21.15.102:3300/0 conn(0x7ff7000cc2f0 0x7ff7000cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.967+0000 7ff706000640 1 --2- 172.21.15.102:0/1876260196 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cfe70 0x7ff70015ae40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.967+0000 7ff706000640 1 -- 172.21.15.102:0/1876260196 >> 172.21.15.102:0/1876260196 conn(0x7ff70008c3e0 msgr2=0x7ff70006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:18.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.967+0000 7ff706000640 1 -- 172.21.15.102:0/1876260196 shutdown_connections 2024-03-18T16:36:18.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.967+0000 7ff706000640 1 -- 172.21.15.102:0/1876260196 wait complete. 2024-03-18T16:36:18.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.969+0000 7ff706000640 1 Processor -- start 2024-03-18T16:36:18.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.970+0000 7ff706000640 1 -- start start 2024-03-18T16:36:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.971+0000 7ff706000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cc2f0 0x7ff700156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.971+0000 7ff706000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7000cfe70 0x7ff700156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.971+0000 7ff706000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7000d0250 0x7ff7001f3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.971+0000 7ff706000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7ff7000cba20 con 0x7ff7000cfe70 2024-03-18T16:36:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.971+0000 7ff706000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7000cb740 con 0x7ff7000d0250 2024-03-18T16:36:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.971+0000 7ff706000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7ff7000cbd00 con 0x7ff7000cc2f0 2024-03-18T16:36:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.972+0000 7ff704c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cc2f0 0x7ff700156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.972+0000 7ff704c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cc2f0 0x7ff700156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:50664/0 (socket says 172.21.15.102:50664) 2024-03-18T16:36:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.972+0000 7ff704c00640 1 -- 172.21.15.102:0/734056475 learned_addr learned my addr 172.21.15.102:0/734056475 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.972+0000 7ff705600640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7000d0250 0x7ff7001f3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.972+0000 7ff6ffe00640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7000cfe70 0x7ff700156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:18.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.975+0000 7ff704c00640 1 -- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7000d0250 msgr2=0x7ff7001f3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:18.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.975+0000 7ff704c00640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7000d0250 0x7ff7001f3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:18.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.975+0000 7ff704c00640 1 -- 172.21.15.102:0/734056475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7000cfe70 msgr2=0x7ff700156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:18.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.975+0000 7ff704c00640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7000cfe70 0x7ff700156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:18.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.975+0000 7ff704c00640 1 -- 172.21.15.102:0/734056475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7001e1960 con 0x7ff7000cc2f0 2024-03-18T16:36:18.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.978+0000 7ff704c00640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cc2f0 0x7ff700156a90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff6f402bf50 tx=0x7ff6f40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.981+0000 7ff6fcc00640 1 -- 172.21.15.102:0/734056475 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4034040 con 0x7ff7000cc2f0 2024-03-18T16:36:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.981+0000 7ff6fcc00640 1 -- 172.21.15.102:0/734056475 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6f4030600 con 0x7ff7000cc2f0 2024-03-18T16:36:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.981+0000 7ff6fcc00640 1 -- 172.21.15.102:0/734056475 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f40423f0 con 0x7ff7000cc2f0 2024-03-18T16:36:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.981+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7001e1c40 con 0x7ff7000cc2f0 2024-03-18T16:36:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.981+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff700156290 con 0x7ff7000cc2f0 2024-03-18T16:36:18.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:18.983+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7000735d0 con 0x7ff7000cc2f0 2024-03-18T16:36:19.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.241+0000 7ff6fcc00640 1 -- 172.21.15.102:0/734056475 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff6f4037070 con 0x7ff7000cc2f0 2024-03-18T16:36:19.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.243+0000 7ff6fcc00640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6e006bc20 0x7ff6e006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:19.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.243+0000 7ff6ffe00640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6e006bc20 0x7ff6e006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:19.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.244+0000 7ff6fcc00640 1 -- 172.21.15.102:0/734056475 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7ff6f40307b0 con 0x7ff7000cc2f0 2024-03-18T16:36:19.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.245+0000 7ff6ffe00640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6e006bc20 0x7ff6e006e0e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff70015a730 tx=0x7ff6ec007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:19.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.262+0000 7ff6fcc00640 1 -- 172.21.15.102:0/734056475 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff7000735d0 con 0x7ff7000cc2f0 2024-03-18T16:36:19.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.537+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff700074fd0 con 0x7ff7000cc2f0 2024-03-18T16:36:19.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.550+0000 7ff6fcc00640 1 -- 172.21.15.102:0/734056475 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7ff6f407b940 con 0x7ff7000cc2f0 2024-03-18T16:36:19.553 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:19.553 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:19.553 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6e006bc20 msgr2=0x7ff6e006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6e006bc20 0x7ff6e006e0e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff70015a730 tx=0x7ff6ec007e80 comp rx=0 tx=0).stop 2024-03-18T16:36:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cc2f0 msgr2=0x7ff700156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cc2f0 0x7ff700156a90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff6f402bf50 tx=0x7ff6f40046c0 comp rx=0 tx=0).stop 2024-03-18T16:36:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 shutdown_connections 2024-03-18T16:36:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7ff6e006bc20 0x7ff6e006e0e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:19.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff7000d0250 0x7ff7001f3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7000cfe70 0x7ff700156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 --2- 172.21.15.102:0/734056475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7ff7000cc2f0 0x7ff700156a90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 >> 172.21.15.102:0/734056475 conn(0x7ff70008c3e0 msgr2=0x7ff7000c63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 shutdown_connections 2024-03-18T16:36:19.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:19.552+0000 7ff706000640 1 -- 172.21.15.102:0/734056475 wait complete. 2024-03-18T16:36:20.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:20.177 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:21.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:21.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.350+0000 7fb5fce00640 1 Processor -- start 2024-03-18T16:36:21.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.350+0000 7fb5fce00640 1 -- start start 2024-03-18T16:36:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.351+0000 7fb5fce00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb5f80cfe70 0x7fb5f815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.351+0000 7fb5fce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80d0250 0x7fb5f80c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.351+0000 7fb5fce00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5f80cabd0 con 0x7fb5f80cc2f0 2024-03-18T16:36:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.351+0000 7fb5fce00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fb5f80caea0 con 0x7fb5f80cfe70 2024-03-18T16:36:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.351+0000 7fb5fce00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f80cb180 con 0x7fb5f80d0250 2024-03-18T16:36:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.352+0000 7fb5f6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80d0250 0x7fb5f80c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.352+0000 7fb5f7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb5f80cfe70 0x7fb5f815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.353+0000 7fb5f6a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80d0250 0x7fb5f80c65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:55234/0 (socket says 172.21.15.102:55234) 2024-03-18T16:36:21.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.353+0000 7fb5f6a00640 1 -- 172.21.15.102:0/3726939047 learned_addr learned my addr 172.21.15.102:0/3726939047 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:21.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.356+0000 7fb5f6a00640 1 -- 172.21.15.102:0/3726939047 >> v2:172.21.15.102:3300/0 conn(0x7fb5f80cfe70 msgr2=0x7fb5f815ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:21.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.356+0000 7fb5f6a00640 1 --2- 172.21.15.102:0/3726939047 >> v2:172.21.15.102:3300/0 conn(0x7fb5f80cfe70 0x7fb5f815ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:21.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.356+0000 7fb5f6a00640 1 -- 172.21.15.102:0/3726939047 >> v1:172.21.15.102:6789/0 conn(0x7fb5f80cc2f0 legacy=0x7fb5f80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:21.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.356+0000 7fb5f6a00640 1 -- 172.21.15.102:0/3726939047 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f80cb460 con 0x7fb5f80d0250 2024-03-18T16:36:21.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.358+0000 7fb5f6a00640 1 --2- 172.21.15.102:0/3726939047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80d0250 0x7fb5f80c65e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009870 tx=0x7fb5e402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=767799d6724cfd44 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:21.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.363+0000 7fb5f6000640 1 -- 172.21.15.102:0/3726939047 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e402f830 con 0x7fb5f80d0250 2024-03-18T16:36:21.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.363+0000 7fb5f6000640 1 -- 172.21.15.102:0/3726939047 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5e402f9d0 con 0x7fb5f80d0250 2024-03-18T16:36:21.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.363+0000 7fb5f6000640 1 -- 172.21.15.102:0/3726939047 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e402fda0 con 0x7fb5f80d0250 2024-03-18T16:36:21.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.364+0000 7fb5fce00640 1 -- 172.21.15.102:0/3726939047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80d0250 msgr2=0x7fb5f80c65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:21.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.364+0000 7fb5fce00640 1 --2- 172.21.15.102:0/3726939047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80d0250 0x7fb5f80c65e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009870 tx=0x7fb5e402ba10 comp rx=0 tx=0).stop 2024-03-18T16:36:21.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.364+0000 7fb5fce00640 1 -- 172.21.15.102:0/3726939047 shutdown_connections 2024-03-18T16:36:21.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.364+0000 7fb5fce00640 1 --2- 172.21.15.102:0/3726939047 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80d0250 0x7fb5f80c65e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:21.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.364+0000 7fb5fce00640 1 --2- 172.21.15.102:0/3726939047 >> v2:172.21.15.102:3300/0 conn(0x7fb5f80cfe70 0x7fb5f815ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:21.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.364+0000 7fb5fce00640 1 -- 172.21.15.102:0/3726939047 >> 172.21.15.102:0/3726939047 conn(0x7fb5f808c3e0 msgr2=0x7fb5f806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:21.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.364+0000 7fb5fce00640 1 -- 172.21.15.102:0/3726939047 shutdown_connections 2024-03-18T16:36:21.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.364+0000 7fb5fce00640 1 -- 172.21.15.102:0/3726939047 wait complete. 2024-03-18T16:36:21.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.366+0000 7fb5fce00640 1 Processor -- start 2024-03-18T16:36:21.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.366+0000 7fb5fce00640 1 -- start start 2024-03-18T16:36:21.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.367+0000 7fb5fce00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80cc2f0 0x7fb5f81503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:21.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.367+0000 7fb5fce00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5f80cfe70 0x7fb5f814fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:21.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.367+0000 7fb5fce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f80d0250 0x7fb5f81f0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:21.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.367+0000 7fb5fce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f80cbd00 con 0x7fb5f80d0250 2024-03-18T16:36:21.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.367+0000 7fb5fce00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f80cba20 con 0x7fb5f80cfe70 2024-03-18T16:36:21.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.367+0000 7fb5fce00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f80c6b20 con 0x7fb5f80cc2f0 2024-03-18T16:36:21.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.367+0000 7fb5f7400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80cc2f0 0x7fb5f81503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:21.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.367+0000 7fb5f6a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5f80cfe70 0x7fb5f814fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:21.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.368+0000 7fb5f6a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5f80cfe70 0x7fb5f814fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34756/0 (socket says 172.21.15.102:34756) 2024-03-18T16:36:21.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.368+0000 7fb5f6a00640 1 -- 172.21.15.102:0/2979126324 learned_addr learned my addr 172.21.15.102:0/2979126324 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:21.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.368+0000 7fb5f7e00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f80d0250 0x7fb5f81f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:21.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.371+0000 7fb5f6a00640 1 -- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80cc2f0 msgr2=0x7fb5f81503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:21.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.371+0000 7fb5f6a00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80cc2f0 0x7fb5f81503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:21.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.371+0000 7fb5f6a00640 1 -- 172.21.15.102:0/2979126324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f80d0250 msgr2=0x7fb5f81f0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:21.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.371+0000 7fb5f6a00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f80d0250 0x7fb5f81f0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:21.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.371+0000 7fb5f6a00640 1 -- 172.21.15.102:0/2979126324 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f81f3810 con 0x7fb5f80cfe70 2024-03-18T16:36:21.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.374+0000 7fb5f6a00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5f80cfe70 0x7fb5f814fda0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009990 tx=0x7fb5e4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.377+0000 7fb5ef400640 1 -- 172.21.15.102:0/2979126324 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4046070 con 0x7fb5f80cfe70 2024-03-18T16:36:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.377+0000 7fb5ef400640 1 -- 172.21.15.102:0/2979126324 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5e4037c60 con 0x7fb5f80cfe70 2024-03-18T16:36:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.377+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f81f3af0 con 0x7fb5f80cfe70 2024-03-18T16:36:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.377+0000 7fb5ef400640 1 -- 172.21.15.102:0/2979126324 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4041680 con 0x7fb5f80cfe70 2024-03-18T16:36:21.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.378+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f81f40b0 con 0x7fb5f80cfe70 2024-03-18T16:36:21.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.379+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f8067d40 con 0x7fb5f80cfe70 2024-03-18T16:36:21.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.399+0000 7fb5ef400640 1 -- 172.21.15.102:0/2979126324 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb5e4049070 con 0x7fb5f80cfe70 2024-03-18T16:36:21.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.400+0000 7fb5ef400640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb5d806bc20 0x7fb5d806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:21.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.400+0000 7fb5f7400640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb5d806bc20 0x7fb5d806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:21.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.401+0000 7fb5f7400640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb5d806bc20 0x7fb5d806e0e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009b40 tx=0x7fb5e8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:21.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.402+0000 7fb5ef400640 1 -- 172.21.15.102:0/2979126324 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fb5e4078ee0 con 0x7fb5f80cfe70 2024-03-18T16:36:21.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.666+0000 7fb5ef400640 1 -- 172.21.15.102:0/2979126324 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb5e4033030 con 0x7fb5f80cfe70 2024-03-18T16:36:21.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.947+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5f806cac0 con 0x7fb5f80cfe70 2024-03-18T16:36:21.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.963+0000 7fb5ef400640 1 -- 172.21.15.102:0/2979126324 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fb5e407ae40 con 0x7fb5f80cfe70 2024-03-18T16:36:21.964 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:21.964 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:21.965 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:21.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb5d806bc20 msgr2=0x7fb5d806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:21.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb5d806bc20 0x7fb5d806e0e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009b40 tx=0x7fb5e8009190 comp rx=0 tx=0).stop 2024-03-18T16:36:21.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5f80cfe70 msgr2=0x7fb5f814fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:21.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5f80cfe70 0x7fb5f814fda0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009990 tx=0x7fb5e4004730 comp rx=0 tx=0).stop 2024-03-18T16:36:21.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 shutdown_connections 2024-03-18T16:36:21.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fb5d806bc20 0x7fb5d806e0e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:21.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5f80d0250 0x7fb5f81f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:21.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5f80cfe70 0x7fb5f814fda0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:21.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 --2- 172.21.15.102:0/2979126324 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fb5f80cc2f0 0x7fb5f81503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:21.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 >> 172.21.15.102:0/2979126324 conn(0x7fb5f808c3e0 msgr2=0x7fb5f80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:21.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.966+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 shutdown_connections 2024-03-18T16:36:21.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:21.967+0000 7fb5fce00640 1 -- 172.21.15.102:0/2979126324 wait complete. 2024-03-18T16:36:22.588 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:22.588 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:23.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:23.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.763+0000 7f7abfe00640 1 Processor -- start 2024-03-18T16:36:23.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.764+0000 7f7abfe00640 1 -- start start 2024-03-18T16:36:23.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.765+0000 7f7abfe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7ac00d0250 0x7f7ac015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:23.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.765+0000 7f7abfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ac00cc2f0 0x7f7ac00cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:23.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.765+0000 7f7abfe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ac00ccc80 con 0x7f7ac00cfe70 2024-03-18T16:36:23.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.765+0000 7f7abfe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7ac00cabd0 con 0x7f7ac00cc2f0 2024-03-18T16:36:23.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.765+0000 7f7abfe00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7ac00caea0 con 0x7f7ac00d0250 2024-03-18T16:36:23.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.806+0000 7f7abf400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ac00cc2f0 0x7f7ac00cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:23.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.807+0000 7f7abf400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ac00cc2f0 0x7f7ac00cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37488/0 (socket says 172.21.15.102:37488) 2024-03-18T16:36:23.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.807+0000 7f7abf400640 1 -- 172.21.15.102:0/2223862124 learned_addr learned my addr 172.21.15.102:0/2223862124 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:23.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.809+0000 7f7abe000640 1 --2- 172.21.15.102:0/2223862124 >> v2:172.21.15.154:3300/0 conn(0x7f7ac00d0250 0x7f7ac015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:23.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.810+0000 7f7abf400640 1 -- 172.21.15.102:0/2223862124 >> v2:172.21.15.154:3300/0 conn(0x7f7ac00d0250 msgr2=0x7f7ac015ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:23.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.810+0000 7f7abf400640 1 --2- 172.21.15.102:0/2223862124 >> v2:172.21.15.154:3300/0 conn(0x7f7ac00d0250 0x7f7ac015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:23.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.810+0000 7f7abf400640 1 -- 172.21.15.102:0/2223862124 >> v1:172.21.15.102:6789/0 conn(0x7f7ac00cfe70 legacy=0x7f7ac00c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:23.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.810+0000 7f7abf400640 1 -- 172.21.15.102:0/2223862124 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac00cb180 con 0x7f7ac00cc2f0 2024-03-18T16:36:23.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.814+0000 7f7abf400640 1 --2- 172.21.15.102:0/2223862124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ac00cc2f0 0x7f7ac00cc740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00c6370 tx=0x7f7ab802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd414beb9ec1e83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:23.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.820+0000 7f7abd600640 1 -- 172.21.15.102:0/2223862124 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab802cb20 con 0x7f7ac00cc2f0 2024-03-18T16:36:23.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.820+0000 7f7abd600640 1 -- 172.21.15.102:0/2223862124 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ab802ccc0 con 0x7f7ac00cc2f0 2024-03-18T16:36:23.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.820+0000 7f7abd600640 1 -- 172.21.15.102:0/2223862124 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8034610 con 0x7f7ac00cc2f0 2024-03-18T16:36:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.821+0000 7f7abfe00640 1 -- 172.21.15.102:0/2223862124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ac00cc2f0 msgr2=0x7f7ac00cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.821+0000 7f7abfe00640 1 --2- 172.21.15.102:0/2223862124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ac00cc2f0 0x7f7ac00cc740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00c6370 tx=0x7f7ab802bf80 comp rx=0 tx=0).stop 2024-03-18T16:36:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.822+0000 7f7abfe00640 1 -- 172.21.15.102:0/2223862124 shutdown_connections 2024-03-18T16:36:23.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.822+0000 7f7abfe00640 1 --2- 172.21.15.102:0/2223862124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ac00cc2f0 0x7f7ac00cc740 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:23.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.822+0000 7f7abfe00640 1 --2- 172.21.15.102:0/2223862124 >> v2:172.21.15.154:3300/0 conn(0x7f7ac00d0250 0x7f7ac015ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:23.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.822+0000 7f7abfe00640 1 -- 172.21.15.102:0/2223862124 >> 172.21.15.102:0/2223862124 conn(0x7f7ac008c3e0 msgr2=0x7f7ac006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:23.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.822+0000 7f7abfe00640 1 -- 172.21.15.102:0/2223862124 shutdown_connections 2024-03-18T16:36:23.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.822+0000 7f7abfe00640 1 -- 172.21.15.102:0/2223862124 wait complete. 2024-03-18T16:36:23.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.824+0000 7f7abfe00640 1 Processor -- start 2024-03-18T16:36:23.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.824+0000 7f7abfe00640 1 -- start start 2024-03-18T16:36:23.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.825+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ac00cc2f0 0x7f7ac01e14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:23.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.825+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ac00cfe70 0x7f7ac0157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:23.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.825+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ac00d0250 0x7f7ac0151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:23.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.825+0000 7f7abfe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac00cba20 con 0x7f7ac00d0250 2024-03-18T16:36:23.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.825+0000 7f7abfe00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac00cb740 con 0x7f7ac00cfe70 2024-03-18T16:36:23.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.825+0000 7f7abfe00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7ac00cbd00 con 0x7f7ac00cc2f0 2024-03-18T16:36:23.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.825+0000 7f7abea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ac00cc2f0 0x7f7ac01e14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:23.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.825+0000 7f7abe000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ac00cfe70 0x7f7ac0157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:23.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.826+0000 7f7abe000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ac00cfe70 0x7f7ac0157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:34778/0 (socket says 172.21.15.102:34778) 2024-03-18T16:36:23.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.826+0000 7f7abe000640 1 -- 172.21.15.102:0/881955208 learned_addr learned my addr 172.21.15.102:0/881955208 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:23.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.831+0000 7f7abe000640 1 -- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ac00cc2f0 msgr2=0x7f7ac01e14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:23.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.831+0000 7f7abe000640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ac00cc2f0 0x7f7ac01e14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:23.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.831+0000 7f7abe000640 1 -- 172.21.15.102:0/881955208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ac00d0250 msgr2=0x7f7ac0151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:23.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.831+0000 7f7abe000640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ac00d0250 0x7f7ac0151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:23.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.831+0000 7f7abe000640 1 -- 172.21.15.102:0/881955208 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac01e0450 con 0x7f7ac00cfe70 2024-03-18T16:36:23.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.834+0000 7f7abe000640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ac00cfe70 0x7f7ac0157d60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00c7c0 tx=0x7f7aac00cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.838+0000 7f7ab6a00640 1 -- 172.21.15.102:0/881955208 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac015070 con 0x7f7ac00cfe70 2024-03-18T16:36:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.838+0000 7f7ab6a00640 1 -- 172.21.15.102:0/881955208 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aac007820 con 0x7f7ac00cfe70 2024-03-18T16:36:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.838+0000 7f7ab6a00640 1 -- 172.21.15.102:0/881955208 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac007bd0 con 0x7f7ac00cfe70 2024-03-18T16:36:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.839+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ac01e0730 con 0x7f7ac00cfe70 2024-03-18T16:36:23.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.839+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ac0157840 con 0x7f7ac00cfe70 2024-03-18T16:36:23.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:23.840+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ac0157a40 con 0x7f7ac00cfe70 2024-03-18T16:36:24.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.106+0000 7f7ab6a00640 1 -- 172.21.15.102:0/881955208 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7aac018070 con 0x7f7ac00cfe70 2024-03-18T16:36:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.110+0000 7f7ab6a00640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7aa006bc50 0x7f7aa006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:24.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.110+0000 7f7abea00640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7aa006bc50 0x7f7aa006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.110+0000 7f7ab6a00640 1 -- 172.21.15.102:0/881955208 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7aac08b030 con 0x7f7ac00cfe70 2024-03-18T16:36:24.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.111+0000 7f7abea00640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7aa006bc50 0x7f7aa006e110 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0004f60 tx=0x7f7ab0007e30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:24.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.120+0000 7f7ab6a00640 1 -- 172.21.15.102:0/881955208 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7ac0157a40 con 0x7f7ac00cfe70 2024-03-18T16:36:24.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.361+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ac0074fd0 con 0x7f7ac00cfe70 2024-03-18T16:36:24.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.376+0000 7f7ab6a00640 1 -- 172.21.15.102:0/881955208 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f7aac0590c0 con 0x7f7ac00cfe70 2024-03-18T16:36:24.378 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:24.378 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:24.379 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:24.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7aa006bc50 msgr2=0x7f7aa006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7aa006bc50 0x7f7aa006e110 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0004f60 tx=0x7f7ab0007e30 comp rx=0 tx=0).stop 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ac00cfe70 msgr2=0x7f7ac0157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ac00cfe70 0x7f7ac0157d60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00c7c0 tx=0x7f7aac00cc80 comp rx=0 tx=0).stop 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 shutdown_connections 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7aa006bc50 0x7f7aa006e110 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ac00d0250 0x7f7ac0151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ac00cfe70 0x7f7ac0157d60 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 --2- 172.21.15.102:0/881955208 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7ac00cc2f0 0x7f7ac01e14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 >> 172.21.15.102:0/881955208 conn(0x7f7ac008c3e0 msgr2=0x7f7ac00cecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 shutdown_connections 2024-03-18T16:36:24.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:24.380+0000 7f7abfe00640 1 -- 172.21.15.102:0/881955208 wait complete. 2024-03-18T16:36:24.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:36:24.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:36:25.004 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:25.004 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:26.005 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.180+0000 7fe4e3200640 1 Processor -- start 2024-03-18T16:36:26.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.180+0000 7fe4e3200640 1 -- start start 2024-03-18T16:36:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.181+0000 7fe4e3200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe4e4071700 0x7fe4e4072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.181+0000 7fe4e3200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4161d30 0x7fe4e4072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.181+0000 7fe4e3200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4e40c4190 con 0x7fe4e4158c00 2024-03-18T16:36:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.181+0000 7fe4e3200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fe4e4158fe0 con 0x7fe4e4071700 2024-03-18T16:36:26.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.181+0000 7fe4e3200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe4e406f3b0 con 0x7fe4e4161d30 2024-03-18T16:36:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.182+0000 7fe4e1e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe4e4158c00 0x7fe4e406f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:44002/0 (socket says 172.21.15.102:44002) 2024-03-18T16:36:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.182+0000 7fe4e1e00640 1 -- 172.21.15.102:0/445085034 learned_addr learned my addr 172.21.15.102:0/445085034 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.182+0000 7fe4e2800640 1 --2- 172.21.15.102:0/445085034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4161d30 0x7fe4e4072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.182+0000 7fe4e1400640 1 --2- 172.21.15.102:0/445085034 >> v2:172.21.15.102:3300/0 conn(0x7fe4e4071700 0x7fe4e4072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:26.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.185+0000 7fe4e0a00640 1 -- 172.21.15.102:0/445085034 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1646361544 0 0) 0x7fe4e40c4190 con 0x7fe4e4158c00 2024-03-18T16:36:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.185+0000 7fe4e0a00640 1 -- 172.21.15.102:0/445085034 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4c8003520 con 0x7fe4e4158c00 2024-03-18T16:36:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.186+0000 7fe4e2800640 1 -- 172.21.15.102:0/445085034 >> v2:172.21.15.102:3300/0 conn(0x7fe4e4071700 msgr2=0x7fe4e4072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.186+0000 7fe4e2800640 1 --2- 172.21.15.102:0/445085034 >> v2:172.21.15.102:3300/0 conn(0x7fe4e4071700 0x7fe4e4072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.186+0000 7fe4e2800640 1 -- 172.21.15.102:0/445085034 >> v1:172.21.15.154:6789/0 conn(0x7fe4e4158c00 legacy=0x7fe4e406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.186+0000 7fe4e2800640 1 -- 172.21.15.102:0/445085034 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e406f060 con 0x7fe4e4161d30 2024-03-18T16:36:26.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.189+0000 7fe4e2800640 1 --2- 172.21.15.102:0/445085034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4161d30 0x7fe4e4072fb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009d90 tx=0x7fe4dc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=698685b7d7514207 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.193+0000 7fe4e0a00640 1 -- 172.21.15.102:0/445085034 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc039070 con 0x7fe4e4161d30 2024-03-18T16:36:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.193+0000 7fe4e0a00640 1 -- 172.21.15.102:0/445085034 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4dc02cc50 con 0x7fe4e4161d30 2024-03-18T16:36:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.193+0000 7fe4e0a00640 1 -- 172.21.15.102:0/445085034 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc034440 con 0x7fe4e4161d30 2024-03-18T16:36:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.193+0000 7fe4e3200640 1 -- 172.21.15.102:0/445085034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4161d30 msgr2=0x7fe4e4072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.193+0000 7fe4e3200640 1 --2- 172.21.15.102:0/445085034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4161d30 0x7fe4e4072fb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009d90 tx=0x7fe4dc02bea0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.193+0000 7fe4e3200640 1 -- 172.21.15.102:0/445085034 shutdown_connections 2024-03-18T16:36:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.193+0000 7fe4e3200640 1 --2- 172.21.15.102:0/445085034 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4161d30 0x7fe4e4072fb0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.193+0000 7fe4e3200640 1 --2- 172.21.15.102:0/445085034 >> v2:172.21.15.102:3300/0 conn(0x7fe4e4071700 0x7fe4e4072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.193+0000 7fe4e3200640 1 -- 172.21.15.102:0/445085034 >> 172.21.15.102:0/445085034 conn(0x7fe4e40cb890 msgr2=0x7fe4e40cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.194+0000 7fe4e3200640 1 -- 172.21.15.102:0/445085034 shutdown_connections 2024-03-18T16:36:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.194+0000 7fe4e3200640 1 -- 172.21.15.102:0/445085034 wait complete. 2024-03-18T16:36:26.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.196+0000 7fe4e3200640 1 Processor -- start 2024-03-18T16:36:26.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.196+0000 7fe4e3200640 1 -- start start 2024-03-18T16:36:26.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.198+0000 7fe4e3200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4e4071700 0x7fe4e415d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:26.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.198+0000 7fe4e3200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4158c00 0x7fe4e414aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:26.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.198+0000 7fe4e3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4e4161d30 0x7fe4e414b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:26.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.198+0000 7fe4e3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e4072270 con 0x7fe4e4161d30 2024-03-18T16:36:26.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.198+0000 7fe4e3200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e4071f90 con 0x7fe4e4071700 2024-03-18T16:36:26.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.198+0000 7fe4e3200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe4e4072550 con 0x7fe4e4158c00 2024-03-18T16:36:26.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.198+0000 7fe4e1400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4158c00 0x7fe4e414aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:26.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.198+0000 7fe4e1e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4e4071700 0x7fe4e415d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:26.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.199+0000 7fe4e1400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4158c00 0x7fe4e414aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:55260/0 (socket says 172.21.15.102:55260) 2024-03-18T16:36:26.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.199+0000 7fe4e1400640 1 -- 172.21.15.102:0/2021389444 learned_addr learned my addr 172.21.15.102:0/2021389444 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:26.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.199+0000 7fe4e2800640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4e4161d30 0x7fe4e414b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:26.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.203+0000 7fe4e2800640 1 -- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4158c00 msgr2=0x7fe4e414aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:26.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.203+0000 7fe4e2800640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4158c00 0x7fe4e414aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.203+0000 7fe4e2800640 1 -- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4e4071700 msgr2=0x7fe4e415d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:26.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.203+0000 7fe4e2800640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4e4071700 0x7fe4e415d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.203+0000 7fe4e2800640 1 -- 172.21.15.102:0/2021389444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e4156c00 con 0x7fe4e4161d30 2024-03-18T16:36:26.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.206+0000 7fe4e2800640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4e4161d30 0x7fe4e414b020 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc033000 tx=0x7fe4dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:26.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.209+0000 7fe4da000640 1 -- 172.21.15.102:0/2021389444 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc039050 con 0x7fe4e4161d30 2024-03-18T16:36:26.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.209+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e415f5b0 con 0x7fe4e4161d30 2024-03-18T16:36:26.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.209+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e415a580 con 0x7fe4e4161d30 2024-03-18T16:36:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.210+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e4074fd0 con 0x7fe4e4161d30 2024-03-18T16:36:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.211+0000 7fe4da000640 1 -- 172.21.15.102:0/2021389444 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4dc045070 con 0x7fe4e4161d30 2024-03-18T16:36:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.211+0000 7fe4da000640 1 -- 172.21.15.102:0/2021389444 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc009450 con 0x7fe4e4161d30 2024-03-18T16:36:26.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.505+0000 7fe4da000640 1 -- 172.21.15.102:0/2021389444 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe4dc03e610 con 0x7fe4e4161d30 2024-03-18T16:36:26.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.507+0000 7fe4da000640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe4c406bbd0 0x7fe4c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:26.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.507+0000 7fe4e1e00640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe4c406bbd0 0x7fe4c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:26.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.508+0000 7fe4e1e00640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe4c406bbd0 0x7fe4c406e090 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4005880 tx=0x7fe4d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:26.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.526+0000 7fe4da000640 1 -- 172.21.15.102:0/2021389444 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe4dc07a2e0 con 0x7fe4e4161d30 2024-03-18T16:36:26.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.527+0000 7fe4da000640 1 -- 172.21.15.102:0/2021389444 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe4dc07c240 con 0x7fe4e4161d30 2024-03-18T16:36:26.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.790+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4e415c190 con 0x7fe4e4161d30 2024-03-18T16:36:26.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.889+0000 7fe4da000640 1 -- 172.21.15.102:0/2021389444 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fe4dc0802e0 con 0x7fe4e4161d30 2024-03-18T16:36:26.891 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:26.891 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:26.891 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.892+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe4c406bbd0 msgr2=0x7fe4c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.892+0000 7fe4e3200640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe4c406bbd0 0x7fe4c406e090 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4005880 tx=0x7fe4d4009040 comp rx=0 tx=0).stop 2024-03-18T16:36:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.892+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4e4161d30 msgr2=0x7fe4e414b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.892+0000 7fe4e3200640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4e4161d30 0x7fe4e414b020 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc033000 tx=0x7fe4dc004a40 comp rx=0 tx=0).stop 2024-03-18T16:36:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.893+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 shutdown_connections 2024-03-18T16:36:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.893+0000 7fe4e3200640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe4c406bbd0 0x7fe4c406e090 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.893+0000 7fe4e3200640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4e4161d30 0x7fe4e414b020 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.893+0000 7fe4e3200640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe4e4158c00 0x7fe4e414aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.893+0000 7fe4e3200640 1 --2- 172.21.15.102:0/2021389444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4e4071700 0x7fe4e415d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.893+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 >> 172.21.15.102:0/2021389444 conn(0x7fe4e40cb890 msgr2=0x7fe4e4162260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.893+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 shutdown_connections 2024-03-18T16:36:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:26.893+0000 7fe4e3200640 1 -- 172.21.15.102:0/2021389444 wait complete. 2024-03-18T16:36:27.516 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:27.516 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:28.516 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:28.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.690+0000 7fc182800640 1 Processor -- start 2024-03-18T16:36:28.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.690+0000 7fc182800640 1 -- start start 2024-03-18T16:36:28.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.692+0000 7fc182800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:28.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.692+0000 7fc182800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc17c06fd80 0x7fc17c072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:28.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.692+0000 7fc182800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c0c4190 con 0x7fc17c071700 2024-03-18T16:36:28.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.692+0000 7fc182800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fc17c158fe0 con 0x7fc17c06fd80 2024-03-18T16:36:28.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.692+0000 7fc182800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc17c06f3b0 con 0x7fc17c158c00 2024-03-18T16:36:28.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.692+0000 7fc181400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:28.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.693+0000 7fc181400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:55274/0 (socket says 172.21.15.102:55274) 2024-03-18T16:36:28.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.693+0000 7fc181400640 1 -- 172.21.15.102:0/3544321581 learned_addr learned my addr 172.21.15.102:0/3544321581 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:28.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.693+0000 7fc180a00640 1 --2- 172.21.15.102:0/3544321581 >> v2:172.21.15.102:3300/0 conn(0x7fc17c06fd80 0x7fc17c072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:28.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.695+0000 7fc181400640 1 -- 172.21.15.102:0/3544321581 >> v2:172.21.15.102:3300/0 conn(0x7fc17c06fd80 msgr2=0x7fc17c072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:28.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.695+0000 7fc181400640 1 --2- 172.21.15.102:0/3544321581 >> v2:172.21.15.102:3300/0 conn(0x7fc17c06fd80 0x7fc17c072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:28.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.695+0000 7fc181400640 1 -- 172.21.15.102:0/3544321581 >> v1:172.21.15.102:6789/0 conn(0x7fc17c071700 legacy=0x7fc17c15e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:28.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.695+0000 7fc181400640 1 -- 172.21.15.102:0/3544321581 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c06f060 con 0x7fc17c158c00 2024-03-18T16:36:28.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.698+0000 7fc181400640 1 --2- 172.21.15.102:0/3544321581 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c06f840 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc170009870 tx=0x7fc17002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d2e95b2663bd9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:28.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.702+0000 7fc17be00640 1 -- 172.21.15.102:0/3544321581 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170034070 con 0x7fc17c158c00 2024-03-18T16:36:28.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.702+0000 7fc17be00640 1 -- 172.21.15.102:0/3544321581 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc170030840 con 0x7fc17c158c00 2024-03-18T16:36:28.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.702+0000 7fc17be00640 1 -- 172.21.15.102:0/3544321581 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170030c10 con 0x7fc17c158c00 2024-03-18T16:36:28.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.703+0000 7fc182800640 1 -- 172.21.15.102:0/3544321581 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 msgr2=0x7fc17c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.703+0000 7fc182800640 1 --2- 172.21.15.102:0/3544321581 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c06f840 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc170009870 tx=0x7fc17002b980 comp rx=0 tx=0).stop 2024-03-18T16:36:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.703+0000 7fc182800640 1 -- 172.21.15.102:0/3544321581 shutdown_connections 2024-03-18T16:36:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.703+0000 7fc182800640 1 --2- 172.21.15.102:0/3544321581 >> v2:172.21.15.102:3300/0 conn(0x7fc17c06fd80 0x7fc17c072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.703+0000 7fc182800640 1 --2- 172.21.15.102:0/3544321581 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c06f840 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.703+0000 7fc182800640 1 -- 172.21.15.102:0/3544321581 >> 172.21.15.102:0/3544321581 conn(0x7fc17c0cb890 msgr2=0x7fc17c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.703+0000 7fc182800640 1 -- 172.21.15.102:0/3544321581 shutdown_connections 2024-03-18T16:36:28.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.703+0000 7fc182800640 1 -- 172.21.15.102:0/3544321581 wait complete. 2024-03-18T16:36:28.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.705+0000 7fc182800640 1 Processor -- start 2024-03-18T16:36:28.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.705+0000 7fc182800640 1 -- start start 2024-03-18T16:36:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.706+0000 7fc182800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc17c06fd80 0x7fc17c15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.706+0000 7fc182800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc17c071700 0x7fc17c0caa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.707+0000 7fc182800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c1488b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.707+0000 7fc182800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c072270 con 0x7fc17c06fd80 2024-03-18T16:36:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.707+0000 7fc182800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c071f90 con 0x7fc17c071700 2024-03-18T16:36:28.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.707+0000 7fc182800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fc17c072550 con 0x7fc17c158c00 2024-03-18T16:36:28.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.707+0000 7fc181e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c1488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:28.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.707+0000 7fc180a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc17c071700 0x7fc17c0caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:28.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.708+0000 7fc181e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c1488b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:55278/0 (socket says 172.21.15.102:55278) 2024-03-18T16:36:28.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.708+0000 7fc181e00640 1 -- 172.21.15.102:0/2473671962 learned_addr learned my addr 172.21.15.102:0/2473671962 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:28.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.708+0000 7fc181400640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc17c06fd80 0x7fc17c15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:28.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.711+0000 7fc181e00640 1 -- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc17c071700 msgr2=0x7fc17c0caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.711+0000 7fc181e00640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc17c071700 0x7fc17c0caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.711+0000 7fc181e00640 1 -- 172.21.15.102:0/2473671962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc17c06fd80 msgr2=0x7fc17c15e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.711+0000 7fc181e00640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc17c06fd80 0x7fc17c15e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.711+0000 7fc181e00640 1 -- 172.21.15.102:0/2473671962 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c15f330 con 0x7fc17c158c00 2024-03-18T16:36:28.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.714+0000 7fc181e00640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c1488b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc17400c7d0 tx=0x7fc17400cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:28.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.718+0000 7fc179600640 1 -- 172.21.15.102:0/2473671962 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174003b70 con 0x7fc17c158c00 2024-03-18T16:36:28.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.718+0000 7fc179600640 1 -- 172.21.15.102:0/2473671962 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc174003d30 con 0x7fc17c158c00 2024-03-18T16:36:28.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.718+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17c15c700 con 0x7fc17c158c00 2024-03-18T16:36:28.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.718+0000 7fc179600640 1 -- 172.21.15.102:0/2473671962 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17400aa90 con 0x7fc17c158c00 2024-03-18T16:36:28.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.718+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc17c15e360 con 0x7fc17c158c00 2024-03-18T16:36:28.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.720+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc17c15cc20 con 0x7fc17c158c00 2024-03-18T16:36:28.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.739+0000 7fc179600640 1 -- 172.21.15.102:0/2473671962 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc17400ac30 con 0x7fc17c158c00 2024-03-18T16:36:28.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.741+0000 7fc179600640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15c06bc20 0x7fc15c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:28.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.741+0000 7fc181400640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15c06bc20 0x7fc15c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:28.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.742+0000 7fc181400640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15c06bc20 0x7fc15c06e0e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc170036000 tx=0x7fc170008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:28.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.743+0000 7fc179600640 1 -- 172.21.15.102:0/2473671962 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fc174057830 con 0x7fc17c158c00 2024-03-18T16:36:28.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:28.989+0000 7fc179600640 1 -- 172.21.15.102:0/2473671962 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc17c15cc20 con 0x7fc17c158c00 2024-03-18T16:36:29.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.234+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc17c15e560 con 0x7fc17c158c00 2024-03-18T16:36:29.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.249+0000 7fc179600640 1 -- 172.21.15.102:0/2473671962 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7fc174059790 con 0x7fc17c158c00 2024-03-18T16:36:29.250 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:29.250 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:29.251 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:29.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15c06bc20 msgr2=0x7fc15c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:29.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15c06bc20 0x7fc15c06e0e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc170036000 tx=0x7fc170008030 comp rx=0 tx=0).stop 2024-03-18T16:36:29.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 msgr2=0x7fc17c1488b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:29.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c1488b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc17400c7d0 tx=0x7fc17400cc90 comp rx=0 tx=0).stop 2024-03-18T16:36:29.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 shutdown_connections 2024-03-18T16:36:29.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fc15c06bc20 0x7fc15c06e0e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:29.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fc17c158c00 0x7fc17c1488b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:29.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc17c071700 0x7fc17c0caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:29.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 --2- 172.21.15.102:0/2473671962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc17c06fd80 0x7fc17c15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:29.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.251+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 >> 172.21.15.102:0/2473671962 conn(0x7fc17c0cb890 msgr2=0x7fc17c157ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:29.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.252+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 shutdown_connections 2024-03-18T16:36:29.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:29.252+0000 7fc182800640 1 -- 172.21.15.102:0/2473671962 wait complete. 2024-03-18T16:36:29.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:29.880 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:30.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:31.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.057+0000 7f3c90e00640 1 Processor -- start 2024-03-18T16:36:31.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.057+0000 7f3c90e00640 1 -- start start 2024-03-18T16:36:31.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.058+0000 7f3c90e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cfe70 0x7f3c8c15ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:31.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.058+0000 7f3c90e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3c8c0d0250 0x7f3c8c0c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:31.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.058+0000 7f3c90e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c8c0cc2f0 0x7f3c8c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:31.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.058+0000 7f3c90e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f3c8c0ccca0 con 0x7f3c8c0cc2f0 2024-03-18T16:36:31.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.058+0000 7f3c90e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3c8c0cabd0 con 0x7f3c8c0d0250 2024-03-18T16:36:31.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.058+0000 7f3c90e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8c0caea0 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.059+0000 7f3c8b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cfe70 0x7f3c8c15ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:31.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.059+0000 7f3c8aa00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3c8c0d0250 0x7f3c8c0c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:31.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.060+0000 7f3c8b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cfe70 0x7f3c8c15ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:40550/0 (socket says 172.21.15.102:40550) 2024-03-18T16:36:31.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.060+0000 7f3c8b400640 1 -- 172.21.15.102:0/843215428 learned_addr learned my addr 172.21.15.102:0/843215428 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:31.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.062+0000 7f3c8b400640 1 -- 172.21.15.102:0/843215428 >> v2:172.21.15.154:3300/0 conn(0x7f3c8c0d0250 msgr2=0x7f3c8c0c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:31.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.062+0000 7f3c8b400640 1 --2- 172.21.15.102:0/843215428 >> v2:172.21.15.154:3300/0 conn(0x7f3c8c0d0250 0x7f3c8c0c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.062+0000 7f3c8b400640 1 -- 172.21.15.102:0/843215428 >> v2:172.21.15.102:3300/0 conn(0x7f3c8c0cc2f0 msgr2=0x7f3c8c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:31.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.062+0000 7f3c8b400640 1 --2- 172.21.15.102:0/843215428 >> v2:172.21.15.102:3300/0 conn(0x7f3c8c0cc2f0 0x7f3c8c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.062+0000 7f3c8b400640 1 -- 172.21.15.102:0/843215428 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c0cb180 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.064+0000 7f3c8b400640 1 --2- 172.21.15.102:0/843215428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cfe70 0x7f3c8c15ae40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c0cd830 tx=0x7f3c7c02b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77c178299e6bbb94 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:31.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.068+0000 7f3c8a000640 1 -- 172.21.15.102:0/843215428 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c034070 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.068+0000 7f3c8a000640 1 -- 172.21.15.102:0/843215428 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c7c030850 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.068+0000 7f3c8a000640 1 -- 172.21.15.102:0/843215428 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c030c20 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.069+0000 7f3c90e00640 1 -- 172.21.15.102:0/843215428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cfe70 msgr2=0x7f3c8c15ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.069+0000 7f3c90e00640 1 --2- 172.21.15.102:0/843215428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cfe70 0x7f3c8c15ae40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c0cd830 tx=0x7f3c7c02b990 comp rx=0 tx=0).stop 2024-03-18T16:36:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.070+0000 7f3c90e00640 1 -- 172.21.15.102:0/843215428 shutdown_connections 2024-03-18T16:36:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.070+0000 7f3c90e00640 1 --2- 172.21.15.102:0/843215428 >> v2:172.21.15.102:3300/0 conn(0x7f3c8c0cc2f0 0x7f3c8c0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.070+0000 7f3c90e00640 1 --2- 172.21.15.102:0/843215428 >> v2:172.21.15.154:3300/0 conn(0x7f3c8c0d0250 0x7f3c8c0c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.070+0000 7f3c90e00640 1 --2- 172.21.15.102:0/843215428 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cfe70 0x7f3c8c15ae40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.070+0000 7f3c90e00640 1 -- 172.21.15.102:0/843215428 >> 172.21.15.102:0/843215428 conn(0x7f3c8c08c3e0 msgr2=0x7f3c8c06c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:31.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.070+0000 7f3c90e00640 1 -- 172.21.15.102:0/843215428 shutdown_connections 2024-03-18T16:36:31.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.070+0000 7f3c90e00640 1 -- 172.21.15.102:0/843215428 wait complete. 2024-03-18T16:36:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.071+0000 7f3c90e00640 1 Processor -- start 2024-03-18T16:36:31.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.071+0000 7f3c90e00640 1 -- start start 2024-03-18T16:36:31.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.072+0000 7f3c90e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cc2f0 0x7f3c8c0c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:31.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.072+0000 7f3c90e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8c0cfe70 0x7f3c8c0c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:31.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.072+0000 7f3c90e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c8c0d0250 0x7f3c8c155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:31.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.072+0000 7f3c90e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c0cba20 con 0x7f3c8c0d0250 2024-03-18T16:36:31.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.072+0000 7f3c90e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c0cb740 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.072+0000 7f3c90e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8c0cbd00 con 0x7f3c8c0cc2f0 2024-03-18T16:36:31.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.073+0000 7f3c8b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cc2f0 0x7f3c8c0c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:31.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.073+0000 7f3c8aa00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8c0cfe70 0x7f3c8c0c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:31.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.073+0000 7f3c8b400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cc2f0 0x7f3c8c0c3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:40552/0 (socket says 172.21.15.102:40552) 2024-03-18T16:36:31.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.073+0000 7f3c8b400640 1 -- 172.21.15.102:0/758738067 learned_addr learned my addr 172.21.15.102:0/758738067 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:31.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.073+0000 7f3c8be00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c8c0d0250 0x7f3c8c155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:31.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.077+0000 7f3c8aa00640 1 -- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cc2f0 msgr2=0x7f3c8c0c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:31.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.077+0000 7f3c8aa00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cc2f0 0x7f3c8c0c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.077+0000 7f3c8aa00640 1 -- 172.21.15.102:0/758738067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c8c0d0250 msgr2=0x7f3c8c155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:31.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.077+0000 7f3c8aa00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c8c0d0250 0x7f3c8c155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.077+0000 7f3c8aa00640 1 -- 172.21.15.102:0/758738067 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c1eff20 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.080+0000 7f3c8aa00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8c0cfe70 0x7f3c8c0c4040 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800e870 tx=0x7f3c7800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:31.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.084+0000 7f3c83400640 1 -- 172.21.15.102:0/758738067 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c78017070 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.084+0000 7f3c83400640 1 -- 172.21.15.102:0/758738067 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c7800ee60 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.084+0000 7f3c83400640 1 -- 172.21.15.102:0/758738067 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7800cb60 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.084+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8c1f0200 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.084+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c8c1f07c0 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.086+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c8c074fd0 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.356+0000 7f3c83400640 1 -- 172.21.15.102:0/758738067 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3c7801a070 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.359+0000 7f3c83400640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:31.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.359+0000 7f3c8b400640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:31.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.360+0000 7f3c8b400640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009630 tx=0x7f3c7c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:31.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.361+0000 7f3c83400640 1 -- 172.21.15.102:0/758738067 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f3c780574f0 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.371+0000 7f3c83400640 1 -- 172.21.15.102:0/758738067 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3c78059450 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.761+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c8c0ccf90 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.776+0000 7f3c83400640 1 -- 172.21.15.102:0/758738067 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6287 (secure 0 0 0) 0x7f3c7805d4f0 con 0x7f3c8c0cfe70 2024-03-18T16:36:31.777 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:31.777 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:24:09.826257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:24:09.826252+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270,4256,4253,4247]},"id":1}]} 2024-03-18T16:36:31.777 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 25 2024-03-18T16:36:31.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c6c06bc20 msgr2=0x7f3c6c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009630 tx=0x7f3c7c008030 comp rx=0 tx=0).stop 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8c0cfe70 msgr2=0x7f3c8c0c4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8c0cfe70 0x7f3c8c0c4040 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f3c7800e870 tx=0x7f3c7800ed30 comp rx=0 tx=0).stop 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 shutdown_connections 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f3c6c06bc20 0x7f3c6c06e0e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c8c0d0250 0x7f3c8c155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c8c0cfe70 0x7f3c8c0c4040 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 --2- 172.21.15.102:0/758738067 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f3c8c0cc2f0 0x7f3c8c0c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 >> 172.21.15.102:0/758738067 conn(0x7f3c8c08c3e0 msgr2=0x7f3c8c0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 shutdown_connections 2024-03-18T16:36:31.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:31.778+0000 7f3c90e00640 1 -- 172.21.15.102:0/758738067 wait complete. 2024-03-18T16:36:32.399 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:24:09.826257+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:24:09.826252+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270, 4256, 4253, 4247]} 2024-03-18T16:36:32.399 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:33.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:33.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.578+0000 7f9d58c00640 1 Processor -- start 2024-03-18T16:36:33.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.578+0000 7f9d58c00640 1 -- start start 2024-03-18T16:36:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.579+0000 7f9d58c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 0x7f9d54072a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.579+0000 7f9d58c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d540c4190 con 0x7f9d54161d30 2024-03-18T16:36:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.579+0000 7f9d58c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5406f060 con 0x7f9d54158c00 2024-03-18T16:36:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.579+0000 7f9d58c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9d54158fe0 con 0x7f9d54071700 2024-03-18T16:36:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.580+0000 7f9d53400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f9d54158c00 0x7f9d5406f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:40394/0 (socket says 172.21.15.102:40394) 2024-03-18T16:36:33.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.580+0000 7f9d53400640 1 -- 172.21.15.102:0/4172052197 learned_addr learned my addr 172.21.15.102:0/4172052197 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:33.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.580+0000 7f9d52a00640 1 --2- 172.21.15.102:0/4172052197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 0x7f9d54072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:33.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.583+0000 7f9d52000640 1 -- 172.21.15.102:0/4172052197 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 244494865 0 0) 0x7f9d5406f060 con 0x7f9d54158c00 2024-03-18T16:36:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.583+0000 7f9d52000640 1 -- 172.21.15.102:0/4172052197 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d38003520 con 0x7f9d54158c00 2024-03-18T16:36:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.584+0000 7f9d52a00640 1 -- 172.21.15.102:0/4172052197 >> v1:172.21.15.154:6789/0 conn(0x7f9d54158c00 legacy=0x7f9d5406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.584+0000 7f9d52a00640 1 -- 172.21.15.102:0/4172052197 >> v1:172.21.15.102:6789/0 conn(0x7f9d54161d30 legacy=0x7f9d54169110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:33.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.584+0000 7f9d52a00640 1 -- 172.21.15.102:0/4172052197 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d54071cb0 con 0x7f9d54071700 2024-03-18T16:36:33.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.587+0000 7f9d52a00640 1 --2- 172.21.15.102:0/4172052197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 0x7f9d54072a70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9d400097d0 tx=0x7f9d4002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2b1d703e8729bc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:33.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.591+0000 7f9d52000640 1 -- 172.21.15.102:0/4172052197 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40034070 con 0x7f9d54071700 2024-03-18T16:36:33.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.591+0000 7f9d52000640 1 -- 172.21.15.102:0/4172052197 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d40030800 con 0x7f9d54071700 2024-03-18T16:36:33.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.591+0000 7f9d52000640 1 -- 172.21.15.102:0/4172052197 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40030bd0 con 0x7f9d54071700 2024-03-18T16:36:33.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.592+0000 7f9d58c00640 1 -- 172.21.15.102:0/4172052197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 msgr2=0x7f9d54072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:33.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.592+0000 7f9d58c00640 1 --2- 172.21.15.102:0/4172052197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 0x7f9d54072a70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9d400097d0 tx=0x7f9d4002b940 comp rx=0 tx=0).stop 2024-03-18T16:36:33.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.592+0000 7f9d58c00640 1 -- 172.21.15.102:0/4172052197 shutdown_connections 2024-03-18T16:36:33.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.592+0000 7f9d58c00640 1 --2- 172.21.15.102:0/4172052197 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 0x7f9d54072a70 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:33.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.592+0000 7f9d58c00640 1 -- 172.21.15.102:0/4172052197 >> 172.21.15.102:0/4172052197 conn(0x7f9d540cb890 msgr2=0x7f9d540cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:33.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.592+0000 7f9d58c00640 1 -- 172.21.15.102:0/4172052197 shutdown_connections 2024-03-18T16:36:33.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.592+0000 7f9d58c00640 1 -- 172.21.15.102:0/4172052197 wait complete. 2024-03-18T16:36:33.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.595+0000 7f9d58c00640 1 Processor -- start 2024-03-18T16:36:33.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.595+0000 7f9d58c00640 1 -- start start 2024-03-18T16:36:33.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.596+0000 7f9d58c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 0x7f9d540c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:33.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.596+0000 7f9d58c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d54158c00 0x7f9d540c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:33.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.596+0000 7f9d58c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d54161d30 0x7f9d540cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:33.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.597+0000 7f9d58c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9d54072270 con 0x7f9d54161d30 2024-03-18T16:36:33.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.597+0000 7f9d58c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5406f3b0 con 0x7f9d54158c00 2024-03-18T16:36:33.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.597+0000 7f9d58c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9d54072550 con 0x7f9d54071700 2024-03-18T16:36:33.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.597+0000 7f9d52a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d54158c00 0x7f9d540c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:33.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.597+0000 7f9d53400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 0x7f9d540c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:33.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.597+0000 7f9d52a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d54158c00 0x7f9d540c9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:55920/0 (socket says 172.21.15.102:55920) 2024-03-18T16:36:33.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.597+0000 7f9d52a00640 1 -- 172.21.15.102:0/3166725731 learned_addr learned my addr 172.21.15.102:0/3166725731 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:33.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.598+0000 7f9d53e00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d54161d30 0x7f9d540cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:33.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.601+0000 7f9d52a00640 1 -- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 msgr2=0x7f9d540c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:33.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.601+0000 7f9d52a00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 0x7f9d540c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:33.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.601+0000 7f9d52a00640 1 -- 172.21.15.102:0/3166725731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d54161d30 msgr2=0x7f9d540cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:33.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.601+0000 7f9d52a00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d54161d30 0x7f9d540cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:33.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.601+0000 7f9d52a00640 1 -- 172.21.15.102:0/3166725731 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5414a400 con 0x7f9d54158c00 2024-03-18T16:36:33.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.603+0000 7f9d52a00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d54158c00 0x7f9d540c9b30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9d400098f0 tx=0x7f9d400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:33.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.608+0000 7f9d4b400640 1 -- 172.21.15.102:0/3166725731 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40034040 con 0x7f9d54158c00 2024-03-18T16:36:33.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.608+0000 7f9d4b400640 1 -- 172.21.15.102:0/3166725731 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4003ada0 con 0x7f9d54158c00 2024-03-18T16:36:33.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.608+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5414a6e0 con 0x7f9d54158c00 2024-03-18T16:36:33.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.608+0000 7f9d4b400640 1 -- 172.21.15.102:0/3166725731 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40006e10 con 0x7f9d54158c00 2024-03-18T16:36:33.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.608+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d5415fc30 con 0x7f9d54158c00 2024-03-18T16:36:33.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.609+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d54067c20 con 0x7f9d54158c00 2024-03-18T16:36:33.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.635+0000 7f9d4b400640 1 -- 172.21.15.102:0/3166725731 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9d4004b060 con 0x7f9d54158c00 2024-03-18T16:36:33.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.636+0000 7f9d4b400640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d3406bc20 0x7f9d3406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:33.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.637+0000 7f9d53400640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d3406bc20 0x7f9d3406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:33.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.637+0000 7f9d53400640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d3406bc20 0x7f9d3406e0e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9d440011e0 tx=0x7f9d44007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:33.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.884+0000 7f9d4b400640 1 -- 172.21.15.102:0/3166725731 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9d40079060 con 0x7f9d54158c00 2024-03-18T16:36:33.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:33.917+0000 7f9d4b400640 1 -- 172.21.15.102:0/3166725731 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9d54067c20 con 0x7f9d54158c00 2024-03-18T16:36:34.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.203+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9d541570d0 con 0x7f9d54158c00 2024-03-18T16:36:34.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.275+0000 7f9d4b400640 1 -- 172.21.15.102:0/3166725731 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6240 (secure 0 0 0) 0x7f9d541570d0 con 0x7f9d54158c00 2024-03-18T16:36:34.277 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:34.277 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:36:32.278787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":41,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4247,4253,4256,4270]},"id":1}]} 2024-03-18T16:36:34.277 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 26 2024-03-18T16:36:34.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.278+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d3406bc20 msgr2=0x7f9d3406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.278+0000 7f9d58c00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d3406bc20 0x7f9d3406e0e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9d440011e0 tx=0x7f9d44007d30 comp rx=0 tx=0).stop 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.278+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d54158c00 msgr2=0x7f9d540c9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.278+0000 7f9d58c00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d54158c00 0x7f9d540c9b30 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9d400098f0 tx=0x7f9d400046c0 comp rx=0 tx=0).stop 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.279+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 shutdown_connections 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.279+0000 7f9d58c00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9d3406bc20 0x7f9d3406e0e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.279+0000 7f9d58c00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d54161d30 0x7f9d540cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.279+0000 7f9d58c00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d54158c00 0x7f9d540c9b30 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.279+0000 7f9d58c00640 1 --2- 172.21.15.102:0/3166725731 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9d54071700 0x7f9d540c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.279+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 >> 172.21.15.102:0/3166725731 conn(0x7f9d540cb890 msgr2=0x7f9d54156640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.279+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 shutdown_connections 2024-03-18T16:36:34.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:34.279+0000 7f9d58c00640 1 -- 172.21.15.102:0/3166725731 wait complete. 2024-03-18T16:36:34.901 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:36:32.278787+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 41, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4247, 4253, 4256, 4270]} 2024-03-18T16:36:34.901 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:35.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:36.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.078+0000 7f0e47200640 1 Processor -- start 2024-03-18T16:36:36.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.078+0000 7f0e47200640 1 -- start start 2024-03-18T16:36:36.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.079+0000 7f0e47200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0e4015a310 0x7f0e4006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:36.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.079+0000 7f0e47200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e400ca500 con 0x7f0e4015a6f0 2024-03-18T16:36:36.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.079+0000 7f0e47200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e400ca7a0 con 0x7f0e4015baa0 2024-03-18T16:36:36.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.079+0000 7f0e47200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0e400cd410 con 0x7f0e4015a310 2024-03-18T16:36:36.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.080+0000 7f0e45e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0e4015a310 0x7f0e4006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:36.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.080+0000 7f0e46800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0e4015baa0 0x7f0e4015bf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:40402/0 (socket says 172.21.15.102:40402) 2024-03-18T16:36:36.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.080+0000 7f0e46800640 1 -- 172.21.15.102:0/2003162000 learned_addr learned my addr 172.21.15.102:0/2003162000 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.084+0000 7f0e45e00640 1 -- 172.21.15.102:0/2003162000 >> v1:172.21.15.154:6789/0 conn(0x7f0e4015baa0 legacy=0x7f0e4015bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.084+0000 7f0e45e00640 1 -- 172.21.15.102:0/2003162000 >> v1:172.21.15.102:6789/0 conn(0x7f0e4015a6f0 legacy=0x7f0e4015ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.084+0000 7f0e45e00640 1 -- 172.21.15.102:0/2003162000 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e400caa80 con 0x7f0e4015a310 2024-03-18T16:36:36.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.088+0000 7f0e45e00640 1 --2- 172.21.15.102:0/2003162000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015a310 0x7f0e4006d3e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0e340097a0 tx=0x7f0e3402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a675714cb6ef9d8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.092+0000 7f0e44a00640 1 -- 172.21.15.102:0/2003162000 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34035020 con 0x7f0e4015a310 2024-03-18T16:36:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.093+0000 7f0e44a00640 1 -- 172.21.15.102:0/2003162000 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e34031830 con 0x7f0e4015a310 2024-03-18T16:36:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.093+0000 7f0e44a00640 1 -- 172.21.15.102:0/2003162000 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34031c00 con 0x7f0e4015a310 2024-03-18T16:36:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.093+0000 7f0e47200640 1 -- 172.21.15.102:0/2003162000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015a310 msgr2=0x7f0e4006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.093+0000 7f0e47200640 1 --2- 172.21.15.102:0/2003162000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015a310 0x7f0e4006d3e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0e340097a0 tx=0x7f0e3402b8d0 comp rx=0 tx=0).stop 2024-03-18T16:36:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.094+0000 7f0e47200640 1 -- 172.21.15.102:0/2003162000 shutdown_connections 2024-03-18T16:36:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.094+0000 7f0e47200640 1 --2- 172.21.15.102:0/2003162000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015a310 0x7f0e4006d3e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.094+0000 7f0e47200640 1 -- 172.21.15.102:0/2003162000 >> 172.21.15.102:0/2003162000 conn(0x7f0e4008c1a0 msgr2=0x7f0e40066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.094+0000 7f0e47200640 1 -- 172.21.15.102:0/2003162000 shutdown_connections 2024-03-18T16:36:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.094+0000 7f0e47200640 1 -- 172.21.15.102:0/2003162000 wait complete. 2024-03-18T16:36:36.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.096+0000 7f0e47200640 1 Processor -- start 2024-03-18T16:36:36.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.097+0000 7f0e47200640 1 -- start start 2024-03-18T16:36:36.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.098+0000 7f0e47200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e4015a310 0x7f0e400c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:36.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.098+0000 7f0e47200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4015a6f0 0x7f0e4014bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:36.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.098+0000 7f0e47200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015baa0 0x7f0e4014c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:36.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.098+0000 7f0e47200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4006d920 con 0x7f0e4015a310 2024-03-18T16:36:36.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.098+0000 7f0e47200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e400cd130 con 0x7f0e4015baa0 2024-03-18T16:36:36.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.098+0000 7f0e47200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0e401548c0 con 0x7f0e4015a6f0 2024-03-18T16:36:36.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.099+0000 7f0e46800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015baa0 0x7f0e4014c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:36.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.100+0000 7f0e46800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015baa0 0x7f0e4014c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:55932/0 (socket says 172.21.15.102:55932) 2024-03-18T16:36:36.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.100+0000 7f0e46800640 1 -- 172.21.15.102:0/936773716 learned_addr learned my addr 172.21.15.102:0/936773716 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:36.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.105+0000 7f0e45e00640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e4015a310 0x7f0e400c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:36.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.105+0000 7f0e46800640 1 -- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4015a6f0 msgr2=0x7f0e4014bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:36.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.105+0000 7f0e46800640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4015a6f0 0x7f0e4014bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:36.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.105+0000 7f0e46800640 1 -- 172.21.15.102:0/936773716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e4015a310 msgr2=0x7f0e400c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:36.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.105+0000 7f0e46800640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e4015a310 0x7f0e400c7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:36.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.105+0000 7f0e46800640 1 -- 172.21.15.102:0/936773716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e400c29a0 con 0x7f0e4015baa0 2024-03-18T16:36:36.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.108+0000 7f0e46800640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015baa0 0x7f0e4014c190 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00ca50 tx=0x7f0e3c00cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:36.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.113+0000 7f0e3a000640 1 -- 172.21.15.102:0/936773716 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c017070 con 0x7f0e4015baa0 2024-03-18T16:36:36.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.113+0000 7f0e3a000640 1 -- 172.21.15.102:0/936773716 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e3c00abb0 con 0x7f0e4015baa0 2024-03-18T16:36:36.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.113+0000 7f0e3a000640 1 -- 172.21.15.102:0/936773716 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3c012400 con 0x7f0e4015baa0 2024-03-18T16:36:36.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.113+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e400c2c60 con 0x7f0e4015baa0 2024-03-18T16:36:36.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.113+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e400c73b0 con 0x7f0e4015baa0 2024-03-18T16:36:36.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.114+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e400ca1d0 con 0x7f0e4015baa0 2024-03-18T16:36:36.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.147+0000 7f0e3a000640 1 -- 172.21.15.102:0/936773716 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0e3c01a070 con 0x7f0e4015baa0 2024-03-18T16:36:36.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.148+0000 7f0e3a000640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0e2406bc20 0x7f0e2406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:36.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.149+0000 7f0e45e00640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0e2406bc20 0x7f0e2406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:36.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.150+0000 7f0e45e00640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0e2406bc20 0x7f0e2406e0e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0e34036000 tx=0x7f0e34008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.386+0000 7f0e3a000640 1 -- 172.21.15.102:0/936773716 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f0e3c057000 con 0x7f0e4015baa0 2024-03-18T16:36:36.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.397+0000 7f0e3a000640 1 -- 172.21.15.102:0/936773716 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0e400ca1d0 con 0x7f0e4015baa0 2024-03-18T16:36:36.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.665+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e400cff60 con 0x7f0e4015baa0 2024-03-18T16:36:36.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.680+0000 7f0e3a000640 1 -- 172.21.15.102:0/936773716 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6240 (secure 0 0 0) 0x7f0e400cff60 con 0x7f0e4015baa0 2024-03-18T16:36:36.681 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:36.682 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:36:32.278787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":41,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4247,4253,4256,4270]},"id":1}]} 2024-03-18T16:36:36.682 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 26 2024-03-18T16:36:36.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.683+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0e2406bc20 msgr2=0x7f0e2406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.683+0000 7f0e47200640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0e2406bc20 0x7f0e2406e0e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0e34036000 tx=0x7f0e34008030 comp rx=0 tx=0).stop 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.683+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015baa0 msgr2=0x7f0e4014c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.683+0000 7f0e47200640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015baa0 0x7f0e4014c190 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c00ca50 tx=0x7f0e3c00cf10 comp rx=0 tx=0).stop 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.684+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 shutdown_connections 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.684+0000 7f0e47200640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0e2406bc20 0x7f0e2406e0e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.684+0000 7f0e47200640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e4015baa0 0x7f0e4014c190 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.684+0000 7f0e47200640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0e4015a6f0 0x7f0e4014bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.684+0000 7f0e47200640 1 --2- 172.21.15.102:0/936773716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e4015a310 0x7f0e400c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.684+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 >> 172.21.15.102:0/936773716 conn(0x7f0e4008c1a0 msgr2=0x7f0e4014e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.684+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 shutdown_connections 2024-03-18T16:36:36.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:36.684+0000 7f0e47200640 1 -- 172.21.15.102:0/936773716 wait complete. 2024-03-18T16:36:37.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:36:32.278787+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 41, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4247, 4253, 4256, 4270]} 2024-03-18T16:36:37.313 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:38.314 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.485+0000 7f9e24e00640 1 Processor -- start 2024-03-18T16:36:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.485+0000 7f9e24e00640 1 -- start start 2024-03-18T16:36:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.486+0000 7f9e24e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2015ad60 0x7f9e200c5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.486+0000 7f9e24e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e2006bf60 con 0x7f9e2006de50 2024-03-18T16:36:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.486+0000 7f9e24e00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e200c94b0 con 0x7f9e2006b470 2024-03-18T16:36:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.486+0000 7f9e24e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9e200ca0a0 con 0x7f9e2015ad60 2024-03-18T16:36:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.487+0000 7f9e1fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2015ad60 0x7f9e200c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.487+0000 7f9e1f400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f9e2006b470 0x7f9e2006b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:40414/0 (socket says 172.21.15.102:40414) 2024-03-18T16:36:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.487+0000 7f9e1f400640 1 -- 172.21.15.102:0/782186637 learned_addr learned my addr 172.21.15.102:0/782186637 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.490+0000 7f9e1fe00640 1 -- 172.21.15.102:0/782186637 >> v1:172.21.15.154:6789/0 conn(0x7f9e2006b470 legacy=0x7f9e2006b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.490+0000 7f9e1fe00640 1 -- 172.21.15.102:0/782186637 >> v1:172.21.15.102:6789/0 conn(0x7f9e2006de50 legacy=0x7f9e20154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.490+0000 7f9e1fe00640 1 -- 172.21.15.102:0/782186637 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e200c9770 con 0x7f9e2015ad60 2024-03-18T16:36:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.493+0000 7f9e1fe00640 1 --2- 172.21.15.102:0/782186637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2015ad60 0x7f9e200c5190 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009dc0 tx=0x7f9e1802bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebcab27ce1f4255 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.497+0000 7f9e1e000640 1 -- 172.21.15.102:0/782186637 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18039070 con 0x7f9e2015ad60 2024-03-18T16:36:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.497+0000 7f9e1e000640 1 -- 172.21.15.102:0/782186637 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e1802ccc0 con 0x7f9e2015ad60 2024-03-18T16:36:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.497+0000 7f9e1e000640 1 -- 172.21.15.102:0/782186637 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18034660 con 0x7f9e2015ad60 2024-03-18T16:36:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.497+0000 7f9e24e00640 1 -- 172.21.15.102:0/782186637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2015ad60 msgr2=0x7f9e200c5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.497+0000 7f9e24e00640 1 --2- 172.21.15.102:0/782186637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2015ad60 0x7f9e200c5190 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009dc0 tx=0x7f9e1802bf10 comp rx=0 tx=0).stop 2024-03-18T16:36:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.497+0000 7f9e24e00640 1 -- 172.21.15.102:0/782186637 shutdown_connections 2024-03-18T16:36:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.497+0000 7f9e24e00640 1 --2- 172.21.15.102:0/782186637 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2015ad60 0x7f9e200c5190 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.497+0000 7f9e24e00640 1 -- 172.21.15.102:0/782186637 >> 172.21.15.102:0/782186637 conn(0x7f9e200727e0 msgr2=0x7f9e20072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.498+0000 7f9e24e00640 1 -- 172.21.15.102:0/782186637 shutdown_connections 2024-03-18T16:36:38.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.498+0000 7f9e24e00640 1 -- 172.21.15.102:0/782186637 wait complete. 2024-03-18T16:36:38.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.499+0000 7f9e24e00640 1 Processor -- start 2024-03-18T16:36:38.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.499+0000 7f9e24e00640 1 -- start start 2024-03-18T16:36:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.500+0000 7f9e24e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2006b470 0x7f9e20152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.500+0000 7f9e24e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2006de50 0x7f9e20157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.500+0000 7f9e24e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e2015ad60 0x7f9e20155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.500+0000 7f9e24e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f9e20154860 con 0x7f9e2015ad60 2024-03-18T16:36:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.500+0000 7f9e24e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9e200c9d30 con 0x7f9e2006b470 2024-03-18T16:36:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.500+0000 7f9e24e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9e201549e0 con 0x7f9e2006de50 2024-03-18T16:36:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.501+0000 7f9e1ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2006de50 0x7f9e20157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.501+0000 7f9e1f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2006b470 0x7f9e20152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.501+0000 7f9e1fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e2015ad60 0x7f9e20155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.501+0000 7f9e1ea00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2006de50 0x7f9e20157290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:40594/0 (socket says 172.21.15.102:40594) 2024-03-18T16:36:38.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.501+0000 7f9e1ea00640 1 -- 172.21.15.102:0/764125921 learned_addr learned my addr 172.21.15.102:0/764125921 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.506+0000 7f9e1fe00640 1 -- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2006de50 msgr2=0x7f9e20157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.506+0000 7f9e1fe00640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2006de50 0x7f9e20157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.506+0000 7f9e1fe00640 1 -- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2006b470 msgr2=0x7f9e20152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.506+0000 7f9e1fe00640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2006b470 0x7f9e20152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:38.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.506+0000 7f9e1fe00640 1 -- 172.21.15.102:0/764125921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e201efb80 con 0x7f9e2015ad60 2024-03-18T16:36:38.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.508+0000 7f9e1fe00640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e2015ad60 0x7f9e20155cf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9e18033000 tx=0x7f9e18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.511+0000 7f9e17400640 1 -- 172.21.15.102:0/764125921 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18039050 con 0x7f9e2015ad60 2024-03-18T16:36:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.511+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e201efe60 con 0x7f9e2015ad60 2024-03-18T16:36:38.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.511+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e201f0420 con 0x7f9e2015ad60 2024-03-18T16:36:38.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.512+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e20074fd0 con 0x7f9e2015ad60 2024-03-18T16:36:38.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.725+0000 7f9e17400640 1 -- 172.21.15.102:0/764125921 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e18045070 con 0x7f9e2015ad60 2024-03-18T16:36:38.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.725+0000 7f9e17400640 1 -- 172.21.15.102:0/764125921 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1803e830 con 0x7f9e2015ad60 2024-03-18T16:36:38.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.783+0000 7f9e17400640 1 -- 172.21.15.102:0/764125921 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9e1803e9d0 con 0x7f9e2015ad60 2024-03-18T16:36:38.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.785+0000 7f9e17400640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9e0006bbd0 0x7f9e0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:38.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.785+0000 7f9e1f400640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9e0006bbd0 0x7f9e0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:38.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.786+0000 7f9e1f400640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9e0006bbd0 0x7f9e0006e090 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9e10005880 tx=0x7f9e10009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:38.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.820+0000 7f9e17400640 1 -- 172.21.15.102:0/764125921 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f9e1807a750 con 0x7f9e2015ad60 2024-03-18T16:36:38.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:38.820+0000 7f9e17400640 1 -- 172.21.15.102:0/764125921 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9e1807c6b0 con 0x7f9e2015ad60 2024-03-18T16:36:39.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.075+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e200c7ea0 con 0x7f9e2015ad60 2024-03-18T16:36:39.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.091+0000 7f9e17400640 1 -- 172.21.15.102:0/764125921 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6240 (secure 0 0 0) 0x7f9e18080750 con 0x7f9e2015ad60 2024-03-18T16:36:39.093 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:39.093 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:36:32.278787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":41,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4247,4253,4256,4270]},"id":1}]} 2024-03-18T16:36:39.093 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 26 2024-03-18T16:36:39.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9e0006bbd0 msgr2=0x7f9e0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9e0006bbd0 0x7f9e0006e090 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9e10005880 tx=0x7f9e10009040 comp rx=0 tx=0).stop 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e2015ad60 msgr2=0x7f9e20155cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e2015ad60 0x7f9e20155cf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9e18033000 tx=0x7f9e18004a40 comp rx=0 tx=0).stop 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 shutdown_connections 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f9e0006bbd0 0x7f9e0006e090 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e2015ad60 0x7f9e20155cf0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9e2006de50 0x7f9e20157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 --2- 172.21.15.102:0/764125921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2006b470 0x7f9e20152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 >> 172.21.15.102:0/764125921 conn(0x7f9e200727e0 msgr2=0x7f9e200cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 shutdown_connections 2024-03-18T16:36:39.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:39.094+0000 7f9e24e00640 1 -- 172.21.15.102:0/764125921 wait complete. 2024-03-18T16:36:39.722 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:36:32.278787+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 41, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4247, 4253, 4256, 4270]} 2024-03-18T16:36:39.723 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:40.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:40.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.907+0000 7f4b3a800640 1 Processor -- start 2024-03-18T16:36:40.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.907+0000 7f4b3a800640 1 -- start start 2024-03-18T16:36:40.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.908+0000 7f4b3a800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4b34141650 0x7f4b34158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:40.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.908+0000 7f4b3a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b340c4fa0 0x7f4b340c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:40.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.908+0000 7f4b3a800640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b340c72d0 con 0x7f4b3415c3a0 2024-03-18T16:36:40.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.909+0000 7f4b3a800640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4b34159370 con 0x7f4b34141650 2024-03-18T16:36:40.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.909+0000 7f4b3a800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4b34158ad0 con 0x7f4b340c4fa0 2024-03-18T16:36:40.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.909+0000 7f4b39e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b340c4fa0 0x7f4b340c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:40.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.909+0000 7f4b38a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4b3415c3a0 0x7f4b340c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58638/0 (socket says 172.21.15.102:58638) 2024-03-18T16:36:40.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.909+0000 7f4b38a00640 1 -- 172.21.15.102:0/2265918729 learned_addr learned my addr 172.21.15.102:0/2265918729 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:40.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.910+0000 7f4b39400640 1 --2- 172.21.15.102:0/2265918729 >> v2:172.21.15.154:3300/0 conn(0x7f4b34141650 0x7f4b34158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:40.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.913+0000 7f4b39e00640 1 -- 172.21.15.102:0/2265918729 >> v2:172.21.15.154:3300/0 conn(0x7f4b34141650 msgr2=0x7f4b34158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:40.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.913+0000 7f4b39e00640 1 --2- 172.21.15.102:0/2265918729 >> v2:172.21.15.154:3300/0 conn(0x7f4b34141650 0x7f4b34158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:40.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.913+0000 7f4b39e00640 1 -- 172.21.15.102:0/2265918729 >> v1:172.21.15.154:6789/0 conn(0x7f4b3415c3a0 legacy=0x7f4b340c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:40.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.913+0000 7f4b39e00640 1 -- 172.21.15.102:0/2265918729 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b34158db0 con 0x7f4b340c4fa0 2024-03-18T16:36:40.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.916+0000 7f4b39e00640 1 --2- 172.21.15.102:0/2265918729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b340c4fa0 0x7f4b340c5430 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009d90 tx=0x7f4b2c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99ae5dd7a66f119f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:40.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b33e00640 1 -- 172.21.15.102:0/2265918729 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c039070 con 0x7f4b340c4fa0 2024-03-18T16:36:40.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b33e00640 1 -- 172.21.15.102:0/2265918729 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b2c02cc50 con 0x7f4b340c4fa0 2024-03-18T16:36:40.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b33e00640 1 -- 172.21.15.102:0/2265918729 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c034440 con 0x7f4b340c4fa0 2024-03-18T16:36:40.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b3a800640 1 -- 172.21.15.102:0/2265918729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b340c4fa0 msgr2=0x7f4b340c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:40.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b3a800640 1 --2- 172.21.15.102:0/2265918729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b340c4fa0 0x7f4b340c5430 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009d90 tx=0x7f4b2c02bea0 comp rx=0 tx=0).stop 2024-03-18T16:36:40.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b3a800640 1 -- 172.21.15.102:0/2265918729 shutdown_connections 2024-03-18T16:36:40.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b3a800640 1 --2- 172.21.15.102:0/2265918729 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b340c4fa0 0x7f4b340c5430 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:40.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b3a800640 1 --2- 172.21.15.102:0/2265918729 >> v2:172.21.15.154:3300/0 conn(0x7f4b34141650 0x7f4b34158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:40.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b3a800640 1 -- 172.21.15.102:0/2265918729 >> 172.21.15.102:0/2265918729 conn(0x7f4b340cb890 msgr2=0x7f4b340cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:40.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.920+0000 7f4b3a800640 1 -- 172.21.15.102:0/2265918729 shutdown_connections 2024-03-18T16:36:40.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.921+0000 7f4b3a800640 1 -- 172.21.15.102:0/2265918729 wait complete. 2024-03-18T16:36:40.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.923+0000 7f4b3a800640 1 Processor -- start 2024-03-18T16:36:40.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.923+0000 7f4b3a800640 1 -- start start 2024-03-18T16:36:40.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.925+0000 7f4b3a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b340c4fa0 0x7f4b34159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:40.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.925+0000 7f4b3a800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b34141650 0x7f4b340caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.925+0000 7f4b3a800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b3415c3a0 0x7f4b34155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.925+0000 7f4b3a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4b340c66a0 con 0x7f4b340c4fa0 2024-03-18T16:36:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.925+0000 7f4b3a800640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b340c63c0 con 0x7f4b34141650 2024-03-18T16:36:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.925+0000 7f4b3a800640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4b3415c7c0 con 0x7f4b3415c3a0 2024-03-18T16:36:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.925+0000 7f4b38a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b34141650 0x7f4b340caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.926+0000 7f4b39400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b340c4fa0 0x7f4b34159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:40.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.926+0000 7f4b38a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b34141650 0x7f4b340caf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56572/0 (socket says 172.21.15.102:56572) 2024-03-18T16:36:40.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.926+0000 7f4b38a00640 1 -- 172.21.15.102:0/3828092622 learned_addr learned my addr 172.21.15.102:0/3828092622 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:40.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.926+0000 7f4b39e00640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b3415c3a0 0x7f4b34155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:40.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.930+0000 7f4b39e00640 1 -- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b34141650 msgr2=0x7f4b340caf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:40.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.930+0000 7f4b39e00640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b34141650 0x7f4b340caf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:40.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.930+0000 7f4b39e00640 1 -- 172.21.15.102:0/3828092622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b340c4fa0 msgr2=0x7f4b34159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:40.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.930+0000 7f4b39e00640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b340c4fa0 0x7f4b34159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:40.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.930+0000 7f4b39e00640 1 -- 172.21.15.102:0/3828092622 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b341f3710 con 0x7f4b3415c3a0 2024-03-18T16:36:40.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.932+0000 7f4b39e00640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b3415c3a0 0x7f4b34155360 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c033000 tx=0x7f4b2c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:40.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.934+0000 7f4b31600640 1 -- 172.21.15.102:0/3828092622 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c039050 con 0x7f4b3415c3a0 2024-03-18T16:36:40.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.934+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b341f39f0 con 0x7f4b3415c3a0 2024-03-18T16:36:40.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.934+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b341f3fb0 con 0x7f4b3415c3a0 2024-03-18T16:36:40.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.935+0000 7f4b31600640 1 -- 172.21.15.102:0/3828092622 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b2c045070 con 0x7f4b3415c3a0 2024-03-18T16:36:40.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.935+0000 7f4b31600640 1 -- 172.21.15.102:0/3828092622 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c009450 con 0x7f4b3415c3a0 2024-03-18T16:36:40.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.936+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b34074fd0 con 0x7f4b3415c3a0 2024-03-18T16:36:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.957+0000 7f4b31600640 1 -- 172.21.15.102:0/3828092622 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4b2c03e610 con 0x7f4b3415c3a0 2024-03-18T16:36:40.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.959+0000 7f4b31600640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4b1406bbd0 0x7f4b1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:40.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.959+0000 7f4b39400640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4b1406bbd0 0x7f4b1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:40.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.960+0000 7f4b39400640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4b1406bbd0 0x7f4b1406e090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4b34158360 tx=0x7f4b28009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:40.960+0000 7f4b31600640 1 -- 172.21.15.102:0/3828092622 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f4b2c07a310 con 0x7f4b3415c3a0 2024-03-18T16:36:41.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.211+0000 7f4b31600640 1 -- 172.21.15.102:0/3828092622 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4b2c07c270 con 0x7f4b3415c3a0 2024-03-18T16:36:41.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.520+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b340b7f00 con 0x7f4b3415c3a0 2024-03-18T16:36:41.574 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:07.920 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:36:41.578 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:07.923 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:36:41.596 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:16:07.856 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:36:41.608 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:16:07.867 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:36:41.655 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:16:08.000 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:36:41.660 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:08.006 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:36:41.672 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:16:08.018 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:36:41.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.671+0000 7f4b31600640 1 -- 172.21.15.102:0/3828092622 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6240 (secure 0 0 0) 0x7f4b2c080310 con 0x7f4b3415c3a0 2024-03-18T16:36:41.673 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:41.673 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:36:32.278787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":41,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4247,4253,4256,4270]},"id":1}]} 2024-03-18T16:36:41.673 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 26 2024-03-18T16:36:41.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4b1406bbd0 msgr2=0x7f4b1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:41.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4b1406bbd0 0x7f4b1406e090 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4b34158360 tx=0x7f4b28009190 comp rx=0 tx=0).stop 2024-03-18T16:36:41.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b3415c3a0 msgr2=0x7f4b34155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:41.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b3415c3a0 0x7f4b34155360 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c033000 tx=0x7f4b2c004a40 comp rx=0 tx=0).stop 2024-03-18T16:36:41.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 shutdown_connections 2024-03-18T16:36:41.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4b1406bbd0 0x7f4b1406e090 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:41.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4b3415c3a0 0x7f4b34155360 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b34141650 0x7f4b340caf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 --2- 172.21.15.102:0/3828092622 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b340c4fa0 0x7f4b34159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.674+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 >> 172.21.15.102:0/3828092622 conn(0x7f4b340cb890 msgr2=0x7f4b340cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.675+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 shutdown_connections 2024-03-18T16:36:41.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:41.675+0000 7f4b3a800640 1 -- 172.21.15.102:0/3828092622 wait complete. 2024-03-18T16:36:41.710 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:16:07.970 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:36:41.803 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:16:08.149 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:36:42.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:36:32.278787+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 41, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4247, 4253, 4256, 4270]} 2024-03-18T16:36:42.305 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:43.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:43.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.483+0000 7f4160a00640 1 Processor -- start 2024-03-18T16:36:43.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.483+0000 7f4160a00640 1 -- start start 2024-03-18T16:36:43.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.485+0000 7f4160a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f415c0c2d90 0x7f415c162aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:43.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.486+0000 7f4160a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f415c073530 con 0x7f415c157d90 2024-03-18T16:36:43.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.486+0000 7f4160a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f415c0c57e0 con 0x7f415c0c7f30 2024-03-18T16:36:43.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.486+0000 7f4160a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f415c0c5d80 con 0x7f415c0c2d90 2024-03-18T16:36:43.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.486+0000 7f415a800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f415c0c7f30 0x7f415c15b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58646/0 (socket says 172.21.15.102:58646) 2024-03-18T16:36:43.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.486+0000 7f415a800640 1 -- 172.21.15.102:0/2592812673 learned_addr learned my addr 172.21.15.102:0/2592812673 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:43.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.486+0000 7f415b200640 1 --2- 172.21.15.102:0/2592812673 >> v2:172.21.15.102:3300/0 conn(0x7f415c0c2d90 0x7f415c162aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:43.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.489+0000 7f4159400640 1 -- 172.21.15.102:0/2592812673 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4202297348 0 0) 0x7f415c0c57e0 con 0x7f415c0c7f30 2024-03-18T16:36:43.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.489+0000 7f4159400640 1 -- 172.21.15.102:0/2592812673 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4140005730 con 0x7f415c0c7f30 2024-03-18T16:36:43.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.491+0000 7f415b200640 1 -- 172.21.15.102:0/2592812673 >> v1:172.21.15.154:6789/0 conn(0x7f415c0c7f30 legacy=0x7f415c15b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:43.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.491+0000 7f415b200640 1 -- 172.21.15.102:0/2592812673 >> v1:172.21.15.102:6789/0 conn(0x7f415c157d90 legacy=0x7f415c072810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:43.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.491+0000 7f415b200640 1 -- 172.21.15.102:0/2592812673 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c0c5aa0 con 0x7f415c0c2d90 2024-03-18T16:36:43.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.497+0000 7f415b200640 1 --2- 172.21.15.102:0/2592812673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f415c0c2d90 0x7f415c162aa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4154009d90 tx=0x7f415402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877a74fdd345e049 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:43.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.502+0000 7f4159400640 1 -- 172.21.15.102:0/2592812673 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4154039070 con 0x7f415c0c2d90 2024-03-18T16:36:43.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.502+0000 7f4159400640 1 -- 172.21.15.102:0/2592812673 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f415402cd80 con 0x7f415c0c2d90 2024-03-18T16:36:43.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.502+0000 7f4159400640 1 -- 172.21.15.102:0/2592812673 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4154034660 con 0x7f415c0c2d90 2024-03-18T16:36:43.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.503+0000 7f4160a00640 1 -- 172.21.15.102:0/2592812673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f415c0c2d90 msgr2=0x7f415c162aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:43.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.503+0000 7f4160a00640 1 --2- 172.21.15.102:0/2592812673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f415c0c2d90 0x7f415c162aa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4154009d90 tx=0x7f415402bfd0 comp rx=0 tx=0).stop 2024-03-18T16:36:43.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.503+0000 7f4160a00640 1 -- 172.21.15.102:0/2592812673 shutdown_connections 2024-03-18T16:36:43.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.503+0000 7f4160a00640 1 --2- 172.21.15.102:0/2592812673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f415c0c2d90 0x7f415c162aa0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:43.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.503+0000 7f4160a00640 1 -- 172.21.15.102:0/2592812673 >> 172.21.15.102:0/2592812673 conn(0x7f415c0cb890 msgr2=0x7f415c0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:43.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.503+0000 7f4160a00640 1 -- 172.21.15.102:0/2592812673 shutdown_connections 2024-03-18T16:36:43.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.504+0000 7f4160a00640 1 -- 172.21.15.102:0/2592812673 wait complete. 2024-03-18T16:36:43.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.506+0000 7f4160a00640 1 Processor -- start 2024-03-18T16:36:43.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.506+0000 7f4160a00640 1 -- start start 2024-03-18T16:36:43.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.508+0000 7f4160a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f415c0c2d90 0x7f415c0c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:43.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.508+0000 7f4160a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f415c0c7f30 0x7f415c153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:43.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.508+0000 7f4160a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f415c157d90 0x7f415c14e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:43.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.508+0000 7f4160a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f415c157590 con 0x7f415c157d90 2024-03-18T16:36:43.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.508+0000 7f4160a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f415c157900 con 0x7f415c0c2d90 2024-03-18T16:36:43.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.508+0000 7f4160a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f415c0c31a0 con 0x7f415c0c7f30 2024-03-18T16:36:43.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.508+0000 7f415a800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f415c0c2d90 0x7f415c0c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:43.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.509+0000 7f415b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f415c157d90 0x7f415c14e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:43.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.509+0000 7f4159e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f415c0c7f30 0x7f415c153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:43.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.509+0000 7f415a800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f415c0c2d90 0x7f415c0c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56576/0 (socket says 172.21.15.102:56576) 2024-03-18T16:36:43.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.509+0000 7f415a800640 1 -- 172.21.15.102:0/2471987851 learned_addr learned my addr 172.21.15.102:0/2471987851 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:43.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.512+0000 7f415a800640 1 -- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f415c0c7f30 msgr2=0x7f415c153960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:43.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.512+0000 7f415a800640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f415c0c7f30 0x7f415c153960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:43.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.512+0000 7f415a800640 1 -- 172.21.15.102:0/2471987851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f415c157d90 msgr2=0x7f415c14e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:43.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.512+0000 7f415a800640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f415c157d90 0x7f415c14e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:43.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.512+0000 7f415a800640 1 -- 172.21.15.102:0/2471987851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c0c9240 con 0x7f415c0c2d90 2024-03-18T16:36:43.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.516+0000 7f415a800640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f415c0c2d90 0x7f415c0c85e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f414c00c560 tx=0x7f414c00ca20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:43.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.520+0000 7f4152a00640 1 -- 172.21.15.102:0/2471987851 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c017070 con 0x7f415c0c2d90 2024-03-18T16:36:43.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.520+0000 7f4152a00640 1 -- 172.21.15.102:0/2471987851 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f414c005ce0 con 0x7f415c0c2d90 2024-03-18T16:36:43.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.520+0000 7f4152a00640 1 -- 172.21.15.102:0/2471987851 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c0072b0 con 0x7f415c0c2d90 2024-03-18T16:36:43.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.520+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f415c072ab0 con 0x7f415c0c2d90 2024-03-18T16:36:43.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.520+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f415c1ec140 con 0x7f415c0c2d90 2024-03-18T16:36:43.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.522+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f415c074fd0 con 0x7f415c0c2d90 2024-03-18T16:36:43.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.548+0000 7f4152a00640 1 -- 172.21.15.102:0/2471987851 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f414c01a070 con 0x7f415c0c2d90 2024-03-18T16:36:43.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.549+0000 7f4152a00640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f413c06bbd0 0x7f413c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:43.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.549+0000 7f4159e00640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f413c06bbd0 0x7f413c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:43.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.550+0000 7f4159e00640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f413c06bbd0 0x7f413c06e090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f415c155bf0 tx=0x7f4148009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.800+0000 7f4152a00640 1 -- 172.21.15.102:0/2471987851 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f414c056820 con 0x7f415c0c2d90 2024-03-18T16:36:43.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:43.811+0000 7f4152a00640 1 -- 172.21.15.102:0/2471987851 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f414c021020 con 0x7f415c0c2d90 2024-03-18T16:36:44.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.064+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f415c0c3480 con 0x7f415c0c2d90 2024-03-18T16:36:44.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.076+0000 7f4152a00640 1 -- 172.21.15.102:0/2471987851 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6240 (secure 0 0 0) 0x7f414c058780 con 0x7f415c0c2d90 2024-03-18T16:36:44.078 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:44.078 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:36:32.278787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":41,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4247,4253,4256,4270]},"id":1}]} 2024-03-18T16:36:44.078 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 26 2024-03-18T16:36:44.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.078+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f413c06bbd0 msgr2=0x7f413c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:44.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.078+0000 7f4160a00640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f413c06bbd0 0x7f413c06e090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f415c155bf0 tx=0x7f4148009040 comp rx=0 tx=0).stop 2024-03-18T16:36:44.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.078+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f415c0c2d90 msgr2=0x7f415c0c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:44.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.078+0000 7f4160a00640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f415c0c2d90 0x7f415c0c85e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f414c00c560 tx=0x7f414c00ca20 comp rx=0 tx=0).stop 2024-03-18T16:36:44.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.079+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 shutdown_connections 2024-03-18T16:36:44.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.079+0000 7f4160a00640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f413c06bbd0 0x7f413c06e090 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:44.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.079+0000 7f4160a00640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f415c157d90 0x7f415c14e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:44.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.079+0000 7f4160a00640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f415c0c7f30 0x7f415c153960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:44.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.079+0000 7f4160a00640 1 --2- 172.21.15.102:0/2471987851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f415c0c2d90 0x7f415c0c85e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:44.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.079+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 >> 172.21.15.102:0/2471987851 conn(0x7f415c0cb890 msgr2=0x7f415c155c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:44.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.079+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 shutdown_connections 2024-03-18T16:36:44.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:44.079+0000 7f4160a00640 1 -- 172.21.15.102:0/2471987851 wait complete. 2024-03-18T16:36:44.700 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:36:32.278787+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 41, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4247, 4253, 4256, 4270]} 2024-03-18T16:36:44.700 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:45.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:36:45.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.878+0000 7f7122600640 1 Processor -- start 2024-03-18T16:36:45.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.878+0000 7f7122600640 1 -- start start 2024-03-18T16:36:45.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.879+0000 7f7122600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06b470 0x7f711c15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:45.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.879+0000 7f7122600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f711c06de50 0x7f711c06b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:45.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.879+0000 7f7122600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f711c0c94b0 con 0x7f711c06bd90 2024-03-18T16:36:45.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.879+0000 7f7122600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f711c0ca0a0 con 0x7f711c06de50 2024-03-18T16:36:45.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.879+0000 7f7122600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f711c0c9770 con 0x7f711c06b470 2024-03-18T16:36:45.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.880+0000 7f7121200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06b470 0x7f711c15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:45.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.880+0000 7f7121200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06b470 0x7f711c15ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:33486/0 (socket says 172.21.15.102:33486) 2024-03-18T16:36:45.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.880+0000 7f7121200640 1 -- 172.21.15.102:0/1305227354 learned_addr learned my addr 172.21.15.102:0/1305227354 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:45.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.882+0000 7f7121200640 1 -- 172.21.15.102:0/1305227354 >> v2:172.21.15.154:3300/0 conn(0x7f711c06de50 msgr2=0x7f711c06b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:45.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.882+0000 7f7121200640 1 --2- 172.21.15.102:0/1305227354 >> v2:172.21.15.154:3300/0 conn(0x7f711c06de50 0x7f711c06b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:45.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.882+0000 7f7121200640 1 -- 172.21.15.102:0/1305227354 >> v1:172.21.15.102:6789/0 conn(0x7f711c06bd90 legacy=0x7f711c154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:45.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.882+0000 7f7121200640 1 -- 172.21.15.102:0/1305227354 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711c0c9a50 con 0x7f711c06b470 2024-03-18T16:36:45.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.885+0000 7f7121200640 1 --2- 172.21.15.102:0/1305227354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06b470 0x7f711c15ad60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7110009870 tx=0x7f711002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fcfe3981391430e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.890+0000 7f711b400640 1 -- 172.21.15.102:0/1305227354 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110035020 con 0x7f711c06b470 2024-03-18T16:36:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.890+0000 7f711b400640 1 -- 172.21.15.102:0/1305227354 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7110031640 con 0x7f711c06b470 2024-03-18T16:36:45.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.890+0000 7f711b400640 1 -- 172.21.15.102:0/1305227354 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711002d040 con 0x7f711c06b470 2024-03-18T16:36:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.890+0000 7f7122600640 1 -- 172.21.15.102:0/1305227354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06b470 msgr2=0x7f711c15ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.890+0000 7f7122600640 1 --2- 172.21.15.102:0/1305227354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06b470 0x7f711c15ad60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f7110009870 tx=0x7f711002b8f0 comp rx=0 tx=0).stop 2024-03-18T16:36:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.891+0000 7f7122600640 1 -- 172.21.15.102:0/1305227354 shutdown_connections 2024-03-18T16:36:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.891+0000 7f7122600640 1 --2- 172.21.15.102:0/1305227354 >> v2:172.21.15.154:3300/0 conn(0x7f711c06de50 0x7f711c06b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.891+0000 7f7122600640 1 --2- 172.21.15.102:0/1305227354 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06b470 0x7f711c15ad60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.891+0000 7f7122600640 1 -- 172.21.15.102:0/1305227354 >> 172.21.15.102:0/1305227354 conn(0x7f711c0727e0 msgr2=0x7f711c072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.891+0000 7f7122600640 1 -- 172.21.15.102:0/1305227354 shutdown_connections 2024-03-18T16:36:45.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.891+0000 7f7122600640 1 -- 172.21.15.102:0/1305227354 wait complete. 2024-03-18T16:36:45.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.893+0000 7f7122600640 1 Processor -- start 2024-03-18T16:36:45.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.894+0000 7f7122600640 1 -- start start 2024-03-18T16:36:45.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.895+0000 7f7122600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f711c06b470 0x7f711c0c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:45.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.895+0000 7f7122600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06bd90 0x7f711c154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.895+0000 7f7122600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c06de50 0x7f711c1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.895+0000 7f7122600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f711c1549e0 con 0x7f711c06b470 2024-03-18T16:36:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.896+0000 7f7122600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f711c154860 con 0x7f711c06de50 2024-03-18T16:36:45.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.896+0000 7f7122600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f711c0c5190 con 0x7f711c06bd90 2024-03-18T16:36:45.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.925+0000 7f7121200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f711c06b470 0x7f711c0c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:45.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.926+0000 7f7121200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f711c06b470 0x7f711c0c7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59852/0 (socket says 172.21.15.102:59852) 2024-03-18T16:36:45.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.926+0000 7f7121200640 1 -- 172.21.15.102:0/3390692183 learned_addr learned my addr 172.21.15.102:0/3390692183 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.932+0000 7f7121200640 1 -- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06bd90 msgr2=0x7f711c154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.932+0000 7f7121200640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06bd90 0x7f711c154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.932+0000 7f7121200640 1 -- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c06de50 msgr2=0x7f711c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.932+0000 7f7121200640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c06de50 0x7f711c1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:45.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.932+0000 7f7121200640 1 -- 172.21.15.102:0/3390692183 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711c1efe20 con 0x7f711c06b470 2024-03-18T16:36:45.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.936+0000 7f7121200640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f711c06b470 0x7f711c0c7320 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f711002b930 tx=0x7f71100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.941+0000 7f7118c00640 1 -- 172.21.15.102:0/3390692183 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711003ac60 con 0x7f711c06b470 2024-03-18T16:36:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.941+0000 7f7118c00640 1 -- 172.21.15.102:0/3390692183 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711003ae00 con 0x7f711c06b470 2024-03-18T16:36:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.941+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711c1f0ca0 con 0x7f711c06b470 2024-03-18T16:36:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.941+0000 7f7118c00640 1 -- 172.21.15.102:0/3390692183 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110044630 con 0x7f711c06b470 2024-03-18T16:36:45.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.941+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f711c1f0370 con 0x7f711c06b470 2024-03-18T16:36:45.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.942+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f711c074fd0 con 0x7f711c06b470 2024-03-18T16:36:45.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.974+0000 7f7118c00640 1 -- 172.21.15.102:0/3390692183 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7110038020 con 0x7f711c06b470 2024-03-18T16:36:45.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.976+0000 7f7118c00640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70fc06bc20 0x7f70fc06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:45.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.976+0000 7f711be00640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70fc06bc20 0x7f70fc06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:45.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.977+0000 7f711be00640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70fc06bc20 0x7f70fc06e0e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f711c156120 tx=0x7f7108006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:45.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:45.979+0000 7f7118c00640 1 -- 172.21.15.102:0/3390692183 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7110079690 con 0x7f711c06b470 2024-03-18T16:36:46.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.277+0000 7f7118c00640 1 -- 172.21.15.102:0/3390692183 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f711003f070 con 0x7f711c06b470 2024-03-18T16:36:46.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.555+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f711c0caca0 con 0x7f711c06b470 2024-03-18T16:36:46.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.578+0000 7f7118c00640 1 -- 172.21.15.102:0/3390692183 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6240 (secure 0 0 0) 0x7f711007bcf0 con 0x7f711c06b470 2024-03-18T16:36:46.579 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:46.579 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:21:22.229008+0000","modified":"2024-03-18T16:36:32.278787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4270,"mds_1":4256,"mds_2":4253,"mds_3":4244,"mds_4":4247},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4244":{"gid":4244,"name":"d","rank":3,"incarnation":20,"state":"up:resolve","state_seq":33,"addr":"172.21.15.154:6839/2372940736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2372940736},{"type":"v1","addr":"172.21.15.154:6839","nonce":2372940736}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4247":{"gid":4247,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":18,"addr":"172.21.15.154:6837/120487002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":120487002},{"type":"v1","addr":"172.21.15.154:6837","nonce":120487002}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:21.512298+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4253":{"gid":4253,"name":"e","rank":2,"incarnation":14,"state":"up:active","state_seq":16,"addr":"172.21.15.102:6835/1730276867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6833","nonce":1730276867},{"type":"v1","addr":"172.21.15.102:6835","nonce":1730276867}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:59.788787+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4256":{"gid":4256,"name":"a","rank":1,"incarnation":10,"state":"up:active","state_seq":15,"addr":"172.21.15.102:6837/971565985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":971565985},{"type":"v1","addr":"172.21.15.102:6837","nonce":971565985}]},"join_fscid":-1,"laggy_since":"2024-03-18T16:23:29.663121+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4270":{"gid":4270,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":41,"addr":"172.21.15.154:6835/2841941336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":2841941336},{"type":"v1","addr":"172.21.15.154:6835","nonce":2841941336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4247,4253,4256,4270]},"id":1}]} 2024-03-18T16:36:46.580 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 26 2024-03-18T16:36:46.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70fc06bc20 msgr2=0x7f70fc06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70fc06bc20 0x7f70fc06e0e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f711c156120 tx=0x7f7108006bf0 comp rx=0 tx=0).stop 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f711c06b470 msgr2=0x7f711c0c7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f711c06b470 0x7f711c0c7320 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f711002b930 tx=0x7f71100046c0 comp rx=0 tx=0).stop 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 shutdown_connections 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f70fc06bc20 0x7f70fc06e0e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f711c06de50 0x7f711c1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f711c06bd90 0x7f711c154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 --2- 172.21.15.102:0/3390692183 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f711c06b470 0x7f711c0c7320 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 >> 172.21.15.102:0/3390692183 conn(0x7f711c0727e0 msgr2=0x7f711c072bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 shutdown_connections 2024-03-18T16:36:46.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:46.582+0000 7f7122600640 1 -- 172.21.15.102:0/3390692183 wait complete. 2024-03-18T16:36:47.211 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:21:22.229008+0000', 'modified': '2024-03-18T16:36:32.278787+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 30, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4270, 'mds_1': 4256, 'mds_2': 4253, 'mds_3': 4244, 'mds_4': 4247}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4244': {'gid': 4244, 'name': 'd', 'rank': 3, 'incarnation': 20, 'state': 'up:resolve', 'state_seq': 33, 'addr': '172.21.15.154:6839/2372940736', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2372940736}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2372940736}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4247': {'gid': 4247, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.154:6837/120487002', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 120487002}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 120487002}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:21.512298+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4253': {'gid': 4253, 'name': 'e', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.102:6835/1730276867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6833', 'nonce': 1730276867}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 1730276867}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:59.788787+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4256': {'gid': 4256, 'name': 'a', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.102:6837/971565985', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 971565985}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 971565985}]}, 'join_fscid': -1, 'laggy_since': '2024-03-18T16:23:29.663121+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4270': {'gid': 4270, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 41, 'addr': '172.21.15.154:6835/2841941336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 2841941336}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 2841941336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4247, 4253, 4256, 4270]} 2024-03-18T16:36:47.211 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4244:up:resolve 2024-03-18T16:36:48.212 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-18T16:21:22.229008+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4244": { "addr": "172.21.15.154:6839/2372940736", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6838", "nonce": 2372940736, "type": "v2" }, { "addr": "172.21.15.154:6839", "nonce": 2372940736, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4244, "incarnation": 20, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:resolve", "state_seq": 33 }, "gid_4247": { "addr": "172.21.15.154:6837/120487002", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6836", "nonce": 120487002, "type": "v2" }, { "addr": "172.21.15.154:6837", "nonce": 120487002, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4247, "incarnation": 18, "join_fscid": -1, "laggy_since": "2024-03-18T16:23:21.512298+0000", "name": "b", "rank": 4, "state": "up:active", "state_seq": 18 }, "gid_4253": { "addr": "172.21.15.102:6835/1730276867", "addrs": { "addrvec": [ { "addr": "172.21.15.102:6833", "nonce": 1730276867, "type": "v2" }, { "addr": "172.21.15.102:6835", "nonce": 1730276867, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4253, "incarnation": 14, "join_fscid": -1, "laggy_since": "2024-03-18T16:23:59.788787+0000", "name": "e", "rank": 2, "state": "up:active", "state_seq": 16 }, "gid_4256": { "addr": "172.21.15.102:6837/971565985", "addrs": { "addrvec": [ { "addr": "172.21.15.102:6836", "nonce": 971565985, "type": "v2" }, { "addr": "172.21.15.102:6837", "nonce": 971565985, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4256, "incarnation": 10, "join_fscid": -1, "laggy_since": "2024-03-18T16:23:29.663121+0000", "name": "a", "rank": 1, "state": "up:active", "state_seq": 15 }, "gid_4270": { "addr": "172.21.15.154:6835/2841941336", "addrs": { "addrvec": [ { "addr": "172.21.15.154:6834", "nonce": 2841941336, "type": "v2" }, { "addr": "172.21.15.154:6835", "nonce": 2841941336, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 4, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-18T16:36:32.278787+0000", "qdb_cluster": [ 4247, 4253, 4256, 4270 ], "qdb_leader": 4270, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4270, "mds_1": 4256, "mds_2": 4253, "mds_3": 4244, "mds_4": 4247 } } } ], "standbys": [] } 2024-03-18T16:36:48.213 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-18T16:36:48.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:36:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.394+0000 7f4eb2000640 1 Processor -- start 2024-03-18T16:36:48.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.394+0000 7f4eb2000640 1 -- start start 2024-03-18T16:36:48.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.395+0000 7f4eb2000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4eac158420 0x7f4eac15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:48.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.395+0000 7f4eb2000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4eac0c4fa0 0x7f4eac0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:48.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.395+0000 7f4eb2000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4eac0c72d0 con 0x7f4eac141650 2024-03-18T16:36:48.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.395+0000 7f4eb2000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f4eac15c8e0 con 0x7f4eac0c4fa0 2024-03-18T16:36:48.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.395+0000 7f4eb2000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4eac159370 con 0x7f4eac158420 2024-03-18T16:36:48.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.396+0000 7f4eb1600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4eac0c4fa0 0x7f4eac0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:48.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.396+0000 7f4eabe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4eac158420 0x7f4eac15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:48.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.396+0000 7f4eb0c00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f4eac141650 0x7f4eac0c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58656/0 (socket says 172.21.15.102:58656) 2024-03-18T16:36:48.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.396+0000 7f4eb0c00640 1 -- 172.21.15.102:0/995777008 learned_addr learned my addr 172.21.15.102:0/995777008 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:48.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.399+0000 7f4eabe00640 1 -- 172.21.15.102:0/995777008 >> v2:172.21.15.102:3300/0 conn(0x7f4eac0c4fa0 msgr2=0x7f4eac0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:48.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.399+0000 7f4eabe00640 1 --2- 172.21.15.102:0/995777008 >> v2:172.21.15.102:3300/0 conn(0x7f4eac0c4fa0 0x7f4eac0c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.399+0000 7f4eabe00640 1 -- 172.21.15.102:0/995777008 >> v1:172.21.15.154:6789/0 conn(0x7f4eac141650 legacy=0x7f4eac0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:48.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.399+0000 7f4eabe00640 1 -- 172.21.15.102:0/995777008 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eac158840 con 0x7f4eac158420 2024-03-18T16:36:48.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.403+0000 7f4eabe00640 1 --2- 172.21.15.102:0/995777008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac158420 0x7f4eac15c3a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4eac068e80 tx=0x7f4e9802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee5129beebd694cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:48.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.407+0000 7f4eab400640 1 -- 172.21.15.102:0/995777008 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e98035020 con 0x7f4eac158420 2024-03-18T16:36:48.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.407+0000 7f4eab400640 1 -- 172.21.15.102:0/995777008 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e98031830 con 0x7f4eac158420 2024-03-18T16:36:48.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.407+0000 7f4eab400640 1 -- 172.21.15.102:0/995777008 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e98031c00 con 0x7f4eac158420 2024-03-18T16:36:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.408+0000 7f4eb2000640 1 -- 172.21.15.102:0/995777008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac158420 msgr2=0x7f4eac15c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.408+0000 7f4eb2000640 1 --2- 172.21.15.102:0/995777008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac158420 0x7f4eac15c3a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4eac068e80 tx=0x7f4e9802b8d0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.408+0000 7f4eb2000640 1 -- 172.21.15.102:0/995777008 shutdown_connections 2024-03-18T16:36:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.408+0000 7f4eb2000640 1 --2- 172.21.15.102:0/995777008 >> v2:172.21.15.102:3300/0 conn(0x7f4eac0c4fa0 0x7f4eac0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.408+0000 7f4eb2000640 1 --2- 172.21.15.102:0/995777008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac158420 0x7f4eac15c3a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.408+0000 7f4eb2000640 1 -- 172.21.15.102:0/995777008 >> 172.21.15.102:0/995777008 conn(0x7f4eac0cb890 msgr2=0x7f4eac0cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.408+0000 7f4eb2000640 1 -- 172.21.15.102:0/995777008 shutdown_connections 2024-03-18T16:36:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.409+0000 7f4eb2000640 1 -- 172.21.15.102:0/995777008 wait complete. 2024-03-18T16:36:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.410+0000 7f4eb2000640 1 Processor -- start 2024-03-18T16:36:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.410+0000 7f4eb2000640 1 -- start start 2024-03-18T16:36:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.411+0000 7f4eb2000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac0c4fa0 0x7f4eac135500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.411+0000 7f4eb2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4eac141650 0x7f4eac13a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.411+0000 7f4eb2000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4eac158420 0x7f4eac13ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.411+0000 7f4eb2000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac159090 con 0x7f4eac141650 2024-03-18T16:36:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.411+0000 7f4eb2000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4eac158db0 con 0x7f4eac0c4fa0 2024-03-18T16:36:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.411+0000 7f4eb2000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f4eac0c63c0 con 0x7f4eac158420 2024-03-18T16:36:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.412+0000 7f4eb0c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac0c4fa0 0x7f4eac135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.412+0000 7f4eb1600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4eac158420 0x7f4eac13ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.412+0000 7f4eabe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4eac141650 0x7f4eac13a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.412+0000 7f4eb0c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac0c4fa0 0x7f4eac135500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56610/0 (socket says 172.21.15.102:56610) 2024-03-18T16:36:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.412+0000 7f4eb0c00640 1 -- 172.21.15.102:0/2975922000 learned_addr learned my addr 172.21.15.102:0/2975922000 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.416+0000 7f4eabe00640 1 -- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4eac158420 msgr2=0x7f4eac13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.416+0000 7f4eabe00640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4eac158420 0x7f4eac13ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.416+0000 7f4eabe00640 1 -- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac0c4fa0 msgr2=0x7f4eac135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.416+0000 7f4eabe00640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac0c4fa0 0x7f4eac135500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.416+0000 7f4eabe00640 1 -- 172.21.15.102:0/2975922000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eac155b80 con 0x7f4eac141650 2024-03-18T16:36:48.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.419+0000 7f4eabe00640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4eac141650 0x7f4eac13a6e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4e98033000 tx=0x7f4e980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:48.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.424+0000 7f4ea8c00640 1 -- 172.21.15.102:0/2975922000 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e98040070 con 0x7f4eac141650 2024-03-18T16:36:48.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.424+0000 7f4ea8c00640 1 -- 172.21.15.102:0/2975922000 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e9803a930 con 0x7f4eac141650 2024-03-18T16:36:48.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.424+0000 7f4ea8c00640 1 -- 172.21.15.102:0/2975922000 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e98035040 con 0x7f4eac141650 2024-03-18T16:36:48.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.424+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eac1f3540 con 0x7f4eac141650 2024-03-18T16:36:48.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.425+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eac1f3b00 con 0x7f4eac141650 2024-03-18T16:36:48.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.426+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eac074fd0 con 0x7f4eac141650 2024-03-18T16:36:48.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.706+0000 7f4ea8c00640 1 -- 172.21.15.102:0/2975922000 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4e9804b060 con 0x7f4eac141650 2024-03-18T16:36:48.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.708+0000 7f4ea8c00640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e8c06bc20 0x7f4e8c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:48.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.708+0000 7f4eb0c00640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e8c06bc20 0x7f4e8c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:48.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.709+0000 7f4eb0c00640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e8c06bc20 0x7f4e8c06e0e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0004e40 tx=0x7f4ea0007f10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:48.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.728+0000 7f4ea8c00640 1 -- 172.21.15.102:0/2975922000 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f4e98078b50 con 0x7f4eac141650 2024-03-18T16:36:48.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.729+0000 7f4ea8c00640 1 -- 172.21.15.102:0/2975922000 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4e98031800 con 0x7f4eac141650 2024-03-18T16:36:48.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.918+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4eac0b7f00 con 0x7f4e8c06bc20 2024-03-18T16:36:48.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.927+0000 7f4ea8c00640 1 -- 172.21.15.102:0/2975922000 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332703 (secure 0 0 0) 0x7f4eac0b7f00 con 0x7f4e8c06bc20 2024-03-18T16:36:48.933 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:48.935 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e8c06bc20 msgr2=0x7f4e8c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e8c06bc20 0x7f4e8c06e0e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0004e40 tx=0x7f4ea0007f10 comp rx=0 tx=0).stop 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4eac141650 msgr2=0x7f4eac13a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4eac141650 0x7f4eac13a6e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4e98033000 tx=0x7f4e980046c0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 shutdown_connections 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f4e8c06bc20 0x7f4e8c06e0e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f4eac158420 0x7f4eac13ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4eac141650 0x7f4eac13a6e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 --2- 172.21.15.102:0/2975922000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4eac0c4fa0 0x7f4eac135500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 >> 172.21.15.102:0/2975922000 conn(0x7f4eac0cb890 msgr2=0x7f4eac13f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 shutdown_connections 2024-03-18T16:36:48.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:48.932+0000 7f4eb2000640 1 -- 172.21.15.102:0/2975922000 wait complete. 2024-03-18T16:36:48.950 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-03-18T16:36:47.033798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219040,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755680,"statfs":{"total":773094113280,"available":772869816320,"internally_reserved":0,"allocated":2949120,"data_stored":2294136,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002939"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204119+0000","last_change":"2024-03-18T16:21:21.402212+0000","last_active":"2024-03-18T16:33:52.204119+0000","last_peered":"2024-03-18T16:33:52.204119+0000","last_clean":"2024-03-18T16:33:52.204119+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:33:52.204119+0000","last_undegraded":"2024-03-18T16:33:52.204119+0000","last_fullsized":"2024-03-18T16:33:52.204119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:33.215915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.163586+0000","last_change":"2024-03-18T16:21:23.045605+0000","last_active":"2024-03-18T16:33:22.163586+0000","last_peered":"2024-03-18T16:33:22.163586+0000","last_clean":"2024-03-18T16:33:22.163586+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:33:22.163586+0000","last_undegraded":"2024-03-18T16:33:22.163586+0000","last_fullsized":"2024-03-18T16:33:22.163586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:37.165226+0000","last_change":"2024-03-18T16:21:29.146958+0000","last_active":"2024-03-18T16:33:37.165226+0000","last_peered":"2024-03-18T16:33:37.165226+0000","last_clean":"2024-03-18T16:33:37.165226+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:33:37.165226+0000","last_undegraded":"2024-03-18T16:33:37.165226+0000","last_fullsized":"2024-03-18T16:33:37.165226+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:55.871927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.163616+0000","last_change":"2024-03-18T16:21:21.413060+0000","last_active":"2024-03-18T16:33:22.163616+0000","last_peered":"2024-03-18T16:33:22.163616+0000","last_clean":"2024-03-18T16:33:22.163616+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:33:22.163616+0000","last_undegraded":"2024-03-18T16:33:22.163616+0000","last_fullsized":"2024-03-18T16:33:22.163616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:40.114937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204105+0000","last_change":"2024-03-18T16:21:23.050849+0000","last_active":"2024-03-18T16:33:52.204105+0000","last_peered":"2024-03-18T16:33:52.204105+0000","last_clean":"2024-03-18T16:33:52.204105+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:33:52.204105+0000","last_undegraded":"2024-03-18T16:33:52.204105+0000","last_fullsized":"2024-03-18T16:33:52.204105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:32.961742+0000","last_change":"2024-03-18T16:21:29.065755+0000","last_active":"2024-03-18T16:33:32.961742+0000","last_peered":"2024-03-18T16:33:32.961742+0000","last_clean":"2024-03-18T16:33:32.961742+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:33:32.961742+0000","last_undegraded":"2024-03-18T16:33:32.961742+0000","last_fullsized":"2024-03-18T16:33:32.961742+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:45.934586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'12","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092522+0000","last_change":"2024-03-18T16:21:21.398188+0000","last_active":"2024-03-18T16:33:52.092522+0000","last_peered":"2024-03-18T16:33:52.092522+0000","last_clean":"2024-03-18T16:33:52.092522+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:33:52.092522+0000","last_undegraded":"2024-03-18T16:33:52.092522+0000","last_fullsized":"2024-03-18T16:33:52.092522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481833+0000","last_change":"2024-03-18T16:21:22.946288+0000","last_active":"2024-03-18T16:33:54.481833+0000","last_peered":"2024-03-18T16:33:54.481833+0000","last_clean":"2024-03-18T16:33:54.481833+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:33:54.481833+0000","last_undegraded":"2024-03-18T16:33:54.481833+0000","last_fullsized":"2024-03-18T16:33:54.481833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:08.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:57.964061+0000","last_change":"2024-03-18T16:21:29.065891+0000","last_active":"2024-03-18T16:33:57.964061+0000","last_peered":"2024-03-18T16:33:57.964061+0000","last_clean":"2024-03-18T16:33:57.964061+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:33:57.964061+0000","last_undegraded":"2024-03-18T16:33:57.964061+0000","last_fullsized":"2024-03-18T16:33:57.964061+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092509+0000","last_change":"2024-03-18T16:21:21.397071+0000","last_active":"2024-03-18T16:33:52.092509+0000","last_peered":"2024-03-18T16:33:52.092509+0000","last_clean":"2024-03-18T16:33:52.092509+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:33:52.092509+0000","last_undegraded":"2024-03-18T16:33:52.092509+0000","last_fullsized":"2024-03-18T16:33:52.092509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481845+0000","last_change":"2024-03-18T16:21:22.946468+0000","last_active":"2024-03-18T16:33:54.481845+0000","last_peered":"2024-03-18T16:33:54.481845+0000","last_clean":"2024-03-18T16:33:54.481845+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:33:54.481845+0000","last_undegraded":"2024-03-18T16:33:54.481845+0000","last_fullsized":"2024-03-18T16:33:54.481845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:57.964049+0000","last_change":"2024-03-18T16:21:29.073724+0000","last_active":"2024-03-18T16:33:57.964049+0000","last_peered":"2024-03-18T16:33:57.964049+0000","last_clean":"2024-03-18T16:33:57.964049+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:33:57.964049+0000","last_undegraded":"2024-03-18T16:33:57.964049+0000","last_fullsized":"2024-03-18T16:33:57.964049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536515+0000","last_change":"2024-03-18T16:21:23.041926+0000","last_active":"2024-03-18T16:34:24.536515+0000","last_peered":"2024-03-18T16:34:24.536515+0000","last_clean":"2024-03-18T16:34:24.536515+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:34:24.536515+0000","last_undegraded":"2024-03-18T16:34:24.536515+0000","last_fullsized":"2024-03-18T16:34:24.536515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:03.532253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085707+0000","last_change":"2024-03-18T16:21:21.394801+0000","last_active":"2024-03-18T16:32:37.085707+0000","last_peered":"2024-03-18T16:32:37.085707+0000","last_clean":"2024-03-18T16:32:37.085707+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:32:37.085707+0000","last_undegraded":"2024-03-18T16:32:37.085707+0000","last_fullsized":"2024-03-18T16:32:37.085707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:42.957341+0000","last_change":"2024-03-18T16:21:29.073757+0000","last_active":"2024-03-18T16:32:42.957341+0000","last_peered":"2024-03-18T16:32:42.957341+0000","last_clean":"2024-03-18T16:32:42.957341+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:32:42.957341+0000","last_undegraded":"2024-03-18T16:32:42.957341+0000","last_fullsized":"2024-03-18T16:32:42.957341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.157732+0000","last_change":"2024-03-18T16:21:23.060776+0000","last_active":"2024-03-18T16:32:37.157732+0000","last_peered":"2024-03-18T16:32:37.157732+0000","last_clean":"2024-03-18T16:32:37.157732+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:32:37.157732+0000","last_undegraded":"2024-03-18T16:32:37.157732+0000","last_fullsized":"2024-03-18T16:32:37.157732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"30'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960340+0000","last_change":"2024-03-18T16:21:21.412733+0000","last_active":"2024-03-18T16:34:22.960340+0000","last_peered":"2024-03-18T16:34:22.960340+0000","last_clean":"2024-03-18T16:34:22.960340+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:34:22.960340+0000","last_undegraded":"2024-03-18T16:34:22.960340+0000","last_fullsized":"2024-03-18T16:34:22.960340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:42.957307+0000","last_change":"2024-03-18T16:21:29.063980+0000","last_active":"2024-03-18T16:32:42.957307+0000","last_peered":"2024-03-18T16:32:42.957307+0000","last_clean":"2024-03-18T16:32:42.957307+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:32:42.957307+0000","last_undegraded":"2024-03-18T16:32:42.957307+0000","last_fullsized":"2024-03-18T16:32:42.957307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:39.469490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:45.657782+0000","last_change":"2024-03-18T16:21:54.838108+0000","last_active":"2024-03-18T16:32:45.657782+0000","last_peered":"2024-03-18T16:32:45.657782+0000","last_clean":"2024-03-18T16:32:45.657782+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:32:45.657782+0000","last_undegraded":"2024-03-18T16:32:45.657782+0000","last_fullsized":"2024-03-18T16:32:45.657782+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:40.182971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960309+0000","last_change":"2024-03-18T16:21:21.407132+0000","last_active":"2024-03-18T16:34:22.960309+0000","last_peered":"2024-03-18T16:34:22.960309+0000","last_clean":"2024-03-18T16:34:22.960309+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:34:22.960309+0000","last_undegraded":"2024-03-18T16:34:22.960309+0000","last_fullsized":"2024-03-18T16:34:22.960309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:10.660930+0000","last_change":"2024-03-18T16:21:55.269661+0000","last_active":"2024-03-18T16:33:10.660930+0000","last_peered":"2024-03-18T16:33:10.660930+0000","last_clean":"2024-03-18T16:33:10.660930+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:33:10.660930+0000","last_undegraded":"2024-03-18T16:33:10.660930+0000","last_fullsized":"2024-03-18T16:33:10.660930+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:38.290937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.966337+0000","last_change":"2024-03-18T16:21:22.840551+0000","last_active":"2024-03-18T16:34:22.966337+0000","last_peered":"2024-03-18T16:34:22.966337+0000","last_clean":"2024-03-18T16:34:22.966337+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:34:22.966337+0000","last_undegraded":"2024-03-18T16:34:22.966337+0000","last_fullsized":"2024-03-18T16:34:22.966337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:17.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.095352+0000","last_change":"2024-03-18T16:21:21.400349+0000","last_active":"2024-03-18T16:34:22.095352+0000","last_peered":"2024-03-18T16:34:22.095352+0000","last_clean":"2024-03-18T16:34:22.095352+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:34:22.095352+0000","last_undegraded":"2024-03-18T16:34:22.095352+0000","last_fullsized":"2024-03-18T16:34:22.095352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:42.957357+0000","last_change":"2024-03-18T16:21:29.064624+0000","last_active":"2024-03-18T16:32:42.957357+0000","last_peered":"2024-03-18T16:32:42.957357+0000","last_clean":"2024-03-18T16:32:42.957357+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:32:42.957357+0000","last_undegraded":"2024-03-18T16:32:42.957357+0000","last_fullsized":"2024-03-18T16:32:42.957357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.207867+0000","last_change":"2024-03-18T16:21:23.055982+0000","last_active":"2024-03-18T16:34:22.207867+0000","last_peered":"2024-03-18T16:34:22.207867+0000","last_clean":"2024-03-18T16:34:22.207867+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:34:22.207867+0000","last_undegraded":"2024-03-18T16:34:22.207867+0000","last_fullsized":"2024-03-18T16:34:22.207867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536571+0000","last_change":"2024-03-18T16:21:21.400843+0000","last_active":"2024-03-18T16:34:24.536571+0000","last_peered":"2024-03-18T16:34:24.536571+0000","last_clean":"2024-03-18T16:34:24.536571+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:34:24.536571+0000","last_undegraded":"2024-03-18T16:34:24.536571+0000","last_fullsized":"2024-03-18T16:34:24.536571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:57.964028+0000","last_change":"2024-03-18T16:21:29.140888+0000","last_active":"2024-03-18T16:33:57.964028+0000","last_peered":"2024-03-18T16:33:57.964028+0000","last_clean":"2024-03-18T16:33:57.964028+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:33:57.964028+0000","last_undegraded":"2024-03-18T16:33:57.964028+0000","last_fullsized":"2024-03-18T16:33:57.964028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.171886+0000","last_change":"2024-03-18T16:21:22.994455+0000","last_active":"2024-03-18T16:34:22.171886+0000","last_peered":"2024-03-18T16:34:22.171886+0000","last_clean":"2024-03-18T16:34:22.171886+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:34:22.171886+0000","last_undegraded":"2024-03-18T16:34:22.171886+0000","last_fullsized":"2024-03-18T16:34:22.171886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960381+0000","last_change":"2024-03-18T16:21:21.402920+0000","last_active":"2024-03-18T16:34:22.960381+0000","last_peered":"2024-03-18T16:34:22.960381+0000","last_clean":"2024-03-18T16:34:22.960381+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:34:22.960381+0000","last_undegraded":"2024-03-18T16:34:22.960381+0000","last_fullsized":"2024-03-18T16:34:22.960381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:32.164696+0000","last_change":"2024-03-18T16:21:29.145235+0000","last_active":"2024-03-18T16:33:32.164696+0000","last_peered":"2024-03-18T16:33:32.164696+0000","last_clean":"2024-03-18T16:33:32.164696+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:33:32.164696+0000","last_undegraded":"2024-03-18T16:33:32.164696+0000","last_fullsized":"2024-03-18T16:33:32.164696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:42.379951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.957241+0000","last_change":"2024-03-18T16:21:22.975172+0000","last_active":"2024-03-18T16:33:52.957241+0000","last_peered":"2024-03-18T16:33:52.957241+0000","last_clean":"2024-03-18T16:33:52.957241+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:33:52.957241+0000","last_undegraded":"2024-03-18T16:33:52.957241+0000","last_fullsized":"2024-03-18T16:33:52.957241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092531+0000","last_change":"2024-03-18T16:21:21.399688+0000","last_active":"2024-03-18T16:33:52.092531+0000","last_peered":"2024-03-18T16:33:52.092531+0000","last_clean":"2024-03-18T16:33:52.092531+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:33:52.092531+0000","last_undegraded":"2024-03-18T16:33:52.092531+0000","last_fullsized":"2024-03-18T16:33:52.092531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:11.933810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.205097+0000","last_change":"2024-03-18T16:21:29.141555+0000","last_active":"2024-03-18T16:34:02.205097+0000","last_peered":"2024-03-18T16:34:02.205097+0000","last_clean":"2024-03-18T16:34:02.205097+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:34:02.205097+0000","last_undegraded":"2024-03-18T16:34:02.205097+0000","last_fullsized":"2024-03-18T16:34:02.205097+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481851+0000","last_change":"2024-03-18T16:21:22.947980+0000","last_active":"2024-03-18T16:33:54.481851+0000","last_peered":"2024-03-18T16:33:54.481851+0000","last_clean":"2024-03-18T16:33:54.481851+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:33:54.481851+0000","last_undegraded":"2024-03-18T16:33:54.481851+0000","last_fullsized":"2024-03-18T16:33:54.481851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:31.739693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"30'2","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.957271+0000","last_change":"2024-03-18T16:21:21.400087+0000","last_active":"2024-03-18T16:33:52.957271+0000","last_peered":"2024-03-18T16:33:52.957271+0000","last_clean":"2024-03-18T16:33:52.957271+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:33:52.957271+0000","last_undegraded":"2024-03-18T16:33:52.957271+0000","last_fullsized":"2024-03-18T16:33:52.957271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.958278+0000","last_change":"2024-03-18T16:21:29.076629+0000","last_active":"2024-03-18T16:34:02.958278+0000","last_peered":"2024-03-18T16:34:02.958278+0000","last_clean":"2024-03-18T16:34:02.958278+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:34:02.958278+0000","last_undegraded":"2024-03-18T16:34:02.958278+0000","last_fullsized":"2024-03-18T16:34:02.958278+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.963620+0000","last_change":"2024-03-18T16:21:23.055810+0000","last_active":"2024-03-18T16:33:52.963620+0000","last_peered":"2024-03-18T16:33:52.963620+0000","last_clean":"2024-03-18T16:33:52.963620+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:33:52.963620+0000","last_undegraded":"2024-03-18T16:33:52.963620+0000","last_fullsized":"2024-03-18T16:33:52.963620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:40.451255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481805+0000","last_change":"2024-03-18T16:21:21.399134+0000","last_active":"2024-03-18T16:33:54.481805+0000","last_peered":"2024-03-18T16:33:54.481805+0000","last_clean":"2024-03-18T16:33:54.481805+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:33:54.481805+0000","last_undegraded":"2024-03-18T16:33:54.481805+0000","last_fullsized":"2024-03-18T16:33:54.481805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.958311+0000","last_change":"2024-03-18T16:21:29.067567+0000","last_active":"2024-03-18T16:34:02.958311+0000","last_peered":"2024-03-18T16:34:02.958311+0000","last_clean":"2024-03-18T16:34:02.958311+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:34:02.958311+0000","last_undegraded":"2024-03-18T16:34:02.958311+0000","last_fullsized":"2024-03-18T16:34:02.958311+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.953865+0000","last_change":"2024-03-18T16:21:22.977089+0000","last_active":"2024-03-18T16:33:22.953865+0000","last_peered":"2024-03-18T16:33:22.953865+0000","last_clean":"2024-03-18T16:33:22.953865+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:33:22.953865+0000","last_undegraded":"2024-03-18T16:33:22.953865+0000","last_fullsized":"2024-03-18T16:33:22.953865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:50.664854+0000","last_change":"2024-03-18T16:21:21.408450+0000","last_active":"2024-03-18T16:33:50.664854+0000","last_peered":"2024-03-18T16:33:50.664854+0000","last_clean":"2024-03-18T16:33:50.664854+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:33:50.664854+0000","last_undegraded":"2024-03-18T16:33:50.664854+0000","last_fullsized":"2024-03-18T16:33:50.664854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:32.961622+0000","last_change":"2024-03-18T16:21:29.076875+0000","last_active":"2024-03-18T16:34:32.961622+0000","last_peered":"2024-03-18T16:34:32.961622+0000","last_clean":"2024-03-18T16:34:32.961622+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:34:32.961622+0000","last_undegraded":"2024-03-18T16:34:32.961622+0000","last_fullsized":"2024-03-18T16:34:32.961622+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204015+0000","last_change":"2024-03-18T16:21:23.054497+0000","last_active":"2024-03-18T16:33:52.204015+0000","last_peered":"2024-03-18T16:33:52.204015+0000","last_clean":"2024-03-18T16:33:52.204015+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:33:52.204015+0000","last_undegraded":"2024-03-18T16:33:52.204015+0000","last_fullsized":"2024-03-18T16:33:52.204015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:37.455180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.089824+0000","last_change":"2024-03-18T16:21:21.404754+0000","last_active":"2024-03-18T16:33:22.089824+0000","last_peered":"2024-03-18T16:33:22.089824+0000","last_clean":"2024-03-18T16:33:22.089824+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:33:22.089824+0000","last_undegraded":"2024-03-18T16:33:22.089824+0000","last_fullsized":"2024-03-18T16:33:22.089824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:32.209201+0000","last_change":"2024-03-18T16:21:29.147316+0000","last_active":"2024-03-18T16:34:32.209201+0000","last_peered":"2024-03-18T16:34:32.209201+0000","last_clean":"2024-03-18T16:34:32.209201+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:34:32.209201+0000","last_undegraded":"2024-03-18T16:34:32.209201+0000","last_fullsized":"2024-03-18T16:34:32.209201+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.167476+0000","last_change":"2024-03-18T16:21:22.947635+0000","last_active":"2024-03-18T16:33:52.167476+0000","last_peered":"2024-03-18T16:33:52.167476+0000","last_clean":"2024-03-18T16:33:52.167476+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:33:52.167476+0000","last_undegraded":"2024-03-18T16:33:52.167476+0000","last_fullsized":"2024-03-18T16:33:52.167476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481823+0000","last_change":"2024-03-18T16:21:21.381306+0000","last_active":"2024-03-18T16:33:54.481823+0000","last_peered":"2024-03-18T16:33:54.481823+0000","last_clean":"2024-03-18T16:33:54.481823+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:33:54.481823+0000","last_undegraded":"2024-03-18T16:33:54.481823+0000","last_fullsized":"2024-03-18T16:33:54.481823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:07.088485+0000","last_change":"2024-03-18T16:21:08.662767+0000","last_active":"2024-03-18T16:33:07.088485+0000","last_peered":"2024-03-18T16:33:07.088485+0000","last_clean":"2024-03-18T16:33:07.088485+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:33:07.088485+0000","last_undegraded":"2024-03-18T16:33:07.088485+0000","last_fullsized":"2024-03-18T16:33:07.088485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_clean_scrub_stamp":"2024-03-18T16:21:05.847994+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:12:10.494707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:59.534564+0000","last_change":"2024-03-18T16:21:29.145697+0000","last_active":"2024-03-18T16:33:59.534564+0000","last_peered":"2024-03-18T16:33:59.534564+0000","last_clean":"2024-03-18T16:33:59.534564+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:33:59.534564+0000","last_undegraded":"2024-03-18T16:33:59.534564+0000","last_fullsized":"2024-03-18T16:33:59.534564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:46.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085732+0000","last_change":"2024-03-18T16:21:22.995291+0000","last_active":"2024-03-18T16:32:37.085732+0000","last_peered":"2024-03-18T16:32:37.085732+0000","last_clean":"2024-03-18T16:32:37.085732+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:32:37.085732+0000","last_undegraded":"2024-03-18T16:32:37.085732+0000","last_fullsized":"2024-03-18T16:32:37.085732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:32.918246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"30'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:35.656730+0000","last_change":"2024-03-18T16:21:21.382137+0000","last_active":"2024-03-18T16:32:35.656730+0000","last_peered":"2024-03-18T16:32:35.656730+0000","last_clean":"2024-03-18T16:32:35.656730+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:32:35.656730+0000","last_undegraded":"2024-03-18T16:32:35.656730+0000","last_fullsized":"2024-03-18T16:32:35.656730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:29.537048+0000","last_change":"2024-03-18T16:21:29.139432+0000","last_active":"2024-03-18T16:34:29.537048+0000","last_peered":"2024-03-18T16:34:29.537048+0000","last_clean":"2024-03-18T16:34:29.537048+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:34:29.537048+0000","last_undegraded":"2024-03-18T16:34:29.537048+0000","last_fullsized":"2024-03-18T16:34:29.537048+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:14.828125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:39.475154+0000","last_change":"2024-03-18T16:21:22.949578+0000","last_active":"2024-03-18T16:32:39.475154+0000","last_peered":"2024-03-18T16:32:39.475154+0000","last_clean":"2024-03-18T16:32:39.475154+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:32:39.475154+0000","last_undegraded":"2024-03-18T16:32:39.475154+0000","last_fullsized":"2024-03-18T16:32:39.475154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:36:20.997720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"30'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.948624+0000","last_change":"2024-03-18T16:21:21.400859+0000","last_active":"2024-03-18T16:32:37.948624+0000","last_peered":"2024-03-18T16:32:37.948624+0000","last_clean":"2024-03-18T16:32:37.948624+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:32:37.948624+0000","last_undegraded":"2024-03-18T16:32:37.948624+0000","last_fullsized":"2024-03-18T16:32:37.948624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:47.197062+0000","last_change":"2024-03-18T16:21:29.076813+0000","last_active":"2024-03-18T16:32:47.197062+0000","last_peered":"2024-03-18T16:32:47.197062+0000","last_clean":"2024-03-18T16:32:47.197062+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:32:47.197062+0000","last_undegraded":"2024-03-18T16:32:47.197062+0000","last_fullsized":"2024-03-18T16:32:47.197062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085756+0000","last_change":"2024-03-18T16:21:23.000009+0000","last_active":"2024-03-18T16:32:37.085756+0000","last_peered":"2024-03-18T16:32:37.085756+0000","last_clean":"2024-03-18T16:32:37.085756+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:32:37.085756+0000","last_undegraded":"2024-03-18T16:32:37.085756+0000","last_fullsized":"2024-03-18T16:32:37.085756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:39.383342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'6","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:35.656773+0000","last_change":"2024-03-18T16:21:21.409271+0000","last_active":"2024-03-18T16:32:35.656773+0000","last_peered":"2024-03-18T16:32:35.656773+0000","last_clean":"2024-03-18T16:32:35.656773+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:32:35.656773+0000","last_undegraded":"2024-03-18T16:32:35.656773+0000","last_fullsized":"2024-03-18T16:32:35.656773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:29.484964+0000","last_change":"2024-03-18T16:21:29.139595+0000","last_active":"2024-03-18T16:34:29.484964+0000","last_peered":"2024-03-18T16:34:29.484964+0000","last_clean":"2024-03-18T16:34:29.484964+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:34:29.484964+0000","last_undegraded":"2024-03-18T16:34:29.484964+0000","last_fullsized":"2024-03-18T16:34:29.484964+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:49.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085742+0000","last_change":"2024-03-18T16:21:23.000054+0000","last_active":"2024-03-18T16:32:37.085742+0000","last_peered":"2024-03-18T16:32:37.085742+0000","last_clean":"2024-03-18T16:32:37.085742+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:32:37.085742+0000","last_undegraded":"2024-03-18T16:32:37.085742+0000","last_fullsized":"2024-03-18T16:32:37.085742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:34:41.535042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"29'2","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.956808+0000","last_change":"2024-03-18T16:21:21.399315+0000","last_active":"2024-03-18T16:32:37.956808+0000","last_peered":"2024-03-18T16:32:37.956808+0000","last_clean":"2024-03-18T16:32:37.956808+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:32:37.956808+0000","last_undegraded":"2024-03-18T16:32:37.956808+0000","last_fullsized":"2024-03-18T16:32:37.956808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:29.543106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:27.966842+0000","last_change":"2024-03-18T16:21:29.065243+0000","last_active":"2024-03-18T16:34:27.966842+0000","last_peered":"2024-03-18T16:34:27.966842+0000","last_clean":"2024-03-18T16:34:27.966842+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:34:27.966842+0000","last_undegraded":"2024-03-18T16:34:27.966842+0000","last_fullsized":"2024-03-18T16:34:27.966842+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.087917+0000","last_change":"2024-03-18T16:21:22.996080+0000","last_active":"2024-03-18T16:33:02.087917+0000","last_peered":"2024-03-18T16:33:02.087917+0000","last_clean":"2024-03-18T16:33:02.087917+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:33:02.087917+0000","last_undegraded":"2024-03-18T16:33:02.087917+0000","last_fullsized":"2024-03-18T16:33:02.087917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'18","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:00.659847+0000","last_change":"2024-03-18T16:21:21.404930+0000","last_active":"2024-03-18T16:33:00.659847+0000","last_peered":"2024-03-18T16:33:00.659847+0000","last_clean":"2024-03-18T16:33:00.659847+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:33:00.659847+0000","last_undegraded":"2024-03-18T16:33:00.659847+0000","last_fullsized":"2024-03-18T16:33:00.659847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:02.571905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:37.202573+0000","last_change":"2024-03-18T16:21:29.079031+0000","last_active":"2024-03-18T16:33:37.202573+0000","last_peered":"2024-03-18T16:33:37.202573+0000","last_clean":"2024-03-18T16:33:37.202573+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:33:37.202573+0000","last_undegraded":"2024-03-18T16:33:37.202573+0000","last_fullsized":"2024-03-18T16:33:37.202573+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529622+0000","last_change":"2024-03-18T16:21:23.056832+0000","last_active":"2024-03-18T16:33:04.529622+0000","last_peered":"2024-03-18T16:33:04.529622+0000","last_clean":"2024-03-18T16:33:04.529622+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:33:04.529622+0000","last_undegraded":"2024-03-18T16:33:04.529622+0000","last_fullsized":"2024-03-18T16:33:04.529622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:56.124747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529644+0000","last_change":"2024-03-18T16:21:21.404392+0000","last_active":"2024-03-18T16:33:04.529644+0000","last_peered":"2024-03-18T16:33:04.529644+0000","last_clean":"2024-03-18T16:33:04.529644+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:33:04.529644+0000","last_undegraded":"2024-03-18T16:33:04.529644+0000","last_fullsized":"2024-03-18T16:33:04.529644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:37.165287+0000","last_change":"2024-03-18T16:21:29.139956+0000","last_active":"2024-03-18T16:33:37.165287+0000","last_peered":"2024-03-18T16:33:37.165287+0000","last_clean":"2024-03-18T16:33:37.165287+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:33:37.165287+0000","last_undegraded":"2024-03-18T16:33:37.165287+0000","last_fullsized":"2024-03-18T16:33:37.165287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:52.356673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.087948+0000","last_change":"2024-03-18T16:21:22.995275+0000","last_active":"2024-03-18T16:33:02.087948+0000","last_peered":"2024-03-18T16:33:02.087948+0000","last_clean":"2024-03-18T16:33:02.087948+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:33:02.087948+0000","last_undegraded":"2024-03-18T16:33:02.087948+0000","last_fullsized":"2024-03-18T16:33:02.087948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:48.024985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.477130+0000","last_change":"2024-03-18T16:21:21.412044+0000","last_active":"2024-03-18T16:33:04.477130+0000","last_peered":"2024-03-18T16:33:04.477130+0000","last_clean":"2024-03-18T16:33:04.477130+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:33:04.477130+0000","last_undegraded":"2024-03-18T16:33:04.477130+0000","last_fullsized":"2024-03-18T16:33:04.477130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.162527+0000","last_change":"2024-03-18T16:21:55.270056+0000","last_active":"2024-03-18T16:33:12.162527+0000","last_peered":"2024-03-18T16:33:12.162527+0000","last_clean":"2024-03-18T16:33:12.162527+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:33:12.162527+0000","last_undegraded":"2024-03-18T16:33:12.162527+0000","last_fullsized":"2024-03-18T16:33:12.162527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:50.466826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.198598+0000","last_change":"2024-03-18T16:21:23.056116+0000","last_active":"2024-03-18T16:33:02.198598+0000","last_peered":"2024-03-18T16:33:02.198598+0000","last_clean":"2024-03-18T16:33:02.198598+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:33:02.198598+0000","last_undegraded":"2024-03-18T16:33:02.198598+0000","last_fullsized":"2024-03-18T16:33:02.198598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:18.455315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.198623+0000","last_change":"2024-03-18T16:21:21.400378+0000","last_active":"2024-03-18T16:33:02.198623+0000","last_peered":"2024-03-18T16:33:02.198623+0000","last_clean":"2024-03-18T16:33:02.198623+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:33:02.198623+0000","last_undegraded":"2024-03-18T16:33:02.198623+0000","last_fullsized":"2024-03-18T16:33:02.198623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:34.479961+0000","last_change":"2024-03-18T16:21:29.141554+0000","last_active":"2024-03-18T16:33:34.479961+0000","last_peered":"2024-03-18T16:33:34.479961+0000","last_clean":"2024-03-18T16:33:34.479961+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:33:34.479961+0000","last_undegraded":"2024-03-18T16:33:34.479961+0000","last_fullsized":"2024-03-18T16:33:34.479961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:25.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.090336+0000","last_change":"2024-03-18T16:21:22.996694+0000","last_active":"2024-03-18T16:33:27.090336+0000","last_peered":"2024-03-18T16:33:27.090336+0000","last_clean":"2024-03-18T16:33:27.090336+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:33:27.090336+0000","last_undegraded":"2024-03-18T16:33:27.090336+0000","last_fullsized":"2024-03-18T16:33:27.090336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:03.704090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529661+0000","last_change":"2024-03-18T16:21:21.412149+0000","last_active":"2024-03-18T16:33:04.529661+0000","last_peered":"2024-03-18T16:33:04.529661+0000","last_clean":"2024-03-18T16:33:04.529661+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:33:04.529661+0000","last_undegraded":"2024-03-18T16:33:04.529661+0000","last_fullsized":"2024-03-18T16:33:04.529661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.952815+0000","last_change":"2024-03-18T16:21:29.077350+0000","last_active":"2024-03-18T16:33:12.952815+0000","last_peered":"2024-03-18T16:33:12.952815+0000","last_clean":"2024-03-18T16:33:12.952815+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:33:12.952815+0000","last_undegraded":"2024-03-18T16:33:12.952815+0000","last_fullsized":"2024-03-18T16:33:12.952815+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.959141+0000","last_change":"2024-03-18T16:21:22.992928+0000","last_active":"2024-03-18T16:33:02.959141+0000","last_peered":"2024-03-18T16:33:02.959141+0000","last_clean":"2024-03-18T16:33:02.959141+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:33:02.959141+0000","last_undegraded":"2024-03-18T16:33:02.959141+0000","last_fullsized":"2024-03-18T16:33:02.959141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:50.978267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.954445+0000","last_change":"2024-03-18T16:21:21.403749+0000","last_active":"2024-03-18T16:33:27.954445+0000","last_peered":"2024-03-18T16:33:27.954445+0000","last_clean":"2024-03-18T16:33:27.954445+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:33:27.954445+0000","last_undegraded":"2024-03-18T16:33:27.954445+0000","last_fullsized":"2024-03-18T16:33:27.954445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:29:30.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:07.959644+0000","last_change":"2024-03-18T16:21:29.065378+0000","last_active":"2024-03-18T16:33:07.959644+0000","last_peered":"2024-03-18T16:33:07.959644+0000","last_clean":"2024-03-18T16:33:07.959644+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:33:07.959644+0000","last_undegraded":"2024-03-18T16:33:07.959644+0000","last_fullsized":"2024-03-18T16:33:07.959644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:22:12.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.201187+0000","last_change":"2024-03-18T16:21:23.054168+0000","last_active":"2024-03-18T16:33:27.201187+0000","last_peered":"2024-03-18T16:33:27.201187+0000","last_clean":"2024-03-18T16:33:27.201187+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:33:27.201187+0000","last_undegraded":"2024-03-18T16:33:27.201187+0000","last_fullsized":"2024-03-18T16:33:27.201187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.479472+0000","last_change":"2024-03-18T16:21:21.398307+0000","last_active":"2024-03-18T16:33:29.479472+0000","last_peered":"2024-03-18T16:33:29.479472+0000","last_clean":"2024-03-18T16:33:29.479472+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:33:29.479472+0000","last_undegraded":"2024-03-18T16:33:29.479472+0000","last_fullsized":"2024-03-18T16:33:29.479472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.162463+0000","last_change":"2024-03-18T16:21:29.065855+0000","last_active":"2024-03-18T16:33:12.162463+0000","last_peered":"2024-03-18T16:33:12.162463+0000","last_clean":"2024-03-18T16:33:12.162463+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:33:12.162463+0000","last_undegraded":"2024-03-18T16:33:12.162463+0000","last_fullsized":"2024-03-18T16:33:12.162463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:43.207302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.479456+0000","last_change":"2024-03-18T16:21:22.993606+0000","last_active":"2024-03-18T16:33:29.479456+0000","last_peered":"2024-03-18T16:33:29.479456+0000","last_clean":"2024-03-18T16:33:29.479456+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:33:29.479456+0000","last_undegraded":"2024-03-18T16:33:29.479456+0000","last_fullsized":"2024-03-18T16:33:29.479456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:38:23.121609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531813+0000","last_change":"2024-03-18T16:21:21.399588+0000","last_active":"2024-03-18T16:33:29.531813+0000","last_peered":"2024-03-18T16:33:29.531813+0000","last_clean":"2024-03-18T16:33:29.531813+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:33:29.531813+0000","last_undegraded":"2024-03-18T16:33:29.531813+0000","last_fullsized":"2024-03-18T16:33:29.531813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.162498+0000","last_change":"2024-03-18T16:21:29.144832+0000","last_active":"2024-03-18T16:33:12.162498+0000","last_peered":"2024-03-18T16:33:12.162498+0000","last_clean":"2024-03-18T16:33:12.162498+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:33:12.162498+0000","last_undegraded":"2024-03-18T16:33:12.162498+0000","last_fullsized":"2024-03-18T16:33:12.162498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:10.103662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:32.201959+0000","last_change":"2024-03-18T16:21:29.148215+0000","last_active":"2024-03-18T16:33:32.201959+0000","last_peered":"2024-03-18T16:33:32.201959+0000","last_clean":"2024-03-18T16:33:32.201959+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:33:32.201959+0000","last_undegraded":"2024-03-18T16:33:32.201959+0000","last_fullsized":"2024-03-18T16:33:32.201959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.954392+0000","last_change":"2024-03-18T16:21:22.980300+0000","last_active":"2024-03-18T16:33:27.954392+0000","last_peered":"2024-03-18T16:33:27.954392+0000","last_clean":"2024-03-18T16:33:27.954392+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:33:27.954392+0000","last_undegraded":"2024-03-18T16:33:27.954392+0000","last_fullsized":"2024-03-18T16:33:27.954392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.479400+0000","last_change":"2024-03-18T16:21:21.329294+0000","last_active":"2024-03-18T16:33:29.479400+0000","last_peered":"2024-03-18T16:33:29.479400+0000","last_clean":"2024-03-18T16:33:29.479400+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:33:29.479400+0000","last_undegraded":"2024-03-18T16:33:29.479400+0000","last_fullsized":"2024-03-18T16:33:29.479400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:59.534588+0000","last_change":"2024-03-18T16:21:29.076706+0000","last_active":"2024-03-18T16:33:59.534588+0000","last_peered":"2024-03-18T16:33:59.534588+0000","last_clean":"2024-03-18T16:33:59.534588+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:33:59.534588+0000","last_undegraded":"2024-03-18T16:33:59.534588+0000","last_fullsized":"2024-03-18T16:33:59.534588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:57.880669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.164115+0000","last_change":"2024-03-18T16:21:23.052989+0000","last_active":"2024-03-18T16:33:27.164115+0000","last_peered":"2024-03-18T16:33:27.164115+0000","last_clean":"2024-03-18T16:33:27.164115+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:33:27.164115+0000","last_undegraded":"2024-03-18T16:33:27.164115+0000","last_fullsized":"2024-03-18T16:33:27.164115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.164154+0000","last_change":"2024-03-18T16:21:21.413020+0000","last_active":"2024-03-18T16:33:27.164154+0000","last_peered":"2024-03-18T16:33:27.164154+0000","last_clean":"2024-03-18T16:33:27.164154+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:33:27.164154+0000","last_undegraded":"2024-03-18T16:33:27.164154+0000","last_fullsized":"2024-03-18T16:33:27.164154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:49.237529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.534182+0000","last_change":"2024-03-18T16:21:21.397354+0000","last_active":"2024-03-18T16:33:54.534182+0000","last_peered":"2024-03-18T16:33:54.534182+0000","last_clean":"2024-03-18T16:33:54.534182+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:33:54.534182+0000","last_undegraded":"2024-03-18T16:33:54.534182+0000","last_fullsized":"2024-03-18T16:33:54.534182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204069+0000","last_change":"2024-03-18T16:21:23.059441+0000","last_active":"2024-03-18T16:33:52.204069+0000","last_peered":"2024-03-18T16:33:52.204069+0000","last_clean":"2024-03-18T16:33:52.204069+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:33:52.204069+0000","last_undegraded":"2024-03-18T16:33:52.204069+0000","last_fullsized":"2024-03-18T16:33:52.204069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:00.666357+0000","last_change":"2024-03-18T16:21:29.058800+0000","last_active":"2024-03-18T16:34:00.666357+0000","last_peered":"2024-03-18T16:34:00.666357+0000","last_clean":"2024-03-18T16:34:00.666357+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:34:00.666357+0000","last_undegraded":"2024-03-18T16:34:00.666357+0000","last_fullsized":"2024-03-18T16:34:00.666357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:20.279579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.167524+0000","last_change":"2024-03-18T16:21:21.412536+0000","last_active":"2024-03-18T16:33:52.167524+0000","last_peered":"2024-03-18T16:33:52.167524+0000","last_clean":"2024-03-18T16:33:52.167524+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:33:52.167524+0000","last_undegraded":"2024-03-18T16:33:52.167524+0000","last_fullsized":"2024-03-18T16:33:52.167524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.963585+0000","last_change":"2024-03-18T16:21:22.974963+0000","last_active":"2024-03-18T16:33:52.963585+0000","last_peered":"2024-03-18T16:33:52.963585+0000","last_clean":"2024-03-18T16:33:52.963585+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:33:52.963585+0000","last_undegraded":"2024-03-18T16:33:52.963585+0000","last_fullsized":"2024-03-18T16:33:52.963585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:51.820583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.169266+0000","last_change":"2024-03-18T16:21:55.151933+0000","last_active":"2024-03-18T16:34:02.169266+0000","last_peered":"2024-03-18T16:34:02.169266+0000","last_clean":"2024-03-18T16:34:02.169266+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:34:02.169266+0000","last_undegraded":"2024-03-18T16:34:02.169266+0000","last_fullsized":"2024-03-18T16:34:02.169266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:38.438575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.534141+0000","last_change":"2024-03-18T16:21:23.056857+0000","last_active":"2024-03-18T16:33:54.534141+0000","last_peered":"2024-03-18T16:33:54.534141+0000","last_clean":"2024-03-18T16:33:54.534141+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:33:54.534141+0000","last_undegraded":"2024-03-18T16:33:54.534141+0000","last_fullsized":"2024-03-18T16:33:54.534141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:13.728335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:55.665554+0000","last_change":"2024-03-18T16:21:22.949289+0000","last_active":"2024-03-18T16:33:55.665554+0000","last_peered":"2024-03-18T16:33:55.665554+0000","last_clean":"2024-03-18T16:33:55.665554+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:33:55.665554+0000","last_undegraded":"2024-03-18T16:33:55.665554+0000","last_fullsized":"2024-03-18T16:33:55.665554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:51:51.357680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:25.669716+0000","last_change":"2024-03-18T16:21:54.845858+0000","last_active":"2024-03-18T16:34:25.669716+0000","last_peered":"2024-03-18T16:34:25.669716+0000","last_clean":"2024-03-18T16:34:25.669716+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:34:25.669716+0000","last_undegraded":"2024-03-18T16:34:25.669716+0000","last_fullsized":"2024-03-18T16:34:25.669716+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204055+0000","last_change":"2024-03-18T16:21:23.059438+0000","last_active":"2024-03-18T16:33:52.204055+0000","last_peered":"2024-03-18T16:33:52.204055+0000","last_clean":"2024-03-18T16:33:52.204055+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:33:52.204055+0000","last_undegraded":"2024-03-18T16:33:52.204055+0000","last_fullsized":"2024-03-18T16:33:52.204055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.171829+0000","last_change":"2024-03-18T16:21:22.849115+0000","last_active":"2024-03-18T16:34:22.171829+0000","last_peered":"2024-03-18T16:34:22.171829+0000","last_clean":"2024-03-18T16:34:22.171829+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:34:22.171829+0000","last_undegraded":"2024-03-18T16:34:22.171829+0000","last_fullsized":"2024-03-18T16:34:22.171829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:57.469261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960363+0000","last_change":"2024-03-18T16:21:22.980253+0000","last_active":"2024-03-18T16:34:22.960363+0000","last_peered":"2024-03-18T16:34:22.960363+0000","last_clean":"2024-03-18T16:34:22.960363+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:34:22.960363+0000","last_undegraded":"2024-03-18T16:34:22.960363+0000","last_fullsized":"2024-03-18T16:34:22.960363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536554+0000","last_change":"2024-03-18T16:21:23.042188+0000","last_active":"2024-03-18T16:34:24.536554+0000","last_peered":"2024-03-18T16:34:24.536554+0000","last_clean":"2024-03-18T16:34:24.536554+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:34:24.536554+0000","last_undegraded":"2024-03-18T16:34:24.536554+0000","last_fullsized":"2024-03-18T16:34:24.536554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.171870+0000","last_change":"2024-03-18T16:21:22.851677+0000","last_active":"2024-03-18T16:34:22.171870+0000","last_peered":"2024-03-18T16:34:22.171870+0000","last_clean":"2024-03-18T16:34:22.171870+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:34:22.171870+0000","last_undegraded":"2024-03-18T16:34:22.171870+0000","last_fullsized":"2024-03-18T16:34:22.171870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.484545+0000","last_change":"2024-03-18T16:21:22.947869+0000","last_active":"2024-03-18T16:34:24.484545+0000","last_peered":"2024-03-18T16:34:24.484545+0000","last_clean":"2024-03-18T16:34:24.484545+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:34:24.484545+0000","last_undegraded":"2024-03-18T16:34:24.484545+0000","last_fullsized":"2024-03-18T16:34:24.484545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.157693+0000","last_change":"2024-03-18T16:21:22.851663+0000","last_active":"2024-03-18T16:32:37.157693+0000","last_peered":"2024-03-18T16:32:37.157693+0000","last_clean":"2024-03-18T16:32:37.157693+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:32:37.157693+0000","last_undegraded":"2024-03-18T16:32:37.157693+0000","last_fullsized":"2024-03-18T16:32:37.157693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:10.660891+0000","last_change":"2024-03-18T16:21:55.268883+0000","last_active":"2024-03-18T16:33:10.660891+0000","last_peered":"2024-03-18T16:33:10.660891+0000","last_clean":"2024-03-18T16:33:10.660891+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:33:10.660891+0000","last_undegraded":"2024-03-18T16:33:10.660891+0000","last_fullsized":"2024-03-18T16:33:10.660891+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.196040+0000","last_change":"2024-03-18T16:21:23.059733+0000","last_active":"2024-03-18T16:32:37.196040+0000","last_peered":"2024-03-18T16:32:37.196040+0000","last_clean":"2024-03-18T16:32:37.196040+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:32:37.196040+0000","last_undegraded":"2024-03-18T16:32:37.196040+0000","last_fullsized":"2024-03-18T16:32:37.196040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:39.527349+0000","last_change":"2024-03-18T16:21:23.042457+0000","last_active":"2024-03-18T16:32:39.527349+0000","last_peered":"2024-03-18T16:32:39.527349+0000","last_clean":"2024-03-18T16:32:39.527349+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:32:39.527349+0000","last_undegraded":"2024-03-18T16:32:39.527349+0000","last_fullsized":"2024-03-18T16:32:39.527349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:09.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.952871+0000","last_change":"2024-03-18T16:21:54.842423+0000","last_active":"2024-03-18T16:33:12.952871+0000","last_peered":"2024-03-18T16:33:12.952871+0000","last_clean":"2024-03-18T16:33:12.952871+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:33:12.952871+0000","last_undegraded":"2024-03-18T16:33:12.952871+0000","last_fullsized":"2024-03-18T16:33:12.952871+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:36.656058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:05.660378+0000","last_change":"2024-03-18T16:21:23.045061+0000","last_active":"2024-03-18T16:33:05.660378+0000","last_peered":"2024-03-18T16:33:05.660378+0000","last_clean":"2024-03-18T16:33:05.660378+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:33:05.660378+0000","last_undegraded":"2024-03-18T16:33:05.660378+0000","last_fullsized":"2024-03-18T16:33:05.660378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:54.119675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.477157+0000","last_change":"2024-03-18T16:21:22.993632+0000","last_active":"2024-03-18T16:33:04.477157+0000","last_peered":"2024-03-18T16:33:04.477157+0000","last_clean":"2024-03-18T16:33:04.477157+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:33:04.477157+0000","last_undegraded":"2024-03-18T16:33:04.477157+0000","last_fullsized":"2024-03-18T16:33:04.477157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:35:42.249037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.477106+0000","last_change":"2024-03-18T16:21:23.056575+0000","last_active":"2024-03-18T16:33:04.477106+0000","last_peered":"2024-03-18T16:33:04.477106+0000","last_clean":"2024-03-18T16:33:04.477106+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:33:04.477106+0000","last_undegraded":"2024-03-18T16:33:04.477106+0000","last_fullsized":"2024-03-18T16:33:04.477106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.951745+0000","last_change":"2024-03-18T16:21:22.973283+0000","last_active":"2024-03-18T16:33:02.951745+0000","last_peered":"2024-03-18T16:33:02.951745+0000","last_clean":"2024-03-18T16:33:02.951745+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:33:02.951745+0000","last_undegraded":"2024-03-18T16:33:02.951745+0000","last_fullsized":"2024-03-18T16:33:02.951745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.198570+0000","last_change":"2024-03-18T16:21:23.057150+0000","last_active":"2024-03-18T16:33:02.198570+0000","last_peered":"2024-03-18T16:33:02.198570+0000","last_clean":"2024-03-18T16:33:02.198570+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:33:02.198570+0000","last_undegraded":"2024-03-18T16:33:02.198570+0000","last_fullsized":"2024-03-18T16:33:02.198570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.164176+0000","last_change":"2024-03-18T16:21:23.047968+0000","last_active":"2024-03-18T16:33:27.164176+0000","last_peered":"2024-03-18T16:33:27.164176+0000","last_clean":"2024-03-18T16:33:27.164176+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:33:27.164176+0000","last_undegraded":"2024-03-18T16:33:27.164176+0000","last_fullsized":"2024-03-18T16:33:27.164176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.961347+0000","last_change":"2024-03-18T16:21:23.041794+0000","last_active":"2024-03-18T16:33:27.961347+0000","last_peered":"2024-03-18T16:33:27.961347+0000","last_clean":"2024-03-18T16:33:27.961347+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:33:27.961347+0000","last_undegraded":"2024-03-18T16:33:27.961347+0000","last_fullsized":"2024-03-18T16:33:27.961347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:13.065494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531782+0000","last_change":"2024-03-18T16:21:23.036940+0000","last_active":"2024-03-18T16:33:29.531782+0000","last_peered":"2024-03-18T16:33:29.531782+0000","last_clean":"2024-03-18T16:33:29.531782+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:33:29.531782+0000","last_undegraded":"2024-03-18T16:33:29.531782+0000","last_fullsized":"2024-03-18T16:33:29.531782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:45:38.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531802+0000","last_change":"2024-03-18T16:21:23.042140+0000","last_active":"2024-03-18T16:33:29.531802+0000","last_peered":"2024-03-18T16:33:29.531802+0000","last_clean":"2024-03-18T16:33:29.531802+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:33:29.531802+0000","last_undegraded":"2024-03-18T16:33:29.531802+0000","last_fullsized":"2024-03-18T16:33:29.531802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.090376+0000","last_change":"2024-03-18T16:21:22.995117+0000","last_active":"2024-03-18T16:33:27.090376+0000","last_peered":"2024-03-18T16:33:27.090376+0000","last_clean":"2024-03-18T16:33:27.090376+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:33:27.090376+0000","last_undegraded":"2024-03-18T16:33:27.090376+0000","last_fullsized":"2024-03-18T16:33:27.090376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:57.544738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.167390+0000","last_change":"2024-03-18T16:21:22.948123+0000","last_active":"2024-03-18T16:33:52.167390+0000","last_peered":"2024-03-18T16:33:52.167390+0000","last_clean":"2024-03-18T16:33:52.167390+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:33:52.167390+0000","last_undegraded":"2024-03-18T16:33:52.167390+0000","last_fullsized":"2024-03-18T16:33:52.167390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092477+0000","last_change":"2024-03-18T16:21:23.049877+0000","last_active":"2024-03-18T16:33:52.092477+0000","last_peered":"2024-03-18T16:33:52.092477+0000","last_clean":"2024-03-18T16:33:52.092477+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:33:52.092477+0000","last_undegraded":"2024-03-18T16:33:52.092477+0000","last_fullsized":"2024-03-18T16:33:52.092477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481775+0000","last_change":"2024-03-18T16:21:23.049483+0000","last_active":"2024-03-18T16:33:54.481775+0000","last_peered":"2024-03-18T16:33:54.481775+0000","last_clean":"2024-03-18T16:33:54.481775+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:33:54.481775+0000","last_undegraded":"2024-03-18T16:33:54.481775+0000","last_fullsized":"2024-03-18T16:33:54.481775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092498+0000","last_change":"2024-03-18T16:21:22.995726+0000","last_active":"2024-03-18T16:33:52.092498+0000","last_peered":"2024-03-18T16:33:52.092498+0000","last_clean":"2024-03-18T16:33:52.092498+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:33:52.092498+0000","last_undegraded":"2024-03-18T16:33:52.092498+0000","last_fullsized":"2024-03-18T16:33:52.092498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:24.478835+0000","last_change":"2024-03-18T16:21:22.949640+0000","last_active":"2024-03-18T16:33:24.478835+0000","last_peered":"2024-03-18T16:33:24.478835+0000","last_clean":"2024-03-18T16:33:24.478835+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:33:24.478835+0000","last_undegraded":"2024-03-18T16:33:24.478835+0000","last_fullsized":"2024-03-18T16:33:24.478835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:21.734293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204083+0000","last_change":"2024-03-18T16:21:23.050203+0000","last_active":"2024-03-18T16:33:52.204083+0000","last_peered":"2024-03-18T16:33:52.204083+0000","last_clean":"2024-03-18T16:33:52.204083+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:33:52.204083+0000","last_undegraded":"2024-03-18T16:33:52.204083+0000","last_fullsized":"2024-03-18T16:33:52.204083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:21.982408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.957297+0000","last_change":"2024-03-18T16:21:23.053003+0000","last_active":"2024-03-18T16:33:52.957297+0000","last_peered":"2024-03-18T16:33:52.957297+0000","last_clean":"2024-03-18T16:33:52.957297+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:33:52.957297+0000","last_undegraded":"2024-03-18T16:33:52.957297+0000","last_fullsized":"2024-03-18T16:33:52.957297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092540+0000","last_change":"2024-03-18T16:21:23.041863+0000","last_active":"2024-03-18T16:33:52.092540+0000","last_peered":"2024-03-18T16:33:52.092540+0000","last_clean":"2024-03-18T16:33:52.092540+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:33:52.092540+0000","last_undegraded":"2024-03-18T16:33:52.092540+0000","last_fullsized":"2024-03-18T16:33:52.092540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:26.190759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509633,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":421888,"data_stored":729077,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509633,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":344064,"data_stored":137939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509631,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":141275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509633,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":138141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509631,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":727851,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542336,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":138071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":141241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575040,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":140541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:36:48.956 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:36:49.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.191+0000 7f15f3000640 1 Processor -- start 2024-03-18T16:36:49.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.191+0000 7f15f3000640 1 -- start start 2024-03-18T16:36:49.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.192+0000 7f15f3000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f15ec06f6b0 0x7f15ec0c8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.192+0000 7f15f3000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f15ec0c8970 0x7f15ec06fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.192+0000 7f15f3000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15ec0c7ca0 con 0x7f15ec06ffd0 2024-03-18T16:36:49.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.192+0000 7f15f3000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f15ec0c73a0 con 0x7f15ec06f6b0 2024-03-18T16:36:49.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.192+0000 7f15f3000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f15ec0c49c0 con 0x7f15ec0c8970 2024-03-18T16:36:49.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.193+0000 7f15ebe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f15ec06f6b0 0x7f15ec0c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.193+0000 7f15eb400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f15ec0c8970 0x7f15ec06fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.193+0000 7f15f1000640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f15ec06ffd0 0x7f15ec15d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58668/0 (socket says 172.21.15.102:58668) 2024-03-18T16:36:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.193+0000 7f15f1000640 1 -- 172.21.15.102:0/3881612225 learned_addr learned my addr 172.21.15.102:0/3881612225 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:49.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.196+0000 7f15eaa00640 1 -- 172.21.15.102:0/3881612225 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1280691187 0 0) 0x7f15ec0c7ca0 con 0x7f15ec06ffd0 2024-03-18T16:36:49.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.196+0000 7f15eaa00640 1 -- 172.21.15.102:0/3881612225 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15d0003520 con 0x7f15ec06ffd0 2024-03-18T16:36:49.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.197+0000 7f15ebe00640 1 -- 172.21.15.102:0/3881612225 >> v2:172.21.15.154:3300/0 conn(0x7f15ec0c8970 msgr2=0x7f15ec06fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:49.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.197+0000 7f15ebe00640 1 --2- 172.21.15.102:0/3881612225 >> v2:172.21.15.154:3300/0 conn(0x7f15ec0c8970 0x7f15ec06fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.197+0000 7f15ebe00640 1 -- 172.21.15.102:0/3881612225 >> v1:172.21.15.154:6789/0 conn(0x7f15ec06ffd0 legacy=0x7f15ec15d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.197+0000 7f15ebe00640 1 -- 172.21.15.102:0/3881612225 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ec15b210 con 0x7f15ec06f6b0 2024-03-18T16:36:49.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.200+0000 7f15ebe00640 1 --2- 172.21.15.102:0/3881612225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15ec06f6b0 0x7f15ec0c8430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0ca030 tx=0x7f15dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4ddaef038d5db0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:49.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.204+0000 7f15eaa00640 1 -- 172.21.15.102:0/3881612225 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc035020 con 0x7f15ec06f6b0 2024-03-18T16:36:49.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.205+0000 7f15eaa00640 1 -- 172.21.15.102:0/3881612225 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15dc031830 con 0x7f15ec06f6b0 2024-03-18T16:36:49.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.205+0000 7f15eaa00640 1 -- 172.21.15.102:0/3881612225 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc031be0 con 0x7f15ec06f6b0 2024-03-18T16:36:49.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.206+0000 7f15f3000640 1 -- 172.21.15.102:0/3881612225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15ec06f6b0 msgr2=0x7f15ec0c8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.206+0000 7f15f3000640 1 --2- 172.21.15.102:0/3881612225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15ec06f6b0 0x7f15ec0c8430 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0ca030 tx=0x7f15dc02b8d0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.206+0000 7f15f3000640 1 -- 172.21.15.102:0/3881612225 shutdown_connections 2024-03-18T16:36:49.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.206+0000 7f15f3000640 1 --2- 172.21.15.102:0/3881612225 >> v2:172.21.15.154:3300/0 conn(0x7f15ec0c8970 0x7f15ec06fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.206+0000 7f15f3000640 1 --2- 172.21.15.102:0/3881612225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15ec06f6b0 0x7f15ec0c8430 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.206+0000 7f15f3000640 1 -- 172.21.15.102:0/3881612225 >> 172.21.15.102:0/3881612225 conn(0x7f15ec0c8f70 msgr2=0x7f15ec0c9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:49.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.206+0000 7f15f3000640 1 -- 172.21.15.102:0/3881612225 shutdown_connections 2024-03-18T16:36:49.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.206+0000 7f15f3000640 1 -- 172.21.15.102:0/3881612225 wait complete. 2024-03-18T16:36:49.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.208+0000 7f15f3000640 1 Processor -- start 2024-03-18T16:36:49.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.208+0000 7f15f3000640 1 -- start start 2024-03-18T16:36:49.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.209+0000 7f15f3000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15ec06f6b0 0x7f15ec150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.209+0000 7f15f3000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15ec06ffd0 0x7f15ec0c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.209+0000 7f15f3000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15ec0c8970 0x7f15ec159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.209+0000 7f15f3000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f15ec0703e0 con 0x7f15ec06ffd0 2024-03-18T16:36:49.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.209+0000 7f15f3000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f15ec0c7f10 con 0x7f15ec0c8970 2024-03-18T16:36:49.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.209+0000 7f15f3000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f15ec154810 con 0x7f15ec06f6b0 2024-03-18T16:36:49.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.210+0000 7f15f1000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15ec0c8970 0x7f15ec159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.210+0000 7f15eb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15ec06ffd0 0x7f15ec0c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.210+0000 7f15ebe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15ec06f6b0 0x7f15ec150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.210+0000 7f15f1000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15ec0c8970 0x7f15ec159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56622/0 (socket says 172.21.15.102:56622) 2024-03-18T16:36:49.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.210+0000 7f15f1000640 1 -- 172.21.15.102:0/3426624263 learned_addr learned my addr 172.21.15.102:0/3426624263 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:49.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.213+0000 7f15f1000640 1 -- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15ec06f6b0 msgr2=0x7f15ec150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.213+0000 7f15f1000640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15ec06f6b0 0x7f15ec150520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.213+0000 7f15f1000640 1 -- 172.21.15.102:0/3426624263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15ec06ffd0 msgr2=0x7f15ec0c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.213+0000 7f15f1000640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15ec06ffd0 0x7f15ec0c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.213+0000 7f15f1000640 1 -- 172.21.15.102:0/3426624263 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15ec150a60 con 0x7f15ec0c8970 2024-03-18T16:36:49.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.216+0000 7f15f1000640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15ec0c8970 0x7f15ec159b60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f15e400cb40 tx=0x7f15e400a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:49.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.219+0000 7f15e3e00640 1 -- 172.21.15.102:0/3426624263 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4017070 con 0x7f15ec0c8970 2024-03-18T16:36:49.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.219+0000 7f15e3e00640 1 -- 172.21.15.102:0/3426624263 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15e400aca0 con 0x7f15ec0c8970 2024-03-18T16:36:49.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.219+0000 7f15e3e00640 1 -- 172.21.15.102:0/3426624263 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4012620 con 0x7f15ec0c8970 2024-03-18T16:36:49.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.219+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15ec14f1a0 con 0x7f15ec0c8970 2024-03-18T16:36:49.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.219+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15ec14f760 con 0x7f15ec0c8970 2024-03-18T16:36:49.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.221+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15ec074fd0 con 0x7f15ec0c8970 2024-03-18T16:36:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.240+0000 7f15e3e00640 1 -- 172.21.15.102:0/3426624263 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f15e401a070 con 0x7f15ec0c8970 2024-03-18T16:36:49.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.242+0000 7f15e3e00640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15cc06bc50 0x7f15cc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.242+0000 7f15e3e00640 1 -- 172.21.15.102:0/3426624263 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f15e4057180 con 0x7f15ec0c8970 2024-03-18T16:36:49.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.242+0000 7f15ebe00640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15cc06bc50 0x7f15cc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.243+0000 7f15ebe00640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15cc06bc50 0x7f15cc06e110 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f15dc036000 tx=0x7f15dc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:49.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.501+0000 7f15e3e00640 1 -- 172.21.15.102:0/3426624263 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f15e4021020 con 0x7f15ec0c8970 2024-03-18T16:36:49.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.690+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f15ec14ef50 con 0x7f15cc06bc50 2024-03-18T16:36:49.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.699+0000 7f15e3e00640 1 -- 172.21.15.102:0/3426624263 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332703 (secure 0 0 0) 0x7f15ec14ef50 con 0x7f15cc06bc50 2024-03-18T16:36:49.705 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:49.707 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:36:49.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.704+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15cc06bc50 msgr2=0x7f15cc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15cc06bc50 0x7f15cc06e110 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f15dc036000 tx=0x7f15dc008030 comp rx=0 tx=0).stop 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15ec0c8970 msgr2=0x7f15ec159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15ec0c8970 0x7f15ec159b60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f15e400cb40 tx=0x7f15e400a060 comp rx=0 tx=0).stop 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 shutdown_connections 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f15cc06bc50 0x7f15cc06e110 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15ec0c8970 0x7f15ec159b60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15ec06ffd0 0x7f15ec0c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 --2- 172.21.15.102:0/3426624263 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f15ec06f6b0 0x7f15ec150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 >> 172.21.15.102:0/3426624263 conn(0x7f15ec0c8f70 msgr2=0x7f15ec14e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 shutdown_connections 2024-03-18T16:36:49.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.705+0000 7f15f3000640 1 -- 172.21.15.102:0/3426624263 wait complete. 2024-03-18T16:36:49.723 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-03-18T16:36:49.034167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219040,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755680,"statfs":{"total":773094113280,"available":772869816320,"internally_reserved":0,"allocated":2949120,"data_stored":2294136,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002813"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204119+0000","last_change":"2024-03-18T16:21:21.402212+0000","last_active":"2024-03-18T16:33:52.204119+0000","last_peered":"2024-03-18T16:33:52.204119+0000","last_clean":"2024-03-18T16:33:52.204119+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:33:52.204119+0000","last_undegraded":"2024-03-18T16:33:52.204119+0000","last_fullsized":"2024-03-18T16:33:52.204119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:33.215915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.163586+0000","last_change":"2024-03-18T16:21:23.045605+0000","last_active":"2024-03-18T16:33:22.163586+0000","last_peered":"2024-03-18T16:33:22.163586+0000","last_clean":"2024-03-18T16:33:22.163586+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:33:22.163586+0000","last_undegraded":"2024-03-18T16:33:22.163586+0000","last_fullsized":"2024-03-18T16:33:22.163586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:37.165226+0000","last_change":"2024-03-18T16:21:29.146958+0000","last_active":"2024-03-18T16:33:37.165226+0000","last_peered":"2024-03-18T16:33:37.165226+0000","last_clean":"2024-03-18T16:33:37.165226+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:33:37.165226+0000","last_undegraded":"2024-03-18T16:33:37.165226+0000","last_fullsized":"2024-03-18T16:33:37.165226+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:55.871927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.163616+0000","last_change":"2024-03-18T16:21:21.413060+0000","last_active":"2024-03-18T16:33:22.163616+0000","last_peered":"2024-03-18T16:33:22.163616+0000","last_clean":"2024-03-18T16:33:22.163616+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:33:22.163616+0000","last_undegraded":"2024-03-18T16:33:22.163616+0000","last_fullsized":"2024-03-18T16:33:22.163616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:40.114937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204105+0000","last_change":"2024-03-18T16:21:23.050849+0000","last_active":"2024-03-18T16:33:52.204105+0000","last_peered":"2024-03-18T16:33:52.204105+0000","last_clean":"2024-03-18T16:33:52.204105+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:33:52.204105+0000","last_undegraded":"2024-03-18T16:33:52.204105+0000","last_fullsized":"2024-03-18T16:33:52.204105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:32.961742+0000","last_change":"2024-03-18T16:21:29.065755+0000","last_active":"2024-03-18T16:33:32.961742+0000","last_peered":"2024-03-18T16:33:32.961742+0000","last_clean":"2024-03-18T16:33:32.961742+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:33:32.961742+0000","last_undegraded":"2024-03-18T16:33:32.961742+0000","last_fullsized":"2024-03-18T16:33:32.961742+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:45.934586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'12","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092522+0000","last_change":"2024-03-18T16:21:21.398188+0000","last_active":"2024-03-18T16:33:52.092522+0000","last_peered":"2024-03-18T16:33:52.092522+0000","last_clean":"2024-03-18T16:33:52.092522+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:33:52.092522+0000","last_undegraded":"2024-03-18T16:33:52.092522+0000","last_fullsized":"2024-03-18T16:33:52.092522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481833+0000","last_change":"2024-03-18T16:21:22.946288+0000","last_active":"2024-03-18T16:33:54.481833+0000","last_peered":"2024-03-18T16:33:54.481833+0000","last_clean":"2024-03-18T16:33:54.481833+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:33:54.481833+0000","last_undegraded":"2024-03-18T16:33:54.481833+0000","last_fullsized":"2024-03-18T16:33:54.481833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:08.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:57.964061+0000","last_change":"2024-03-18T16:21:29.065891+0000","last_active":"2024-03-18T16:33:57.964061+0000","last_peered":"2024-03-18T16:33:57.964061+0000","last_clean":"2024-03-18T16:33:57.964061+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:33:57.964061+0000","last_undegraded":"2024-03-18T16:33:57.964061+0000","last_fullsized":"2024-03-18T16:33:57.964061+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092509+0000","last_change":"2024-03-18T16:21:21.397071+0000","last_active":"2024-03-18T16:33:52.092509+0000","last_peered":"2024-03-18T16:33:52.092509+0000","last_clean":"2024-03-18T16:33:52.092509+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:33:52.092509+0000","last_undegraded":"2024-03-18T16:33:52.092509+0000","last_fullsized":"2024-03-18T16:33:52.092509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481845+0000","last_change":"2024-03-18T16:21:22.946468+0000","last_active":"2024-03-18T16:33:54.481845+0000","last_peered":"2024-03-18T16:33:54.481845+0000","last_clean":"2024-03-18T16:33:54.481845+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:33:54.481845+0000","last_undegraded":"2024-03-18T16:33:54.481845+0000","last_fullsized":"2024-03-18T16:33:54.481845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:57.964049+0000","last_change":"2024-03-18T16:21:29.073724+0000","last_active":"2024-03-18T16:33:57.964049+0000","last_peered":"2024-03-18T16:33:57.964049+0000","last_clean":"2024-03-18T16:33:57.964049+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:33:57.964049+0000","last_undegraded":"2024-03-18T16:33:57.964049+0000","last_fullsized":"2024-03-18T16:33:57.964049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536515+0000","last_change":"2024-03-18T16:21:23.041926+0000","last_active":"2024-03-18T16:34:24.536515+0000","last_peered":"2024-03-18T16:34:24.536515+0000","last_clean":"2024-03-18T16:34:24.536515+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:34:24.536515+0000","last_undegraded":"2024-03-18T16:34:24.536515+0000","last_fullsized":"2024-03-18T16:34:24.536515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:03.532253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085707+0000","last_change":"2024-03-18T16:21:21.394801+0000","last_active":"2024-03-18T16:32:37.085707+0000","last_peered":"2024-03-18T16:32:37.085707+0000","last_clean":"2024-03-18T16:32:37.085707+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:32:37.085707+0000","last_undegraded":"2024-03-18T16:32:37.085707+0000","last_fullsized":"2024-03-18T16:32:37.085707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:42.957341+0000","last_change":"2024-03-18T16:21:29.073757+0000","last_active":"2024-03-18T16:32:42.957341+0000","last_peered":"2024-03-18T16:32:42.957341+0000","last_clean":"2024-03-18T16:32:42.957341+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:32:42.957341+0000","last_undegraded":"2024-03-18T16:32:42.957341+0000","last_fullsized":"2024-03-18T16:32:42.957341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.157732+0000","last_change":"2024-03-18T16:21:23.060776+0000","last_active":"2024-03-18T16:32:37.157732+0000","last_peered":"2024-03-18T16:32:37.157732+0000","last_clean":"2024-03-18T16:32:37.157732+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:32:37.157732+0000","last_undegraded":"2024-03-18T16:32:37.157732+0000","last_fullsized":"2024-03-18T16:32:37.157732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"30'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960340+0000","last_change":"2024-03-18T16:21:21.412733+0000","last_active":"2024-03-18T16:34:22.960340+0000","last_peered":"2024-03-18T16:34:22.960340+0000","last_clean":"2024-03-18T16:34:22.960340+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:34:22.960340+0000","last_undegraded":"2024-03-18T16:34:22.960340+0000","last_fullsized":"2024-03-18T16:34:22.960340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:42.957307+0000","last_change":"2024-03-18T16:21:29.063980+0000","last_active":"2024-03-18T16:32:42.957307+0000","last_peered":"2024-03-18T16:32:42.957307+0000","last_clean":"2024-03-18T16:32:42.957307+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:32:42.957307+0000","last_undegraded":"2024-03-18T16:32:42.957307+0000","last_fullsized":"2024-03-18T16:32:42.957307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:39.469490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:45.657782+0000","last_change":"2024-03-18T16:21:54.838108+0000","last_active":"2024-03-18T16:32:45.657782+0000","last_peered":"2024-03-18T16:32:45.657782+0000","last_clean":"2024-03-18T16:32:45.657782+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:32:45.657782+0000","last_undegraded":"2024-03-18T16:32:45.657782+0000","last_fullsized":"2024-03-18T16:32:45.657782+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:40.182971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960309+0000","last_change":"2024-03-18T16:21:21.407132+0000","last_active":"2024-03-18T16:34:22.960309+0000","last_peered":"2024-03-18T16:34:22.960309+0000","last_clean":"2024-03-18T16:34:22.960309+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:34:22.960309+0000","last_undegraded":"2024-03-18T16:34:22.960309+0000","last_fullsized":"2024-03-18T16:34:22.960309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:10.660930+0000","last_change":"2024-03-18T16:21:55.269661+0000","last_active":"2024-03-18T16:33:10.660930+0000","last_peered":"2024-03-18T16:33:10.660930+0000","last_clean":"2024-03-18T16:33:10.660930+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:33:10.660930+0000","last_undegraded":"2024-03-18T16:33:10.660930+0000","last_fullsized":"2024-03-18T16:33:10.660930+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:38.290937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.966337+0000","last_change":"2024-03-18T16:21:22.840551+0000","last_active":"2024-03-18T16:34:22.966337+0000","last_peered":"2024-03-18T16:34:22.966337+0000","last_clean":"2024-03-18T16:34:22.966337+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:34:22.966337+0000","last_undegraded":"2024-03-18T16:34:22.966337+0000","last_fullsized":"2024-03-18T16:34:22.966337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:17.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.095352+0000","last_change":"2024-03-18T16:21:21.400349+0000","last_active":"2024-03-18T16:34:22.095352+0000","last_peered":"2024-03-18T16:34:22.095352+0000","last_clean":"2024-03-18T16:34:22.095352+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:34:22.095352+0000","last_undegraded":"2024-03-18T16:34:22.095352+0000","last_fullsized":"2024-03-18T16:34:22.095352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:42.957357+0000","last_change":"2024-03-18T16:21:29.064624+0000","last_active":"2024-03-18T16:32:42.957357+0000","last_peered":"2024-03-18T16:32:42.957357+0000","last_clean":"2024-03-18T16:32:42.957357+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:32:42.957357+0000","last_undegraded":"2024-03-18T16:32:42.957357+0000","last_fullsized":"2024-03-18T16:32:42.957357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.207867+0000","last_change":"2024-03-18T16:21:23.055982+0000","last_active":"2024-03-18T16:34:22.207867+0000","last_peered":"2024-03-18T16:34:22.207867+0000","last_clean":"2024-03-18T16:34:22.207867+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:34:22.207867+0000","last_undegraded":"2024-03-18T16:34:22.207867+0000","last_fullsized":"2024-03-18T16:34:22.207867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536571+0000","last_change":"2024-03-18T16:21:21.400843+0000","last_active":"2024-03-18T16:34:24.536571+0000","last_peered":"2024-03-18T16:34:24.536571+0000","last_clean":"2024-03-18T16:34:24.536571+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:34:24.536571+0000","last_undegraded":"2024-03-18T16:34:24.536571+0000","last_fullsized":"2024-03-18T16:34:24.536571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:57.964028+0000","last_change":"2024-03-18T16:21:29.140888+0000","last_active":"2024-03-18T16:33:57.964028+0000","last_peered":"2024-03-18T16:33:57.964028+0000","last_clean":"2024-03-18T16:33:57.964028+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:33:57.964028+0000","last_undegraded":"2024-03-18T16:33:57.964028+0000","last_fullsized":"2024-03-18T16:33:57.964028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.171886+0000","last_change":"2024-03-18T16:21:22.994455+0000","last_active":"2024-03-18T16:34:22.171886+0000","last_peered":"2024-03-18T16:34:22.171886+0000","last_clean":"2024-03-18T16:34:22.171886+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:34:22.171886+0000","last_undegraded":"2024-03-18T16:34:22.171886+0000","last_fullsized":"2024-03-18T16:34:22.171886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960381+0000","last_change":"2024-03-18T16:21:21.402920+0000","last_active":"2024-03-18T16:34:22.960381+0000","last_peered":"2024-03-18T16:34:22.960381+0000","last_clean":"2024-03-18T16:34:22.960381+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:34:22.960381+0000","last_undegraded":"2024-03-18T16:34:22.960381+0000","last_fullsized":"2024-03-18T16:34:22.960381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:32.164696+0000","last_change":"2024-03-18T16:21:29.145235+0000","last_active":"2024-03-18T16:33:32.164696+0000","last_peered":"2024-03-18T16:33:32.164696+0000","last_clean":"2024-03-18T16:33:32.164696+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:33:32.164696+0000","last_undegraded":"2024-03-18T16:33:32.164696+0000","last_fullsized":"2024-03-18T16:33:32.164696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:42.379951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.957241+0000","last_change":"2024-03-18T16:21:22.975172+0000","last_active":"2024-03-18T16:33:52.957241+0000","last_peered":"2024-03-18T16:33:52.957241+0000","last_clean":"2024-03-18T16:33:52.957241+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:33:52.957241+0000","last_undegraded":"2024-03-18T16:33:52.957241+0000","last_fullsized":"2024-03-18T16:33:52.957241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092531+0000","last_change":"2024-03-18T16:21:21.399688+0000","last_active":"2024-03-18T16:33:52.092531+0000","last_peered":"2024-03-18T16:33:52.092531+0000","last_clean":"2024-03-18T16:33:52.092531+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:33:52.092531+0000","last_undegraded":"2024-03-18T16:33:52.092531+0000","last_fullsized":"2024-03-18T16:33:52.092531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:11.933810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.205097+0000","last_change":"2024-03-18T16:21:29.141555+0000","last_active":"2024-03-18T16:34:02.205097+0000","last_peered":"2024-03-18T16:34:02.205097+0000","last_clean":"2024-03-18T16:34:02.205097+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:34:02.205097+0000","last_undegraded":"2024-03-18T16:34:02.205097+0000","last_fullsized":"2024-03-18T16:34:02.205097+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481851+0000","last_change":"2024-03-18T16:21:22.947980+0000","last_active":"2024-03-18T16:33:54.481851+0000","last_peered":"2024-03-18T16:33:54.481851+0000","last_clean":"2024-03-18T16:33:54.481851+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:33:54.481851+0000","last_undegraded":"2024-03-18T16:33:54.481851+0000","last_fullsized":"2024-03-18T16:33:54.481851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:31.739693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"30'2","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.957271+0000","last_change":"2024-03-18T16:21:21.400087+0000","last_active":"2024-03-18T16:33:52.957271+0000","last_peered":"2024-03-18T16:33:52.957271+0000","last_clean":"2024-03-18T16:33:52.957271+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:33:52.957271+0000","last_undegraded":"2024-03-18T16:33:52.957271+0000","last_fullsized":"2024-03-18T16:33:52.957271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.958278+0000","last_change":"2024-03-18T16:21:29.076629+0000","last_active":"2024-03-18T16:34:02.958278+0000","last_peered":"2024-03-18T16:34:02.958278+0000","last_clean":"2024-03-18T16:34:02.958278+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:34:02.958278+0000","last_undegraded":"2024-03-18T16:34:02.958278+0000","last_fullsized":"2024-03-18T16:34:02.958278+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.963620+0000","last_change":"2024-03-18T16:21:23.055810+0000","last_active":"2024-03-18T16:33:52.963620+0000","last_peered":"2024-03-18T16:33:52.963620+0000","last_clean":"2024-03-18T16:33:52.963620+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:33:52.963620+0000","last_undegraded":"2024-03-18T16:33:52.963620+0000","last_fullsized":"2024-03-18T16:33:52.963620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:40.451255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481805+0000","last_change":"2024-03-18T16:21:21.399134+0000","last_active":"2024-03-18T16:33:54.481805+0000","last_peered":"2024-03-18T16:33:54.481805+0000","last_clean":"2024-03-18T16:33:54.481805+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:33:54.481805+0000","last_undegraded":"2024-03-18T16:33:54.481805+0000","last_fullsized":"2024-03-18T16:33:54.481805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.958311+0000","last_change":"2024-03-18T16:21:29.067567+0000","last_active":"2024-03-18T16:34:02.958311+0000","last_peered":"2024-03-18T16:34:02.958311+0000","last_clean":"2024-03-18T16:34:02.958311+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:34:02.958311+0000","last_undegraded":"2024-03-18T16:34:02.958311+0000","last_fullsized":"2024-03-18T16:34:02.958311+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.953865+0000","last_change":"2024-03-18T16:21:22.977089+0000","last_active":"2024-03-18T16:33:22.953865+0000","last_peered":"2024-03-18T16:33:22.953865+0000","last_clean":"2024-03-18T16:33:22.953865+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:33:22.953865+0000","last_undegraded":"2024-03-18T16:33:22.953865+0000","last_fullsized":"2024-03-18T16:33:22.953865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:50.664854+0000","last_change":"2024-03-18T16:21:21.408450+0000","last_active":"2024-03-18T16:33:50.664854+0000","last_peered":"2024-03-18T16:33:50.664854+0000","last_clean":"2024-03-18T16:33:50.664854+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:33:50.664854+0000","last_undegraded":"2024-03-18T16:33:50.664854+0000","last_fullsized":"2024-03-18T16:33:50.664854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:32.961622+0000","last_change":"2024-03-18T16:21:29.076875+0000","last_active":"2024-03-18T16:34:32.961622+0000","last_peered":"2024-03-18T16:34:32.961622+0000","last_clean":"2024-03-18T16:34:32.961622+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:34:32.961622+0000","last_undegraded":"2024-03-18T16:34:32.961622+0000","last_fullsized":"2024-03-18T16:34:32.961622+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204015+0000","last_change":"2024-03-18T16:21:23.054497+0000","last_active":"2024-03-18T16:33:52.204015+0000","last_peered":"2024-03-18T16:33:52.204015+0000","last_clean":"2024-03-18T16:33:52.204015+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:33:52.204015+0000","last_undegraded":"2024-03-18T16:33:52.204015+0000","last_fullsized":"2024-03-18T16:33:52.204015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:37.455180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.089824+0000","last_change":"2024-03-18T16:21:21.404754+0000","last_active":"2024-03-18T16:33:22.089824+0000","last_peered":"2024-03-18T16:33:22.089824+0000","last_clean":"2024-03-18T16:33:22.089824+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:33:22.089824+0000","last_undegraded":"2024-03-18T16:33:22.089824+0000","last_fullsized":"2024-03-18T16:33:22.089824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:32.209201+0000","last_change":"2024-03-18T16:21:29.147316+0000","last_active":"2024-03-18T16:34:32.209201+0000","last_peered":"2024-03-18T16:34:32.209201+0000","last_clean":"2024-03-18T16:34:32.209201+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:34:32.209201+0000","last_undegraded":"2024-03-18T16:34:32.209201+0000","last_fullsized":"2024-03-18T16:34:32.209201+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.167476+0000","last_change":"2024-03-18T16:21:22.947635+0000","last_active":"2024-03-18T16:33:52.167476+0000","last_peered":"2024-03-18T16:33:52.167476+0000","last_clean":"2024-03-18T16:33:52.167476+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:33:52.167476+0000","last_undegraded":"2024-03-18T16:33:52.167476+0000","last_fullsized":"2024-03-18T16:33:52.167476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481823+0000","last_change":"2024-03-18T16:21:21.381306+0000","last_active":"2024-03-18T16:33:54.481823+0000","last_peered":"2024-03-18T16:33:54.481823+0000","last_clean":"2024-03-18T16:33:54.481823+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:33:54.481823+0000","last_undegraded":"2024-03-18T16:33:54.481823+0000","last_fullsized":"2024-03-18T16:33:54.481823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:07.088485+0000","last_change":"2024-03-18T16:21:08.662767+0000","last_active":"2024-03-18T16:33:07.088485+0000","last_peered":"2024-03-18T16:33:07.088485+0000","last_clean":"2024-03-18T16:33:07.088485+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:33:07.088485+0000","last_undegraded":"2024-03-18T16:33:07.088485+0000","last_fullsized":"2024-03-18T16:33:07.088485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_clean_scrub_stamp":"2024-03-18T16:21:05.847994+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:12:10.494707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:59.534564+0000","last_change":"2024-03-18T16:21:29.145697+0000","last_active":"2024-03-18T16:33:59.534564+0000","last_peered":"2024-03-18T16:33:59.534564+0000","last_clean":"2024-03-18T16:33:59.534564+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:33:59.534564+0000","last_undegraded":"2024-03-18T16:33:59.534564+0000","last_fullsized":"2024-03-18T16:33:59.534564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:46.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085732+0000","last_change":"2024-03-18T16:21:22.995291+0000","last_active":"2024-03-18T16:32:37.085732+0000","last_peered":"2024-03-18T16:32:37.085732+0000","last_clean":"2024-03-18T16:32:37.085732+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:32:37.085732+0000","last_undegraded":"2024-03-18T16:32:37.085732+0000","last_fullsized":"2024-03-18T16:32:37.085732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:32.918246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"30'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:35.656730+0000","last_change":"2024-03-18T16:21:21.382137+0000","last_active":"2024-03-18T16:32:35.656730+0000","last_peered":"2024-03-18T16:32:35.656730+0000","last_clean":"2024-03-18T16:32:35.656730+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:32:35.656730+0000","last_undegraded":"2024-03-18T16:32:35.656730+0000","last_fullsized":"2024-03-18T16:32:35.656730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:29.537048+0000","last_change":"2024-03-18T16:21:29.139432+0000","last_active":"2024-03-18T16:34:29.537048+0000","last_peered":"2024-03-18T16:34:29.537048+0000","last_clean":"2024-03-18T16:34:29.537048+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:34:29.537048+0000","last_undegraded":"2024-03-18T16:34:29.537048+0000","last_fullsized":"2024-03-18T16:34:29.537048+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:14.828125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:39.475154+0000","last_change":"2024-03-18T16:21:22.949578+0000","last_active":"2024-03-18T16:32:39.475154+0000","last_peered":"2024-03-18T16:32:39.475154+0000","last_clean":"2024-03-18T16:32:39.475154+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:32:39.475154+0000","last_undegraded":"2024-03-18T16:32:39.475154+0000","last_fullsized":"2024-03-18T16:32:39.475154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:36:20.997720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"30'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.948624+0000","last_change":"2024-03-18T16:21:21.400859+0000","last_active":"2024-03-18T16:32:37.948624+0000","last_peered":"2024-03-18T16:32:37.948624+0000","last_clean":"2024-03-18T16:32:37.948624+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:32:37.948624+0000","last_undegraded":"2024-03-18T16:32:37.948624+0000","last_fullsized":"2024-03-18T16:32:37.948624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:47.197062+0000","last_change":"2024-03-18T16:21:29.076813+0000","last_active":"2024-03-18T16:32:47.197062+0000","last_peered":"2024-03-18T16:32:47.197062+0000","last_clean":"2024-03-18T16:32:47.197062+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:32:47.197062+0000","last_undegraded":"2024-03-18T16:32:47.197062+0000","last_fullsized":"2024-03-18T16:32:47.197062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085756+0000","last_change":"2024-03-18T16:21:23.000009+0000","last_active":"2024-03-18T16:32:37.085756+0000","last_peered":"2024-03-18T16:32:37.085756+0000","last_clean":"2024-03-18T16:32:37.085756+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:32:37.085756+0000","last_undegraded":"2024-03-18T16:32:37.085756+0000","last_fullsized":"2024-03-18T16:32:37.085756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:39.383342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'6","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:35.656773+0000","last_change":"2024-03-18T16:21:21.409271+0000","last_active":"2024-03-18T16:32:35.656773+0000","last_peered":"2024-03-18T16:32:35.656773+0000","last_clean":"2024-03-18T16:32:35.656773+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:32:35.656773+0000","last_undegraded":"2024-03-18T16:32:35.656773+0000","last_fullsized":"2024-03-18T16:32:35.656773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:29.484964+0000","last_change":"2024-03-18T16:21:29.139595+0000","last_active":"2024-03-18T16:34:29.484964+0000","last_peered":"2024-03-18T16:34:29.484964+0000","last_clean":"2024-03-18T16:34:29.484964+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:34:29.484964+0000","last_undegraded":"2024-03-18T16:34:29.484964+0000","last_fullsized":"2024-03-18T16:34:29.484964+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:49.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085742+0000","last_change":"2024-03-18T16:21:23.000054+0000","last_active":"2024-03-18T16:32:37.085742+0000","last_peered":"2024-03-18T16:32:37.085742+0000","last_clean":"2024-03-18T16:32:37.085742+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:32:37.085742+0000","last_undegraded":"2024-03-18T16:32:37.085742+0000","last_fullsized":"2024-03-18T16:32:37.085742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:34:41.535042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"29'2","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.956808+0000","last_change":"2024-03-18T16:21:21.399315+0000","last_active":"2024-03-18T16:32:37.956808+0000","last_peered":"2024-03-18T16:32:37.956808+0000","last_clean":"2024-03-18T16:32:37.956808+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:32:37.956808+0000","last_undegraded":"2024-03-18T16:32:37.956808+0000","last_fullsized":"2024-03-18T16:32:37.956808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:29.543106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:27.966842+0000","last_change":"2024-03-18T16:21:29.065243+0000","last_active":"2024-03-18T16:34:27.966842+0000","last_peered":"2024-03-18T16:34:27.966842+0000","last_clean":"2024-03-18T16:34:27.966842+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:34:27.966842+0000","last_undegraded":"2024-03-18T16:34:27.966842+0000","last_fullsized":"2024-03-18T16:34:27.966842+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.087917+0000","last_change":"2024-03-18T16:21:22.996080+0000","last_active":"2024-03-18T16:33:02.087917+0000","last_peered":"2024-03-18T16:33:02.087917+0000","last_clean":"2024-03-18T16:33:02.087917+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:33:02.087917+0000","last_undegraded":"2024-03-18T16:33:02.087917+0000","last_fullsized":"2024-03-18T16:33:02.087917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'18","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:00.659847+0000","last_change":"2024-03-18T16:21:21.404930+0000","last_active":"2024-03-18T16:33:00.659847+0000","last_peered":"2024-03-18T16:33:00.659847+0000","last_clean":"2024-03-18T16:33:00.659847+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:33:00.659847+0000","last_undegraded":"2024-03-18T16:33:00.659847+0000","last_fullsized":"2024-03-18T16:33:00.659847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:02.571905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:37.202573+0000","last_change":"2024-03-18T16:21:29.079031+0000","last_active":"2024-03-18T16:33:37.202573+0000","last_peered":"2024-03-18T16:33:37.202573+0000","last_clean":"2024-03-18T16:33:37.202573+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:33:37.202573+0000","last_undegraded":"2024-03-18T16:33:37.202573+0000","last_fullsized":"2024-03-18T16:33:37.202573+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529622+0000","last_change":"2024-03-18T16:21:23.056832+0000","last_active":"2024-03-18T16:33:04.529622+0000","last_peered":"2024-03-18T16:33:04.529622+0000","last_clean":"2024-03-18T16:33:04.529622+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:33:04.529622+0000","last_undegraded":"2024-03-18T16:33:04.529622+0000","last_fullsized":"2024-03-18T16:33:04.529622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:56.124747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529644+0000","last_change":"2024-03-18T16:21:21.404392+0000","last_active":"2024-03-18T16:33:04.529644+0000","last_peered":"2024-03-18T16:33:04.529644+0000","last_clean":"2024-03-18T16:33:04.529644+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:33:04.529644+0000","last_undegraded":"2024-03-18T16:33:04.529644+0000","last_fullsized":"2024-03-18T16:33:04.529644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:37.165287+0000","last_change":"2024-03-18T16:21:29.139956+0000","last_active":"2024-03-18T16:33:37.165287+0000","last_peered":"2024-03-18T16:33:37.165287+0000","last_clean":"2024-03-18T16:33:37.165287+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:33:37.165287+0000","last_undegraded":"2024-03-18T16:33:37.165287+0000","last_fullsized":"2024-03-18T16:33:37.165287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:52.356673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.087948+0000","last_change":"2024-03-18T16:21:22.995275+0000","last_active":"2024-03-18T16:33:02.087948+0000","last_peered":"2024-03-18T16:33:02.087948+0000","last_clean":"2024-03-18T16:33:02.087948+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:33:02.087948+0000","last_undegraded":"2024-03-18T16:33:02.087948+0000","last_fullsized":"2024-03-18T16:33:02.087948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:48.024985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.477130+0000","last_change":"2024-03-18T16:21:21.412044+0000","last_active":"2024-03-18T16:33:04.477130+0000","last_peered":"2024-03-18T16:33:04.477130+0000","last_clean":"2024-03-18T16:33:04.477130+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:33:04.477130+0000","last_undegraded":"2024-03-18T16:33:04.477130+0000","last_fullsized":"2024-03-18T16:33:04.477130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.162527+0000","last_change":"2024-03-18T16:21:55.270056+0000","last_active":"2024-03-18T16:33:12.162527+0000","last_peered":"2024-03-18T16:33:12.162527+0000","last_clean":"2024-03-18T16:33:12.162527+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:33:12.162527+0000","last_undegraded":"2024-03-18T16:33:12.162527+0000","last_fullsized":"2024-03-18T16:33:12.162527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:50.466826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.198598+0000","last_change":"2024-03-18T16:21:23.056116+0000","last_active":"2024-03-18T16:33:02.198598+0000","last_peered":"2024-03-18T16:33:02.198598+0000","last_clean":"2024-03-18T16:33:02.198598+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:33:02.198598+0000","last_undegraded":"2024-03-18T16:33:02.198598+0000","last_fullsized":"2024-03-18T16:33:02.198598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:18.455315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.198623+0000","last_change":"2024-03-18T16:21:21.400378+0000","last_active":"2024-03-18T16:33:02.198623+0000","last_peered":"2024-03-18T16:33:02.198623+0000","last_clean":"2024-03-18T16:33:02.198623+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:33:02.198623+0000","last_undegraded":"2024-03-18T16:33:02.198623+0000","last_fullsized":"2024-03-18T16:33:02.198623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:34.479961+0000","last_change":"2024-03-18T16:21:29.141554+0000","last_active":"2024-03-18T16:33:34.479961+0000","last_peered":"2024-03-18T16:33:34.479961+0000","last_clean":"2024-03-18T16:33:34.479961+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:33:34.479961+0000","last_undegraded":"2024-03-18T16:33:34.479961+0000","last_fullsized":"2024-03-18T16:33:34.479961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:25.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.090336+0000","last_change":"2024-03-18T16:21:22.996694+0000","last_active":"2024-03-18T16:33:27.090336+0000","last_peered":"2024-03-18T16:33:27.090336+0000","last_clean":"2024-03-18T16:33:27.090336+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:33:27.090336+0000","last_undegraded":"2024-03-18T16:33:27.090336+0000","last_fullsized":"2024-03-18T16:33:27.090336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:03.704090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529661+0000","last_change":"2024-03-18T16:21:21.412149+0000","last_active":"2024-03-18T16:33:04.529661+0000","last_peered":"2024-03-18T16:33:04.529661+0000","last_clean":"2024-03-18T16:33:04.529661+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:33:04.529661+0000","last_undegraded":"2024-03-18T16:33:04.529661+0000","last_fullsized":"2024-03-18T16:33:04.529661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.952815+0000","last_change":"2024-03-18T16:21:29.077350+0000","last_active":"2024-03-18T16:33:12.952815+0000","last_peered":"2024-03-18T16:33:12.952815+0000","last_clean":"2024-03-18T16:33:12.952815+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:33:12.952815+0000","last_undegraded":"2024-03-18T16:33:12.952815+0000","last_fullsized":"2024-03-18T16:33:12.952815+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.959141+0000","last_change":"2024-03-18T16:21:22.992928+0000","last_active":"2024-03-18T16:33:02.959141+0000","last_peered":"2024-03-18T16:33:02.959141+0000","last_clean":"2024-03-18T16:33:02.959141+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:33:02.959141+0000","last_undegraded":"2024-03-18T16:33:02.959141+0000","last_fullsized":"2024-03-18T16:33:02.959141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:50.978267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.954445+0000","last_change":"2024-03-18T16:21:21.403749+0000","last_active":"2024-03-18T16:33:27.954445+0000","last_peered":"2024-03-18T16:33:27.954445+0000","last_clean":"2024-03-18T16:33:27.954445+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:33:27.954445+0000","last_undegraded":"2024-03-18T16:33:27.954445+0000","last_fullsized":"2024-03-18T16:33:27.954445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:29:30.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:07.959644+0000","last_change":"2024-03-18T16:21:29.065378+0000","last_active":"2024-03-18T16:33:07.959644+0000","last_peered":"2024-03-18T16:33:07.959644+0000","last_clean":"2024-03-18T16:33:07.959644+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:33:07.959644+0000","last_undegraded":"2024-03-18T16:33:07.959644+0000","last_fullsized":"2024-03-18T16:33:07.959644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:22:12.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.201187+0000","last_change":"2024-03-18T16:21:23.054168+0000","last_active":"2024-03-18T16:33:27.201187+0000","last_peered":"2024-03-18T16:33:27.201187+0000","last_clean":"2024-03-18T16:33:27.201187+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:33:27.201187+0000","last_undegraded":"2024-03-18T16:33:27.201187+0000","last_fullsized":"2024-03-18T16:33:27.201187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.479472+0000","last_change":"2024-03-18T16:21:21.398307+0000","last_active":"2024-03-18T16:33:29.479472+0000","last_peered":"2024-03-18T16:33:29.479472+0000","last_clean":"2024-03-18T16:33:29.479472+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:33:29.479472+0000","last_undegraded":"2024-03-18T16:33:29.479472+0000","last_fullsized":"2024-03-18T16:33:29.479472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.162463+0000","last_change":"2024-03-18T16:21:29.065855+0000","last_active":"2024-03-18T16:33:12.162463+0000","last_peered":"2024-03-18T16:33:12.162463+0000","last_clean":"2024-03-18T16:33:12.162463+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:33:12.162463+0000","last_undegraded":"2024-03-18T16:33:12.162463+0000","last_fullsized":"2024-03-18T16:33:12.162463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:43.207302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.479456+0000","last_change":"2024-03-18T16:21:22.993606+0000","last_active":"2024-03-18T16:33:29.479456+0000","last_peered":"2024-03-18T16:33:29.479456+0000","last_clean":"2024-03-18T16:33:29.479456+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:33:29.479456+0000","last_undegraded":"2024-03-18T16:33:29.479456+0000","last_fullsized":"2024-03-18T16:33:29.479456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:38:23.121609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531813+0000","last_change":"2024-03-18T16:21:21.399588+0000","last_active":"2024-03-18T16:33:29.531813+0000","last_peered":"2024-03-18T16:33:29.531813+0000","last_clean":"2024-03-18T16:33:29.531813+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:33:29.531813+0000","last_undegraded":"2024-03-18T16:33:29.531813+0000","last_fullsized":"2024-03-18T16:33:29.531813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.162498+0000","last_change":"2024-03-18T16:21:29.144832+0000","last_active":"2024-03-18T16:33:12.162498+0000","last_peered":"2024-03-18T16:33:12.162498+0000","last_clean":"2024-03-18T16:33:12.162498+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:33:12.162498+0000","last_undegraded":"2024-03-18T16:33:12.162498+0000","last_fullsized":"2024-03-18T16:33:12.162498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:10.103662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:32.201959+0000","last_change":"2024-03-18T16:21:29.148215+0000","last_active":"2024-03-18T16:33:32.201959+0000","last_peered":"2024-03-18T16:33:32.201959+0000","last_clean":"2024-03-18T16:33:32.201959+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:33:32.201959+0000","last_undegraded":"2024-03-18T16:33:32.201959+0000","last_fullsized":"2024-03-18T16:33:32.201959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.954392+0000","last_change":"2024-03-18T16:21:22.980300+0000","last_active":"2024-03-18T16:33:27.954392+0000","last_peered":"2024-03-18T16:33:27.954392+0000","last_clean":"2024-03-18T16:33:27.954392+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:33:27.954392+0000","last_undegraded":"2024-03-18T16:33:27.954392+0000","last_fullsized":"2024-03-18T16:33:27.954392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.479400+0000","last_change":"2024-03-18T16:21:21.329294+0000","last_active":"2024-03-18T16:33:29.479400+0000","last_peered":"2024-03-18T16:33:29.479400+0000","last_clean":"2024-03-18T16:33:29.479400+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:33:29.479400+0000","last_undegraded":"2024-03-18T16:33:29.479400+0000","last_fullsized":"2024-03-18T16:33:29.479400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:59.534588+0000","last_change":"2024-03-18T16:21:29.076706+0000","last_active":"2024-03-18T16:33:59.534588+0000","last_peered":"2024-03-18T16:33:59.534588+0000","last_clean":"2024-03-18T16:33:59.534588+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:33:59.534588+0000","last_undegraded":"2024-03-18T16:33:59.534588+0000","last_fullsized":"2024-03-18T16:33:59.534588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:57.880669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.164115+0000","last_change":"2024-03-18T16:21:23.052989+0000","last_active":"2024-03-18T16:33:27.164115+0000","last_peered":"2024-03-18T16:33:27.164115+0000","last_clean":"2024-03-18T16:33:27.164115+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:33:27.164115+0000","last_undegraded":"2024-03-18T16:33:27.164115+0000","last_fullsized":"2024-03-18T16:33:27.164115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.164154+0000","last_change":"2024-03-18T16:21:21.413020+0000","last_active":"2024-03-18T16:33:27.164154+0000","last_peered":"2024-03-18T16:33:27.164154+0000","last_clean":"2024-03-18T16:33:27.164154+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:33:27.164154+0000","last_undegraded":"2024-03-18T16:33:27.164154+0000","last_fullsized":"2024-03-18T16:33:27.164154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:49.237529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.534182+0000","last_change":"2024-03-18T16:21:21.397354+0000","last_active":"2024-03-18T16:33:54.534182+0000","last_peered":"2024-03-18T16:33:54.534182+0000","last_clean":"2024-03-18T16:33:54.534182+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:33:54.534182+0000","last_undegraded":"2024-03-18T16:33:54.534182+0000","last_fullsized":"2024-03-18T16:33:54.534182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204069+0000","last_change":"2024-03-18T16:21:23.059441+0000","last_active":"2024-03-18T16:33:52.204069+0000","last_peered":"2024-03-18T16:33:52.204069+0000","last_clean":"2024-03-18T16:33:52.204069+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:33:52.204069+0000","last_undegraded":"2024-03-18T16:33:52.204069+0000","last_fullsized":"2024-03-18T16:33:52.204069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:00.666357+0000","last_change":"2024-03-18T16:21:29.058800+0000","last_active":"2024-03-18T16:34:00.666357+0000","last_peered":"2024-03-18T16:34:00.666357+0000","last_clean":"2024-03-18T16:34:00.666357+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:34:00.666357+0000","last_undegraded":"2024-03-18T16:34:00.666357+0000","last_fullsized":"2024-03-18T16:34:00.666357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:20.279579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.167524+0000","last_change":"2024-03-18T16:21:21.412536+0000","last_active":"2024-03-18T16:33:52.167524+0000","last_peered":"2024-03-18T16:33:52.167524+0000","last_clean":"2024-03-18T16:33:52.167524+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:33:52.167524+0000","last_undegraded":"2024-03-18T16:33:52.167524+0000","last_fullsized":"2024-03-18T16:33:52.167524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.963585+0000","last_change":"2024-03-18T16:21:22.974963+0000","last_active":"2024-03-18T16:33:52.963585+0000","last_peered":"2024-03-18T16:33:52.963585+0000","last_clean":"2024-03-18T16:33:52.963585+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:33:52.963585+0000","last_undegraded":"2024-03-18T16:33:52.963585+0000","last_fullsized":"2024-03-18T16:33:52.963585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:51.820583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.169266+0000","last_change":"2024-03-18T16:21:55.151933+0000","last_active":"2024-03-18T16:34:02.169266+0000","last_peered":"2024-03-18T16:34:02.169266+0000","last_clean":"2024-03-18T16:34:02.169266+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:34:02.169266+0000","last_undegraded":"2024-03-18T16:34:02.169266+0000","last_fullsized":"2024-03-18T16:34:02.169266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:38.438575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.534141+0000","last_change":"2024-03-18T16:21:23.056857+0000","last_active":"2024-03-18T16:33:54.534141+0000","last_peered":"2024-03-18T16:33:54.534141+0000","last_clean":"2024-03-18T16:33:54.534141+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:33:54.534141+0000","last_undegraded":"2024-03-18T16:33:54.534141+0000","last_fullsized":"2024-03-18T16:33:54.534141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:13.728335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:55.665554+0000","last_change":"2024-03-18T16:21:22.949289+0000","last_active":"2024-03-18T16:33:55.665554+0000","last_peered":"2024-03-18T16:33:55.665554+0000","last_clean":"2024-03-18T16:33:55.665554+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:33:55.665554+0000","last_undegraded":"2024-03-18T16:33:55.665554+0000","last_fullsized":"2024-03-18T16:33:55.665554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:51:51.357680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:25.669716+0000","last_change":"2024-03-18T16:21:54.845858+0000","last_active":"2024-03-18T16:34:25.669716+0000","last_peered":"2024-03-18T16:34:25.669716+0000","last_clean":"2024-03-18T16:34:25.669716+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:34:25.669716+0000","last_undegraded":"2024-03-18T16:34:25.669716+0000","last_fullsized":"2024-03-18T16:34:25.669716+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204055+0000","last_change":"2024-03-18T16:21:23.059438+0000","last_active":"2024-03-18T16:33:52.204055+0000","last_peered":"2024-03-18T16:33:52.204055+0000","last_clean":"2024-03-18T16:33:52.204055+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:33:52.204055+0000","last_undegraded":"2024-03-18T16:33:52.204055+0000","last_fullsized":"2024-03-18T16:33:52.204055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.171829+0000","last_change":"2024-03-18T16:21:22.849115+0000","last_active":"2024-03-18T16:34:22.171829+0000","last_peered":"2024-03-18T16:34:22.171829+0000","last_clean":"2024-03-18T16:34:22.171829+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:34:22.171829+0000","last_undegraded":"2024-03-18T16:34:22.171829+0000","last_fullsized":"2024-03-18T16:34:22.171829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:57.469261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960363+0000","last_change":"2024-03-18T16:21:22.980253+0000","last_active":"2024-03-18T16:34:22.960363+0000","last_peered":"2024-03-18T16:34:22.960363+0000","last_clean":"2024-03-18T16:34:22.960363+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:34:22.960363+0000","last_undegraded":"2024-03-18T16:34:22.960363+0000","last_fullsized":"2024-03-18T16:34:22.960363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536554+0000","last_change":"2024-03-18T16:21:23.042188+0000","last_active":"2024-03-18T16:34:24.536554+0000","last_peered":"2024-03-18T16:34:24.536554+0000","last_clean":"2024-03-18T16:34:24.536554+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:34:24.536554+0000","last_undegraded":"2024-03-18T16:34:24.536554+0000","last_fullsized":"2024-03-18T16:34:24.536554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.171870+0000","last_change":"2024-03-18T16:21:22.851677+0000","last_active":"2024-03-18T16:34:22.171870+0000","last_peered":"2024-03-18T16:34:22.171870+0000","last_clean":"2024-03-18T16:34:22.171870+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:34:22.171870+0000","last_undegraded":"2024-03-18T16:34:22.171870+0000","last_fullsized":"2024-03-18T16:34:22.171870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.484545+0000","last_change":"2024-03-18T16:21:22.947869+0000","last_active":"2024-03-18T16:34:24.484545+0000","last_peered":"2024-03-18T16:34:24.484545+0000","last_clean":"2024-03-18T16:34:24.484545+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:34:24.484545+0000","last_undegraded":"2024-03-18T16:34:24.484545+0000","last_fullsized":"2024-03-18T16:34:24.484545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.157693+0000","last_change":"2024-03-18T16:21:22.851663+0000","last_active":"2024-03-18T16:32:37.157693+0000","last_peered":"2024-03-18T16:32:37.157693+0000","last_clean":"2024-03-18T16:32:37.157693+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:32:37.157693+0000","last_undegraded":"2024-03-18T16:32:37.157693+0000","last_fullsized":"2024-03-18T16:32:37.157693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:10.660891+0000","last_change":"2024-03-18T16:21:55.268883+0000","last_active":"2024-03-18T16:33:10.660891+0000","last_peered":"2024-03-18T16:33:10.660891+0000","last_clean":"2024-03-18T16:33:10.660891+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:33:10.660891+0000","last_undegraded":"2024-03-18T16:33:10.660891+0000","last_fullsized":"2024-03-18T16:33:10.660891+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.196040+0000","last_change":"2024-03-18T16:21:23.059733+0000","last_active":"2024-03-18T16:32:37.196040+0000","last_peered":"2024-03-18T16:32:37.196040+0000","last_clean":"2024-03-18T16:32:37.196040+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:32:37.196040+0000","last_undegraded":"2024-03-18T16:32:37.196040+0000","last_fullsized":"2024-03-18T16:32:37.196040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:39.527349+0000","last_change":"2024-03-18T16:21:23.042457+0000","last_active":"2024-03-18T16:32:39.527349+0000","last_peered":"2024-03-18T16:32:39.527349+0000","last_clean":"2024-03-18T16:32:39.527349+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:32:39.527349+0000","last_undegraded":"2024-03-18T16:32:39.527349+0000","last_fullsized":"2024-03-18T16:32:39.527349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:09.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.952871+0000","last_change":"2024-03-18T16:21:54.842423+0000","last_active":"2024-03-18T16:33:12.952871+0000","last_peered":"2024-03-18T16:33:12.952871+0000","last_clean":"2024-03-18T16:33:12.952871+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:33:12.952871+0000","last_undegraded":"2024-03-18T16:33:12.952871+0000","last_fullsized":"2024-03-18T16:33:12.952871+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:36.656058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:05.660378+0000","last_change":"2024-03-18T16:21:23.045061+0000","last_active":"2024-03-18T16:33:05.660378+0000","last_peered":"2024-03-18T16:33:05.660378+0000","last_clean":"2024-03-18T16:33:05.660378+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:33:05.660378+0000","last_undegraded":"2024-03-18T16:33:05.660378+0000","last_fullsized":"2024-03-18T16:33:05.660378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:54.119675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.477157+0000","last_change":"2024-03-18T16:21:22.993632+0000","last_active":"2024-03-18T16:33:04.477157+0000","last_peered":"2024-03-18T16:33:04.477157+0000","last_clean":"2024-03-18T16:33:04.477157+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:33:04.477157+0000","last_undegraded":"2024-03-18T16:33:04.477157+0000","last_fullsized":"2024-03-18T16:33:04.477157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:35:42.249037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.477106+0000","last_change":"2024-03-18T16:21:23.056575+0000","last_active":"2024-03-18T16:33:04.477106+0000","last_peered":"2024-03-18T16:33:04.477106+0000","last_clean":"2024-03-18T16:33:04.477106+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:33:04.477106+0000","last_undegraded":"2024-03-18T16:33:04.477106+0000","last_fullsized":"2024-03-18T16:33:04.477106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.951745+0000","last_change":"2024-03-18T16:21:22.973283+0000","last_active":"2024-03-18T16:33:02.951745+0000","last_peered":"2024-03-18T16:33:02.951745+0000","last_clean":"2024-03-18T16:33:02.951745+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:33:02.951745+0000","last_undegraded":"2024-03-18T16:33:02.951745+0000","last_fullsized":"2024-03-18T16:33:02.951745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.198570+0000","last_change":"2024-03-18T16:21:23.057150+0000","last_active":"2024-03-18T16:33:02.198570+0000","last_peered":"2024-03-18T16:33:02.198570+0000","last_clean":"2024-03-18T16:33:02.198570+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:33:02.198570+0000","last_undegraded":"2024-03-18T16:33:02.198570+0000","last_fullsized":"2024-03-18T16:33:02.198570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.164176+0000","last_change":"2024-03-18T16:21:23.047968+0000","last_active":"2024-03-18T16:33:27.164176+0000","last_peered":"2024-03-18T16:33:27.164176+0000","last_clean":"2024-03-18T16:33:27.164176+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:33:27.164176+0000","last_undegraded":"2024-03-18T16:33:27.164176+0000","last_fullsized":"2024-03-18T16:33:27.164176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.961347+0000","last_change":"2024-03-18T16:21:23.041794+0000","last_active":"2024-03-18T16:33:27.961347+0000","last_peered":"2024-03-18T16:33:27.961347+0000","last_clean":"2024-03-18T16:33:27.961347+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:33:27.961347+0000","last_undegraded":"2024-03-18T16:33:27.961347+0000","last_fullsized":"2024-03-18T16:33:27.961347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:13.065494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531782+0000","last_change":"2024-03-18T16:21:23.036940+0000","last_active":"2024-03-18T16:33:29.531782+0000","last_peered":"2024-03-18T16:33:29.531782+0000","last_clean":"2024-03-18T16:33:29.531782+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:33:29.531782+0000","last_undegraded":"2024-03-18T16:33:29.531782+0000","last_fullsized":"2024-03-18T16:33:29.531782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:45:38.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531802+0000","last_change":"2024-03-18T16:21:23.042140+0000","last_active":"2024-03-18T16:33:29.531802+0000","last_peered":"2024-03-18T16:33:29.531802+0000","last_clean":"2024-03-18T16:33:29.531802+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:33:29.531802+0000","last_undegraded":"2024-03-18T16:33:29.531802+0000","last_fullsized":"2024-03-18T16:33:29.531802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.090376+0000","last_change":"2024-03-18T16:21:22.995117+0000","last_active":"2024-03-18T16:33:27.090376+0000","last_peered":"2024-03-18T16:33:27.090376+0000","last_clean":"2024-03-18T16:33:27.090376+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:33:27.090376+0000","last_undegraded":"2024-03-18T16:33:27.090376+0000","last_fullsized":"2024-03-18T16:33:27.090376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:57.544738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.167390+0000","last_change":"2024-03-18T16:21:22.948123+0000","last_active":"2024-03-18T16:33:52.167390+0000","last_peered":"2024-03-18T16:33:52.167390+0000","last_clean":"2024-03-18T16:33:52.167390+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:33:52.167390+0000","last_undegraded":"2024-03-18T16:33:52.167390+0000","last_fullsized":"2024-03-18T16:33:52.167390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092477+0000","last_change":"2024-03-18T16:21:23.049877+0000","last_active":"2024-03-18T16:33:52.092477+0000","last_peered":"2024-03-18T16:33:52.092477+0000","last_clean":"2024-03-18T16:33:52.092477+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:33:52.092477+0000","last_undegraded":"2024-03-18T16:33:52.092477+0000","last_fullsized":"2024-03-18T16:33:52.092477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481775+0000","last_change":"2024-03-18T16:21:23.049483+0000","last_active":"2024-03-18T16:33:54.481775+0000","last_peered":"2024-03-18T16:33:54.481775+0000","last_clean":"2024-03-18T16:33:54.481775+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:33:54.481775+0000","last_undegraded":"2024-03-18T16:33:54.481775+0000","last_fullsized":"2024-03-18T16:33:54.481775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092498+0000","last_change":"2024-03-18T16:21:22.995726+0000","last_active":"2024-03-18T16:33:52.092498+0000","last_peered":"2024-03-18T16:33:52.092498+0000","last_clean":"2024-03-18T16:33:52.092498+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:33:52.092498+0000","last_undegraded":"2024-03-18T16:33:52.092498+0000","last_fullsized":"2024-03-18T16:33:52.092498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:24.478835+0000","last_change":"2024-03-18T16:21:22.949640+0000","last_active":"2024-03-18T16:33:24.478835+0000","last_peered":"2024-03-18T16:33:24.478835+0000","last_clean":"2024-03-18T16:33:24.478835+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:33:24.478835+0000","last_undegraded":"2024-03-18T16:33:24.478835+0000","last_fullsized":"2024-03-18T16:33:24.478835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:21.734293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204083+0000","last_change":"2024-03-18T16:21:23.050203+0000","last_active":"2024-03-18T16:33:52.204083+0000","last_peered":"2024-03-18T16:33:52.204083+0000","last_clean":"2024-03-18T16:33:52.204083+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:33:52.204083+0000","last_undegraded":"2024-03-18T16:33:52.204083+0000","last_fullsized":"2024-03-18T16:33:52.204083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:21.982408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.957297+0000","last_change":"2024-03-18T16:21:23.053003+0000","last_active":"2024-03-18T16:33:52.957297+0000","last_peered":"2024-03-18T16:33:52.957297+0000","last_clean":"2024-03-18T16:33:52.957297+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:33:52.957297+0000","last_undegraded":"2024-03-18T16:33:52.957297+0000","last_fullsized":"2024-03-18T16:33:52.957297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092540+0000","last_change":"2024-03-18T16:21:23.041863+0000","last_active":"2024-03-18T16:33:52.092540+0000","last_peered":"2024-03-18T16:33:52.092540+0000","last_clean":"2024-03-18T16:33:52.092540+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:33:52.092540+0000","last_undegraded":"2024-03-18T16:33:52.092540+0000","last_fullsized":"2024-03-18T16:33:52.092540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:26.190759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509633,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":421888,"data_stored":729077,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509634,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":344064,"data_stored":137939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509632,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":141275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509634,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":138141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509632,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":727851,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542336,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":138071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":141241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575040,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":140541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:36:49.729 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-18T16:36:49.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:36:49.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.933+0000 7f0ae5c00640 1 Processor -- start 2024-03-18T16:36:49.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.933+0000 7f0ae5c00640 1 -- start start 2024-03-18T16:36:49.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.934+0000 7f0ae5c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0ae00cfe70 0x7f0ae015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.934+0000 7f0ae5c00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0ae00d0250 0x7f0ae00c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.934+0000 7f0ae5c00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae00cabd0 con 0x7f0ae00cc2f0 2024-03-18T16:36:49.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.934+0000 7f0ae5c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae00caea0 con 0x7f0ae00cfe70 2024-03-18T16:36:49.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.934+0000 7f0ae5c00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae00cb180 con 0x7f0ae00d0250 2024-03-18T16:36:49.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.935+0000 7f0adfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0ae00cfe70 0x7f0ae015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.935+0000 7f0ae5200640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0ae00cc2f0 0x7f0ae00cc740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58676/0 (socket says 172.21.15.102:58676) 2024-03-18T16:36:49.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.935+0000 7f0ae5200640 1 -- 172.21.15.102:0/3818818955 learned_addr learned my addr 172.21.15.102:0/3818818955 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:49.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.935+0000 7f0adf400640 1 --2- 172.21.15.102:0/3818818955 >> v2:172.21.15.154:3300/0 conn(0x7f0ae00d0250 0x7f0ae00c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.938+0000 7f0adfe00640 1 -- 172.21.15.102:0/3818818955 >> v2:172.21.15.154:3300/0 conn(0x7f0ae00d0250 msgr2=0x7f0ae00c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:49.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.938+0000 7f0adfe00640 1 --2- 172.21.15.102:0/3818818955 >> v2:172.21.15.154:3300/0 conn(0x7f0ae00d0250 0x7f0ae00c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.938+0000 7f0adfe00640 1 -- 172.21.15.102:0/3818818955 >> v1:172.21.15.154:6789/0 conn(0x7f0ae00cc2f0 legacy=0x7f0ae00cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.938+0000 7f0adfe00640 1 -- 172.21.15.102:0/3818818955 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae00cb460 con 0x7f0ae00cfe70 2024-03-18T16:36:49.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.941+0000 7f0adfe00640 1 --2- 172.21.15.102:0/3818818955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cfe70 0x7f0ae015ae40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00c6370 tx=0x7f0ad002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a938282ca6e36c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.943+0000 7f0adea00640 1 -- 172.21.15.102:0/3818818955 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0034070 con 0x7f0ae00cfe70 2024-03-18T16:36:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.946+0000 7f0adea00640 1 -- 172.21.15.102:0/3818818955 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad0030610 con 0x7f0ae00cfe70 2024-03-18T16:36:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.946+0000 7f0adea00640 1 -- 172.21.15.102:0/3818818955 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad00309e0 con 0x7f0ae00cfe70 2024-03-18T16:36:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.947+0000 7f0ae5c00640 1 -- 172.21.15.102:0/3818818955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cfe70 msgr2=0x7f0ae015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.947+0000 7f0ae5c00640 1 --2- 172.21.15.102:0/3818818955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cfe70 0x7f0ae015ae40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0ae00c6370 tx=0x7f0ad002b960 comp rx=0 tx=0).stop 2024-03-18T16:36:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.947+0000 7f0ae5c00640 1 -- 172.21.15.102:0/3818818955 shutdown_connections 2024-03-18T16:36:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.947+0000 7f0ae5c00640 1 --2- 172.21.15.102:0/3818818955 >> v2:172.21.15.154:3300/0 conn(0x7f0ae00d0250 0x7f0ae00c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.947+0000 7f0ae5c00640 1 --2- 172.21.15.102:0/3818818955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cfe70 0x7f0ae015ae40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.947+0000 7f0ae5c00640 1 -- 172.21.15.102:0/3818818955 >> 172.21.15.102:0/3818818955 conn(0x7f0ae008c3e0 msgr2=0x7f0ae006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.947+0000 7f0ae5c00640 1 -- 172.21.15.102:0/3818818955 shutdown_connections 2024-03-18T16:36:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.948+0000 7f0ae5c00640 1 -- 172.21.15.102:0/3818818955 wait complete. 2024-03-18T16:36:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.950+0000 7f0ae5c00640 1 Processor -- start 2024-03-18T16:36:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.950+0000 7f0ae5c00640 1 -- start start 2024-03-18T16:36:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.952+0000 7f0ae5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cc2f0 0x7f0ae01f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.952+0000 7f0ae5c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae00cfe70 0x7f0ae0156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.952+0000 7f0ae5c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae00d0250 0x7f0ae01596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:49.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.952+0000 7f0ae5c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae00cbd00 con 0x7f0ae00cc2f0 2024-03-18T16:36:49.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.952+0000 7f0ae5c00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae00cba20 con 0x7f0ae00cfe70 2024-03-18T16:36:49.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.952+0000 7f0ae5c00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae00c6b20 con 0x7f0ae00d0250 2024-03-18T16:36:49.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.953+0000 7f0ae5200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae00d0250 0x7f0ae01596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.953+0000 7f0adfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cc2f0 0x7f0ae01f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.953+0000 7f0ae5200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae00d0250 0x7f0ae01596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:33528/0 (socket says 172.21.15.102:33528) 2024-03-18T16:36:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.953+0000 7f0ae5200640 1 -- 172.21.15.102:0/1734286192 learned_addr learned my addr 172.21.15.102:0/1734286192 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:49.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.954+0000 7f0adf400640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae00cfe70 0x7f0ae0156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:49.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.957+0000 7f0adfe00640 1 -- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae00d0250 msgr2=0x7f0ae01596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.957+0000 7f0adfe00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae00d0250 0x7f0ae01596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.957+0000 7f0adfe00640 1 -- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae00cfe70 msgr2=0x7f0ae0156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.957+0000 7f0adfe00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae00cfe70 0x7f0ae0156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:49.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.957+0000 7f0adfe00640 1 -- 172.21.15.102:0/1734286192 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae01f4040 con 0x7f0ae00cc2f0 2024-03-18T16:36:49.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.960+0000 7f0adfe00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cc2f0 0x7f0ae01f3160 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0033000 tx=0x7f0ad00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:49.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.967+0000 7f0ad7e00640 1 -- 172.21.15.102:0/1734286192 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0034040 con 0x7f0ae00cc2f0 2024-03-18T16:36:49.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.967+0000 7f0ad7e00640 1 -- 172.21.15.102:0/1734286192 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad0030410 con 0x7f0ae00cc2f0 2024-03-18T16:36:49.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.967+0000 7f0ad7e00640 1 -- 172.21.15.102:0/1734286192 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0039c70 con 0x7f0ae00cc2f0 2024-03-18T16:36:49.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.967+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae01f37a0 con 0x7f0ae00cc2f0 2024-03-18T16:36:49.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.967+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae01f3d60 con 0x7f0ae00cc2f0 2024-03-18T16:36:49.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:49.969+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae0074fd0 con 0x7f0ae00cc2f0 2024-03-18T16:36:50.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.253+0000 7f0ad7e00640 1 -- 172.21.15.102:0/1734286192 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0ad0037070 con 0x7f0ae00cc2f0 2024-03-18T16:36:50.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.255+0000 7f0ad7e00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ac006bc20 0x7f0ac006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:50.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.255+0000 7f0adf400640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ac006bc20 0x7f0ac006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:50.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.256+0000 7f0adf400640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ac006bc20 0x7f0ac006e0e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0159fd0 tx=0x7f0acc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:50.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.270+0000 7f0ad7e00640 1 -- 172.21.15.102:0/1734286192 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f0ad00305c0 con 0x7f0ae00cc2f0 2024-03-18T16:36:50.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.271+0000 7f0ad7e00640 1 -- 172.21.15.102:0/1734286192 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0ad007b800 con 0x7f0ae00cc2f0 2024-03-18T16:36:50.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.465+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ae00c8240 con 0x7f0ac006bc20 2024-03-18T16:36:50.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.473+0000 7f0ad7e00640 1 -- 172.21.15.102:0/1734286192 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332703 (secure 0 0 0) 0x7f0ae00c8240 con 0x7f0ac006bc20 2024-03-18T16:36:50.479 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:50.481 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:36:50.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.478+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ac006bc20 msgr2=0x7f0ac006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:50.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.478+0000 7f0ae5c00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ac006bc20 0x7f0ac006e0e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0159fd0 tx=0x7f0acc009190 comp rx=0 tx=0).stop 2024-03-18T16:36:50.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.478+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cc2f0 msgr2=0x7f0ae01f3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:50.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.478+0000 7f0ae5c00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cc2f0 0x7f0ae01f3160 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0033000 tx=0x7f0ad00046c0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.479+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 shutdown_connections 2024-03-18T16:36:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.479+0000 7f0ae5c00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f0ac006bc20 0x7f0ac006e0e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.479+0000 7f0ae5c00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0ae00d0250 0x7f0ae01596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.479+0000 7f0ae5c00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ae00cfe70 0x7f0ae0156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.479+0000 7f0ae5c00640 1 --2- 172.21.15.102:0/1734286192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ae00cc2f0 0x7f0ae01f3160 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.479+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 >> 172.21.15.102:0/1734286192 conn(0x7f0ae008c3e0 msgr2=0x7f0ae00c3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.479+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 shutdown_connections 2024-03-18T16:36:50.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.479+0000 7f0ae5c00640 1 -- 172.21.15.102:0/1734286192 wait complete. 2024-03-18T16:36:50.499 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-03-18T16:36:49.034167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":306,"ondisk_log_size":306,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219040,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755680,"statfs":{"total":773094113280,"available":772869816320,"internally_reserved":0,"allocated":2949120,"data_stored":2294136,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002813"},"pg_stats":[{"pgid":"2.1f","version":"29'13","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204119+0000","last_change":"2024-03-18T16:21:21.402212+0000","last_active":"2024-03-18T16:33:52.204119+0000","last_peered":"2024-03-18T16:33:52.204119+0000","last_clean":"2024-03-18T16:33:52.204119+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:33:52.204119+0000","last_undegraded":"2024-03-18T16:33:52.204119+0000","last_fullsized":"2024-03-18T16:33:52.204119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:46:33.215915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.163586+0000","last_change":"2024-03-18T16:21:23.045605+0000","last_active":"2024-03-18T16:33:22.163586+0000","last_peered":"2024-03-18T16:33:22.163586+0000","last_clean":"2024-03-18T16:33:22.163586+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:33:22.163586+0000","last_undegraded":"2024-03-18T16:33:22.163586+0000","last_fullsized":"2024-03-18T16:33:22.163586+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:37.165226+0000","last_change":"2024-03-18T16:21:29.146958+0000","last_active":"2024-03-18T16:33:37.165226+0000","last_peered":"2024-03-18T16:33:37.165226+0000","last_clean":"2024-03-18T16:33:37.165226+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:33:37.165226+0000","last_undegraded":"2024-03-18T16:33:37.165226+0000","last_fullsized":"2024-03-18T16:33:37.165226+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:52:55.871927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.163616+0000","last_change":"2024-03-18T16:21:21.413060+0000","last_active":"2024-03-18T16:33:22.163616+0000","last_peered":"2024-03-18T16:33:22.163616+0000","last_clean":"2024-03-18T16:33:22.163616+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:33:22.163616+0000","last_undegraded":"2024-03-18T16:33:22.163616+0000","last_fullsized":"2024-03-18T16:33:22.163616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:40.114937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204105+0000","last_change":"2024-03-18T16:21:23.050849+0000","last_active":"2024-03-18T16:33:52.204105+0000","last_peered":"2024-03-18T16:33:52.204105+0000","last_clean":"2024-03-18T16:33:52.204105+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:33:52.204105+0000","last_undegraded":"2024-03-18T16:33:52.204105+0000","last_fullsized":"2024-03-18T16:33:52.204105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:32.961742+0000","last_change":"2024-03-18T16:21:29.065755+0000","last_active":"2024-03-18T16:33:32.961742+0000","last_peered":"2024-03-18T16:33:32.961742+0000","last_clean":"2024-03-18T16:33:32.961742+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:33:32.961742+0000","last_undegraded":"2024-03-18T16:33:32.961742+0000","last_fullsized":"2024-03-18T16:33:32.961742+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:45.934586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'12","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092522+0000","last_change":"2024-03-18T16:21:21.398188+0000","last_active":"2024-03-18T16:33:52.092522+0000","last_peered":"2024-03-18T16:33:52.092522+0000","last_clean":"2024-03-18T16:33:52.092522+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:33:52.092522+0000","last_undegraded":"2024-03-18T16:33:52.092522+0000","last_fullsized":"2024-03-18T16:33:52.092522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481833+0000","last_change":"2024-03-18T16:21:22.946288+0000","last_active":"2024-03-18T16:33:54.481833+0000","last_peered":"2024-03-18T16:33:54.481833+0000","last_clean":"2024-03-18T16:33:54.481833+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:33:54.481833+0000","last_undegraded":"2024-03-18T16:33:54.481833+0000","last_fullsized":"2024-03-18T16:33:54.481833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:08.276881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:57.964061+0000","last_change":"2024-03-18T16:21:29.065891+0000","last_active":"2024-03-18T16:33:57.964061+0000","last_peered":"2024-03-18T16:33:57.964061+0000","last_clean":"2024-03-18T16:33:57.964061+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:33:57.964061+0000","last_undegraded":"2024-03-18T16:33:57.964061+0000","last_fullsized":"2024-03-18T16:33:57.964061+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092509+0000","last_change":"2024-03-18T16:21:21.397071+0000","last_active":"2024-03-18T16:33:52.092509+0000","last_peered":"2024-03-18T16:33:52.092509+0000","last_clean":"2024-03-18T16:33:52.092509+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:33:52.092509+0000","last_undegraded":"2024-03-18T16:33:52.092509+0000","last_fullsized":"2024-03-18T16:33:52.092509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481845+0000","last_change":"2024-03-18T16:21:22.946468+0000","last_active":"2024-03-18T16:33:54.481845+0000","last_peered":"2024-03-18T16:33:54.481845+0000","last_clean":"2024-03-18T16:33:54.481845+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:33:54.481845+0000","last_undegraded":"2024-03-18T16:33:54.481845+0000","last_fullsized":"2024-03-18T16:33:54.481845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:57.964049+0000","last_change":"2024-03-18T16:21:29.073724+0000","last_active":"2024-03-18T16:33:57.964049+0000","last_peered":"2024-03-18T16:33:57.964049+0000","last_clean":"2024-03-18T16:33:57.964049+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:33:57.964049+0000","last_undegraded":"2024-03-18T16:33:57.964049+0000","last_fullsized":"2024-03-18T16:33:57.964049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536515+0000","last_change":"2024-03-18T16:21:23.041926+0000","last_active":"2024-03-18T16:34:24.536515+0000","last_peered":"2024-03-18T16:34:24.536515+0000","last_clean":"2024-03-18T16:34:24.536515+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:34:24.536515+0000","last_undegraded":"2024-03-18T16:34:24.536515+0000","last_fullsized":"2024-03-18T16:34:24.536515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:03.532253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085707+0000","last_change":"2024-03-18T16:21:21.394801+0000","last_active":"2024-03-18T16:32:37.085707+0000","last_peered":"2024-03-18T16:32:37.085707+0000","last_clean":"2024-03-18T16:32:37.085707+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:32:37.085707+0000","last_undegraded":"2024-03-18T16:32:37.085707+0000","last_fullsized":"2024-03-18T16:32:37.085707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:42.957341+0000","last_change":"2024-03-18T16:21:29.073757+0000","last_active":"2024-03-18T16:32:42.957341+0000","last_peered":"2024-03-18T16:32:42.957341+0000","last_clean":"2024-03-18T16:32:42.957341+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:32:42.957341+0000","last_undegraded":"2024-03-18T16:32:42.957341+0000","last_fullsized":"2024-03-18T16:32:42.957341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.157732+0000","last_change":"2024-03-18T16:21:23.060776+0000","last_active":"2024-03-18T16:32:37.157732+0000","last_peered":"2024-03-18T16:32:37.157732+0000","last_clean":"2024-03-18T16:32:37.157732+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:32:37.157732+0000","last_undegraded":"2024-03-18T16:32:37.157732+0000","last_fullsized":"2024-03-18T16:32:37.157732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"30'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960340+0000","last_change":"2024-03-18T16:21:21.412733+0000","last_active":"2024-03-18T16:34:22.960340+0000","last_peered":"2024-03-18T16:34:22.960340+0000","last_clean":"2024-03-18T16:34:22.960340+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:34:22.960340+0000","last_undegraded":"2024-03-18T16:34:22.960340+0000","last_fullsized":"2024-03-18T16:34:22.960340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:42.957307+0000","last_change":"2024-03-18T16:21:29.063980+0000","last_active":"2024-03-18T16:32:42.957307+0000","last_peered":"2024-03-18T16:32:42.957307+0000","last_clean":"2024-03-18T16:32:42.957307+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:32:42.957307+0000","last_undegraded":"2024-03-18T16:32:42.957307+0000","last_fullsized":"2024-03-18T16:32:42.957307+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:39.469490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:45.657782+0000","last_change":"2024-03-18T16:21:54.838108+0000","last_active":"2024-03-18T16:32:45.657782+0000","last_peered":"2024-03-18T16:32:45.657782+0000","last_clean":"2024-03-18T16:32:45.657782+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:32:45.657782+0000","last_undegraded":"2024-03-18T16:32:45.657782+0000","last_fullsized":"2024-03-18T16:32:45.657782+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:11:40.182971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960309+0000","last_change":"2024-03-18T16:21:21.407132+0000","last_active":"2024-03-18T16:34:22.960309+0000","last_peered":"2024-03-18T16:34:22.960309+0000","last_clean":"2024-03-18T16:34:22.960309+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:34:22.960309+0000","last_undegraded":"2024-03-18T16:34:22.960309+0000","last_fullsized":"2024-03-18T16:34:22.960309+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:10.660930+0000","last_change":"2024-03-18T16:21:55.269661+0000","last_active":"2024-03-18T16:33:10.660930+0000","last_peered":"2024-03-18T16:33:10.660930+0000","last_clean":"2024-03-18T16:33:10.660930+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:33:10.660930+0000","last_undegraded":"2024-03-18T16:33:10.660930+0000","last_fullsized":"2024-03-18T16:33:10.660930+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:38.290937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.966337+0000","last_change":"2024-03-18T16:21:22.840551+0000","last_active":"2024-03-18T16:34:22.966337+0000","last_peered":"2024-03-18T16:34:22.966337+0000","last_clean":"2024-03-18T16:34:22.966337+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:34:22.966337+0000","last_undegraded":"2024-03-18T16:34:22.966337+0000","last_fullsized":"2024-03-18T16:34:22.966337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:17.136164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.095352+0000","last_change":"2024-03-18T16:21:21.400349+0000","last_active":"2024-03-18T16:34:22.095352+0000","last_peered":"2024-03-18T16:34:22.095352+0000","last_clean":"2024-03-18T16:34:22.095352+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:34:22.095352+0000","last_undegraded":"2024-03-18T16:34:22.095352+0000","last_fullsized":"2024-03-18T16:34:22.095352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:42.957357+0000","last_change":"2024-03-18T16:21:29.064624+0000","last_active":"2024-03-18T16:32:42.957357+0000","last_peered":"2024-03-18T16:32:42.957357+0000","last_clean":"2024-03-18T16:32:42.957357+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:32:42.957357+0000","last_undegraded":"2024-03-18T16:32:42.957357+0000","last_fullsized":"2024-03-18T16:32:42.957357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.207867+0000","last_change":"2024-03-18T16:21:23.055982+0000","last_active":"2024-03-18T16:34:22.207867+0000","last_peered":"2024-03-18T16:34:22.207867+0000","last_clean":"2024-03-18T16:34:22.207867+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:34:22.207867+0000","last_undegraded":"2024-03-18T16:34:22.207867+0000","last_fullsized":"2024-03-18T16:34:22.207867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536571+0000","last_change":"2024-03-18T16:21:21.400843+0000","last_active":"2024-03-18T16:34:24.536571+0000","last_peered":"2024-03-18T16:34:24.536571+0000","last_clean":"2024-03-18T16:34:24.536571+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:34:24.536571+0000","last_undegraded":"2024-03-18T16:34:24.536571+0000","last_fullsized":"2024-03-18T16:34:24.536571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:57.964028+0000","last_change":"2024-03-18T16:21:29.140888+0000","last_active":"2024-03-18T16:33:57.964028+0000","last_peered":"2024-03-18T16:33:57.964028+0000","last_clean":"2024-03-18T16:33:57.964028+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:33:57.964028+0000","last_undegraded":"2024-03-18T16:33:57.964028+0000","last_fullsized":"2024-03-18T16:33:57.964028+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.171886+0000","last_change":"2024-03-18T16:21:22.994455+0000","last_active":"2024-03-18T16:34:22.171886+0000","last_peered":"2024-03-18T16:34:22.171886+0000","last_clean":"2024-03-18T16:34:22.171886+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:34:22.171886+0000","last_undegraded":"2024-03-18T16:34:22.171886+0000","last_fullsized":"2024-03-18T16:34:22.171886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960381+0000","last_change":"2024-03-18T16:21:21.402920+0000","last_active":"2024-03-18T16:34:22.960381+0000","last_peered":"2024-03-18T16:34:22.960381+0000","last_clean":"2024-03-18T16:34:22.960381+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:34:22.960381+0000","last_undegraded":"2024-03-18T16:34:22.960381+0000","last_fullsized":"2024-03-18T16:34:22.960381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:32.164696+0000","last_change":"2024-03-18T16:21:29.145235+0000","last_active":"2024-03-18T16:33:32.164696+0000","last_peered":"2024-03-18T16:33:32.164696+0000","last_clean":"2024-03-18T16:33:32.164696+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:33:32.164696+0000","last_undegraded":"2024-03-18T16:33:32.164696+0000","last_fullsized":"2024-03-18T16:33:32.164696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:55:42.379951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.957241+0000","last_change":"2024-03-18T16:21:22.975172+0000","last_active":"2024-03-18T16:33:52.957241+0000","last_peered":"2024-03-18T16:33:52.957241+0000","last_clean":"2024-03-18T16:33:52.957241+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:33:52.957241+0000","last_undegraded":"2024-03-18T16:33:52.957241+0000","last_fullsized":"2024-03-18T16:33:52.957241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092531+0000","last_change":"2024-03-18T16:21:21.399688+0000","last_active":"2024-03-18T16:33:52.092531+0000","last_peered":"2024-03-18T16:33:52.092531+0000","last_clean":"2024-03-18T16:33:52.092531+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:33:52.092531+0000","last_undegraded":"2024-03-18T16:33:52.092531+0000","last_fullsized":"2024-03-18T16:33:52.092531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:11.933810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.205097+0000","last_change":"2024-03-18T16:21:29.141555+0000","last_active":"2024-03-18T16:34:02.205097+0000","last_peered":"2024-03-18T16:34:02.205097+0000","last_clean":"2024-03-18T16:34:02.205097+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:34:02.205097+0000","last_undegraded":"2024-03-18T16:34:02.205097+0000","last_fullsized":"2024-03-18T16:34:02.205097+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481851+0000","last_change":"2024-03-18T16:21:22.947980+0000","last_active":"2024-03-18T16:33:54.481851+0000","last_peered":"2024-03-18T16:33:54.481851+0000","last_clean":"2024-03-18T16:33:54.481851+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:33:54.481851+0000","last_undegraded":"2024-03-18T16:33:54.481851+0000","last_fullsized":"2024-03-18T16:33:54.481851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:53:31.739693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"30'2","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.957271+0000","last_change":"2024-03-18T16:21:21.400087+0000","last_active":"2024-03-18T16:33:52.957271+0000","last_peered":"2024-03-18T16:33:52.957271+0000","last_clean":"2024-03-18T16:33:52.957271+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:33:52.957271+0000","last_undegraded":"2024-03-18T16:33:52.957271+0000","last_fullsized":"2024-03-18T16:33:52.957271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.958278+0000","last_change":"2024-03-18T16:21:29.076629+0000","last_active":"2024-03-18T16:34:02.958278+0000","last_peered":"2024-03-18T16:34:02.958278+0000","last_clean":"2024-03-18T16:34:02.958278+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:34:02.958278+0000","last_undegraded":"2024-03-18T16:34:02.958278+0000","last_fullsized":"2024-03-18T16:34:02.958278+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.963620+0000","last_change":"2024-03-18T16:21:23.055810+0000","last_active":"2024-03-18T16:33:52.963620+0000","last_peered":"2024-03-18T16:33:52.963620+0000","last_clean":"2024-03-18T16:33:52.963620+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:33:52.963620+0000","last_undegraded":"2024-03-18T16:33:52.963620+0000","last_fullsized":"2024-03-18T16:33:52.963620+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:40.451255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481805+0000","last_change":"2024-03-18T16:21:21.399134+0000","last_active":"2024-03-18T16:33:54.481805+0000","last_peered":"2024-03-18T16:33:54.481805+0000","last_clean":"2024-03-18T16:33:54.481805+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:33:54.481805+0000","last_undegraded":"2024-03-18T16:33:54.481805+0000","last_fullsized":"2024-03-18T16:33:54.481805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.958311+0000","last_change":"2024-03-18T16:21:29.067567+0000","last_active":"2024-03-18T16:34:02.958311+0000","last_peered":"2024-03-18T16:34:02.958311+0000","last_clean":"2024-03-18T16:34:02.958311+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:34:02.958311+0000","last_undegraded":"2024-03-18T16:34:02.958311+0000","last_fullsized":"2024-03-18T16:34:02.958311+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.953865+0000","last_change":"2024-03-18T16:21:22.977089+0000","last_active":"2024-03-18T16:33:22.953865+0000","last_peered":"2024-03-18T16:33:22.953865+0000","last_clean":"2024-03-18T16:33:22.953865+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:33:22.953865+0000","last_undegraded":"2024-03-18T16:33:22.953865+0000","last_fullsized":"2024-03-18T16:33:22.953865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"29'5","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:50.664854+0000","last_change":"2024-03-18T16:21:21.408450+0000","last_active":"2024-03-18T16:33:50.664854+0000","last_peered":"2024-03-18T16:33:50.664854+0000","last_clean":"2024-03-18T16:33:50.664854+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:33:50.664854+0000","last_undegraded":"2024-03-18T16:33:50.664854+0000","last_fullsized":"2024-03-18T16:33:50.664854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:32.961622+0000","last_change":"2024-03-18T16:21:29.076875+0000","last_active":"2024-03-18T16:34:32.961622+0000","last_peered":"2024-03-18T16:34:32.961622+0000","last_clean":"2024-03-18T16:34:32.961622+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:34:32.961622+0000","last_undegraded":"2024-03-18T16:34:32.961622+0000","last_fullsized":"2024-03-18T16:34:32.961622+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204015+0000","last_change":"2024-03-18T16:21:23.054497+0000","last_active":"2024-03-18T16:33:52.204015+0000","last_peered":"2024-03-18T16:33:52.204015+0000","last_clean":"2024-03-18T16:33:52.204015+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:33:52.204015+0000","last_undegraded":"2024-03-18T16:33:52.204015+0000","last_fullsized":"2024-03-18T16:33:52.204015+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:37.455180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.089824+0000","last_change":"2024-03-18T16:21:21.404754+0000","last_active":"2024-03-18T16:33:22.089824+0000","last_peered":"2024-03-18T16:33:22.089824+0000","last_clean":"2024-03-18T16:33:22.089824+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:33:22.089824+0000","last_undegraded":"2024-03-18T16:33:22.089824+0000","last_fullsized":"2024-03-18T16:33:22.089824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:32.209201+0000","last_change":"2024-03-18T16:21:29.147316+0000","last_active":"2024-03-18T16:34:32.209201+0000","last_peered":"2024-03-18T16:34:32.209201+0000","last_clean":"2024-03-18T16:34:32.209201+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:34:32.209201+0000","last_undegraded":"2024-03-18T16:34:32.209201+0000","last_fullsized":"2024-03-18T16:34:32.209201+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:20.892947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.167476+0000","last_change":"2024-03-18T16:21:22.947635+0000","last_active":"2024-03-18T16:33:52.167476+0000","last_peered":"2024-03-18T16:33:52.167476+0000","last_clean":"2024-03-18T16:33:52.167476+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:33:52.167476+0000","last_undegraded":"2024-03-18T16:33:52.167476+0000","last_fullsized":"2024-03-18T16:33:52.167476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:12:40.724311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481823+0000","last_change":"2024-03-18T16:21:21.381306+0000","last_active":"2024-03-18T16:33:54.481823+0000","last_peered":"2024-03-18T16:33:54.481823+0000","last_clean":"2024-03-18T16:33:54.481823+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:33:54.481823+0000","last_undegraded":"2024-03-18T16:33:54.481823+0000","last_fullsized":"2024-03-18T16:33:54.481823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:07.088485+0000","last_change":"2024-03-18T16:21:08.662767+0000","last_active":"2024-03-18T16:33:07.088485+0000","last_peered":"2024-03-18T16:33:07.088485+0000","last_clean":"2024-03-18T16:33:07.088485+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:33:07.088485+0000","last_undegraded":"2024-03-18T16:33:07.088485+0000","last_fullsized":"2024-03-18T16:33:07.088485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_clean_scrub_stamp":"2024-03-18T16:21:05.847994+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:12:10.494707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:59.534564+0000","last_change":"2024-03-18T16:21:29.145697+0000","last_active":"2024-03-18T16:33:59.534564+0000","last_peered":"2024-03-18T16:33:59.534564+0000","last_clean":"2024-03-18T16:33:59.534564+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:33:59.534564+0000","last_undegraded":"2024-03-18T16:33:59.534564+0000","last_fullsized":"2024-03-18T16:33:59.534564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:46.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085732+0000","last_change":"2024-03-18T16:21:22.995291+0000","last_active":"2024-03-18T16:32:37.085732+0000","last_peered":"2024-03-18T16:32:37.085732+0000","last_clean":"2024-03-18T16:32:37.085732+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:32:37.085732+0000","last_undegraded":"2024-03-18T16:32:37.085732+0000","last_fullsized":"2024-03-18T16:32:37.085732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:32.918246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"30'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:35.656730+0000","last_change":"2024-03-18T16:21:21.382137+0000","last_active":"2024-03-18T16:32:35.656730+0000","last_peered":"2024-03-18T16:32:35.656730+0000","last_clean":"2024-03-18T16:32:35.656730+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:32:35.656730+0000","last_undegraded":"2024-03-18T16:32:35.656730+0000","last_fullsized":"2024-03-18T16:32:35.656730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:29.537048+0000","last_change":"2024-03-18T16:21:29.139432+0000","last_active":"2024-03-18T16:34:29.537048+0000","last_peered":"2024-03-18T16:34:29.537048+0000","last_clean":"2024-03-18T16:34:29.537048+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:34:29.537048+0000","last_undegraded":"2024-03-18T16:34:29.537048+0000","last_fullsized":"2024-03-18T16:34:29.537048+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:14.828125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:39.475154+0000","last_change":"2024-03-18T16:21:22.949578+0000","last_active":"2024-03-18T16:32:39.475154+0000","last_peered":"2024-03-18T16:32:39.475154+0000","last_clean":"2024-03-18T16:32:39.475154+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:32:39.475154+0000","last_undegraded":"2024-03-18T16:32:39.475154+0000","last_fullsized":"2024-03-18T16:32:39.475154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:36:20.997720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"30'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.948624+0000","last_change":"2024-03-18T16:21:21.400859+0000","last_active":"2024-03-18T16:32:37.948624+0000","last_peered":"2024-03-18T16:32:37.948624+0000","last_clean":"2024-03-18T16:32:37.948624+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:32:37.948624+0000","last_undegraded":"2024-03-18T16:32:37.948624+0000","last_fullsized":"2024-03-18T16:32:37.948624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:47.197062+0000","last_change":"2024-03-18T16:21:29.076813+0000","last_active":"2024-03-18T16:32:47.197062+0000","last_peered":"2024-03-18T16:32:47.197062+0000","last_clean":"2024-03-18T16:32:47.197062+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:32:47.197062+0000","last_undegraded":"2024-03-18T16:32:47.197062+0000","last_fullsized":"2024-03-18T16:32:47.197062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:50.227421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085756+0000","last_change":"2024-03-18T16:21:23.000009+0000","last_active":"2024-03-18T16:32:37.085756+0000","last_peered":"2024-03-18T16:32:37.085756+0000","last_clean":"2024-03-18T16:32:37.085756+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:32:37.085756+0000","last_undegraded":"2024-03-18T16:32:37.085756+0000","last_fullsized":"2024-03-18T16:32:37.085756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:39.383342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'6","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:35.656773+0000","last_change":"2024-03-18T16:21:21.409271+0000","last_active":"2024-03-18T16:32:35.656773+0000","last_peered":"2024-03-18T16:32:35.656773+0000","last_clean":"2024-03-18T16:32:35.656773+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:32:35.656773+0000","last_undegraded":"2024-03-18T16:32:35.656773+0000","last_fullsized":"2024-03-18T16:32:35.656773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:29.484964+0000","last_change":"2024-03-18T16:21:29.139595+0000","last_active":"2024-03-18T16:34:29.484964+0000","last_peered":"2024-03-18T16:34:29.484964+0000","last_clean":"2024-03-18T16:34:29.484964+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:34:29.484964+0000","last_undegraded":"2024-03-18T16:34:29.484964+0000","last_fullsized":"2024-03-18T16:34:29.484964+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:53:49.941645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085742+0000","last_change":"2024-03-18T16:21:23.000054+0000","last_active":"2024-03-18T16:32:37.085742+0000","last_peered":"2024-03-18T16:32:37.085742+0000","last_clean":"2024-03-18T16:32:37.085742+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:32:37.085742+0000","last_undegraded":"2024-03-18T16:32:37.085742+0000","last_fullsized":"2024-03-18T16:32:37.085742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:34:41.535042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"29'2","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.956808+0000","last_change":"2024-03-18T16:21:21.399315+0000","last_active":"2024-03-18T16:32:37.956808+0000","last_peered":"2024-03-18T16:32:37.956808+0000","last_clean":"2024-03-18T16:32:37.956808+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:32:37.956808+0000","last_undegraded":"2024-03-18T16:32:37.956808+0000","last_fullsized":"2024-03-18T16:32:37.956808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:57:29.543106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:27.966842+0000","last_change":"2024-03-18T16:21:29.065243+0000","last_active":"2024-03-18T16:34:27.966842+0000","last_peered":"2024-03-18T16:34:27.966842+0000","last_clean":"2024-03-18T16:34:27.966842+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:34:27.966842+0000","last_undegraded":"2024-03-18T16:34:27.966842+0000","last_fullsized":"2024-03-18T16:34:27.966842+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:27:24.187546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.087917+0000","last_change":"2024-03-18T16:21:22.996080+0000","last_active":"2024-03-18T16:33:02.087917+0000","last_peered":"2024-03-18T16:33:02.087917+0000","last_clean":"2024-03-18T16:33:02.087917+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:33:02.087917+0000","last_undegraded":"2024-03-18T16:33:02.087917+0000","last_fullsized":"2024-03-18T16:33:02.087917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'18","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:00.659847+0000","last_change":"2024-03-18T16:21:21.404930+0000","last_active":"2024-03-18T16:33:00.659847+0000","last_peered":"2024-03-18T16:33:00.659847+0000","last_clean":"2024-03-18T16:33:00.659847+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:33:00.659847+0000","last_undegraded":"2024-03-18T16:33:00.659847+0000","last_fullsized":"2024-03-18T16:33:00.659847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:12:02.571905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:37.202573+0000","last_change":"2024-03-18T16:21:29.079031+0000","last_active":"2024-03-18T16:33:37.202573+0000","last_peered":"2024-03-18T16:33:37.202573+0000","last_clean":"2024-03-18T16:33:37.202573+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:33:37.202573+0000","last_undegraded":"2024-03-18T16:33:37.202573+0000","last_fullsized":"2024-03-18T16:33:37.202573+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:20.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529622+0000","last_change":"2024-03-18T16:21:23.056832+0000","last_active":"2024-03-18T16:33:04.529622+0000","last_peered":"2024-03-18T16:33:04.529622+0000","last_clean":"2024-03-18T16:33:04.529622+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:33:04.529622+0000","last_undegraded":"2024-03-18T16:33:04.529622+0000","last_fullsized":"2024-03-18T16:33:04.529622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:56.124747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529644+0000","last_change":"2024-03-18T16:21:21.404392+0000","last_active":"2024-03-18T16:33:04.529644+0000","last_peered":"2024-03-18T16:33:04.529644+0000","last_clean":"2024-03-18T16:33:04.529644+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:33:04.529644+0000","last_undegraded":"2024-03-18T16:33:04.529644+0000","last_fullsized":"2024-03-18T16:33:04.529644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:37.165287+0000","last_change":"2024-03-18T16:21:29.139956+0000","last_active":"2024-03-18T16:33:37.165287+0000","last_peered":"2024-03-18T16:33:37.165287+0000","last_clean":"2024-03-18T16:33:37.165287+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:33:37.165287+0000","last_undegraded":"2024-03-18T16:33:37.165287+0000","last_fullsized":"2024-03-18T16:33:37.165287+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:52.356673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.087948+0000","last_change":"2024-03-18T16:21:22.995275+0000","last_active":"2024-03-18T16:33:02.087948+0000","last_peered":"2024-03-18T16:33:02.087948+0000","last_clean":"2024-03-18T16:33:02.087948+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:33:02.087948+0000","last_undegraded":"2024-03-18T16:33:02.087948+0000","last_fullsized":"2024-03-18T16:33:02.087948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:48.024985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.477130+0000","last_change":"2024-03-18T16:21:21.412044+0000","last_active":"2024-03-18T16:33:04.477130+0000","last_peered":"2024-03-18T16:33:04.477130+0000","last_clean":"2024-03-18T16:33:04.477130+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:33:04.477130+0000","last_undegraded":"2024-03-18T16:33:04.477130+0000","last_fullsized":"2024-03-18T16:33:04.477130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.162527+0000","last_change":"2024-03-18T16:21:55.270056+0000","last_active":"2024-03-18T16:33:12.162527+0000","last_peered":"2024-03-18T16:33:12.162527+0000","last_clean":"2024-03-18T16:33:12.162527+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:33:12.162527+0000","last_undegraded":"2024-03-18T16:33:12.162527+0000","last_fullsized":"2024-03-18T16:33:12.162527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:50.466826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.198598+0000","last_change":"2024-03-18T16:21:23.056116+0000","last_active":"2024-03-18T16:33:02.198598+0000","last_peered":"2024-03-18T16:33:02.198598+0000","last_clean":"2024-03-18T16:33:02.198598+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:33:02.198598+0000","last_undegraded":"2024-03-18T16:33:02.198598+0000","last_fullsized":"2024-03-18T16:33:02.198598+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:18.455315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"29'9","reported_seq":48,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.198623+0000","last_change":"2024-03-18T16:21:21.400378+0000","last_active":"2024-03-18T16:33:02.198623+0000","last_peered":"2024-03-18T16:33:02.198623+0000","last_clean":"2024-03-18T16:33:02.198623+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:33:02.198623+0000","last_undegraded":"2024-03-18T16:33:02.198623+0000","last_fullsized":"2024-03-18T16:33:02.198623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:34.479961+0000","last_change":"2024-03-18T16:21:29.141554+0000","last_active":"2024-03-18T16:33:34.479961+0000","last_peered":"2024-03-18T16:33:34.479961+0000","last_clean":"2024-03-18T16:33:34.479961+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:33:34.479961+0000","last_undegraded":"2024-03-18T16:33:34.479961+0000","last_fullsized":"2024-03-18T16:33:34.479961+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:25.856323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.090336+0000","last_change":"2024-03-18T16:21:22.996694+0000","last_active":"2024-03-18T16:33:27.090336+0000","last_peered":"2024-03-18T16:33:27.090336+0000","last_clean":"2024-03-18T16:33:27.090336+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:33:27.090336+0000","last_undegraded":"2024-03-18T16:33:27.090336+0000","last_fullsized":"2024-03-18T16:33:27.090336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:03.704090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529661+0000","last_change":"2024-03-18T16:21:21.412149+0000","last_active":"2024-03-18T16:33:04.529661+0000","last_peered":"2024-03-18T16:33:04.529661+0000","last_clean":"2024-03-18T16:33:04.529661+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:33:04.529661+0000","last_undegraded":"2024-03-18T16:33:04.529661+0000","last_fullsized":"2024-03-18T16:33:04.529661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.952815+0000","last_change":"2024-03-18T16:21:29.077350+0000","last_active":"2024-03-18T16:33:12.952815+0000","last_peered":"2024-03-18T16:33:12.952815+0000","last_clean":"2024-03-18T16:33:12.952815+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:33:12.952815+0000","last_undegraded":"2024-03-18T16:33:12.952815+0000","last_fullsized":"2024-03-18T16:33:12.952815+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:12.874189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.959141+0000","last_change":"2024-03-18T16:21:22.992928+0000","last_active":"2024-03-18T16:33:02.959141+0000","last_peered":"2024-03-18T16:33:02.959141+0000","last_clean":"2024-03-18T16:33:02.959141+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:33:02.959141+0000","last_undegraded":"2024-03-18T16:33:02.959141+0000","last_fullsized":"2024-03-18T16:33:02.959141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:54:50.978267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.954445+0000","last_change":"2024-03-18T16:21:21.403749+0000","last_active":"2024-03-18T16:33:27.954445+0000","last_peered":"2024-03-18T16:33:27.954445+0000","last_clean":"2024-03-18T16:33:27.954445+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:33:27.954445+0000","last_undegraded":"2024-03-18T16:33:27.954445+0000","last_fullsized":"2024-03-18T16:33:27.954445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:29:30.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:07.959644+0000","last_change":"2024-03-18T16:21:29.065378+0000","last_active":"2024-03-18T16:33:07.959644+0000","last_peered":"2024-03-18T16:33:07.959644+0000","last_clean":"2024-03-18T16:33:07.959644+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:33:07.959644+0000","last_undegraded":"2024-03-18T16:33:07.959644+0000","last_fullsized":"2024-03-18T16:33:07.959644+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:22:12.556720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.201187+0000","last_change":"2024-03-18T16:21:23.054168+0000","last_active":"2024-03-18T16:33:27.201187+0000","last_peered":"2024-03-18T16:33:27.201187+0000","last_clean":"2024-03-18T16:33:27.201187+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:33:27.201187+0000","last_undegraded":"2024-03-18T16:33:27.201187+0000","last_fullsized":"2024-03-18T16:33:27.201187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.479472+0000","last_change":"2024-03-18T16:21:21.398307+0000","last_active":"2024-03-18T16:33:29.479472+0000","last_peered":"2024-03-18T16:33:29.479472+0000","last_clean":"2024-03-18T16:33:29.479472+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:33:29.479472+0000","last_undegraded":"2024-03-18T16:33:29.479472+0000","last_fullsized":"2024-03-18T16:33:29.479472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.162463+0000","last_change":"2024-03-18T16:21:29.065855+0000","last_active":"2024-03-18T16:33:12.162463+0000","last_peered":"2024-03-18T16:33:12.162463+0000","last_clean":"2024-03-18T16:33:12.162463+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:33:12.162463+0000","last_undegraded":"2024-03-18T16:33:12.162463+0000","last_fullsized":"2024-03-18T16:33:12.162463+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:43.207302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.479456+0000","last_change":"2024-03-18T16:21:22.993606+0000","last_active":"2024-03-18T16:33:29.479456+0000","last_peered":"2024-03-18T16:33:29.479456+0000","last_clean":"2024-03-18T16:33:29.479456+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:33:29.479456+0000","last_undegraded":"2024-03-18T16:33:29.479456+0000","last_fullsized":"2024-03-18T16:33:29.479456+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:38:23.121609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531813+0000","last_change":"2024-03-18T16:21:21.399588+0000","last_active":"2024-03-18T16:33:29.531813+0000","last_peered":"2024-03-18T16:33:29.531813+0000","last_clean":"2024-03-18T16:33:29.531813+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:33:29.531813+0000","last_undegraded":"2024-03-18T16:33:29.531813+0000","last_fullsized":"2024-03-18T16:33:29.531813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.162498+0000","last_change":"2024-03-18T16:21:29.144832+0000","last_active":"2024-03-18T16:33:12.162498+0000","last_peered":"2024-03-18T16:33:12.162498+0000","last_clean":"2024-03-18T16:33:12.162498+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:33:12.162498+0000","last_undegraded":"2024-03-18T16:33:12.162498+0000","last_fullsized":"2024-03-18T16:33:12.162498+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:29:10.103662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:32.201959+0000","last_change":"2024-03-18T16:21:29.148215+0000","last_active":"2024-03-18T16:33:32.201959+0000","last_peered":"2024-03-18T16:33:32.201959+0000","last_clean":"2024-03-18T16:33:32.201959+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:33:32.201959+0000","last_undegraded":"2024-03-18T16:33:32.201959+0000","last_fullsized":"2024-03-18T16:33:32.201959+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:15:16.575572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.954392+0000","last_change":"2024-03-18T16:21:22.980300+0000","last_active":"2024-03-18T16:33:27.954392+0000","last_peered":"2024-03-18T16:33:27.954392+0000","last_clean":"2024-03-18T16:33:27.954392+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:33:27.954392+0000","last_undegraded":"2024-03-18T16:33:27.954392+0000","last_fullsized":"2024-03-18T16:33:27.954392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:04:15.839557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.479400+0000","last_change":"2024-03-18T16:21:21.329294+0000","last_active":"2024-03-18T16:33:29.479400+0000","last_peered":"2024-03-18T16:33:29.479400+0000","last_clean":"2024-03-18T16:33:29.479400+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:33:29.479400+0000","last_undegraded":"2024-03-18T16:33:29.479400+0000","last_fullsized":"2024-03-18T16:33:29.479400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:59.534588+0000","last_change":"2024-03-18T16:21:29.076706+0000","last_active":"2024-03-18T16:33:59.534588+0000","last_peered":"2024-03-18T16:33:59.534588+0000","last_clean":"2024-03-18T16:33:59.534588+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:33:59.534588+0000","last_undegraded":"2024-03-18T16:33:59.534588+0000","last_fullsized":"2024-03-18T16:33:59.534588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:57.880669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.164115+0000","last_change":"2024-03-18T16:21:23.052989+0000","last_active":"2024-03-18T16:33:27.164115+0000","last_peered":"2024-03-18T16:33:27.164115+0000","last_clean":"2024-03-18T16:33:27.164115+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:33:27.164115+0000","last_undegraded":"2024-03-18T16:33:27.164115+0000","last_fullsized":"2024-03-18T16:33:27.164115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:12.627158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.164154+0000","last_change":"2024-03-18T16:21:21.413020+0000","last_active":"2024-03-18T16:33:27.164154+0000","last_peered":"2024-03-18T16:33:27.164154+0000","last_clean":"2024-03-18T16:33:27.164154+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:33:27.164154+0000","last_undegraded":"2024-03-18T16:33:27.164154+0000","last_fullsized":"2024-03-18T16:33:27.164154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:06:49.237529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.534182+0000","last_change":"2024-03-18T16:21:21.397354+0000","last_active":"2024-03-18T16:33:54.534182+0000","last_peered":"2024-03-18T16:33:54.534182+0000","last_clean":"2024-03-18T16:33:54.534182+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:33:54.534182+0000","last_undegraded":"2024-03-18T16:33:54.534182+0000","last_fullsized":"2024-03-18T16:33:54.534182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204069+0000","last_change":"2024-03-18T16:21:23.059441+0000","last_active":"2024-03-18T16:33:52.204069+0000","last_peered":"2024-03-18T16:33:52.204069+0000","last_clean":"2024-03-18T16:33:52.204069+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:33:52.204069+0000","last_undegraded":"2024-03-18T16:33:52.204069+0000","last_fullsized":"2024-03-18T16:33:52.204069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:00.666357+0000","last_change":"2024-03-18T16:21:29.058800+0000","last_active":"2024-03-18T16:34:00.666357+0000","last_peered":"2024-03-18T16:34:00.666357+0000","last_clean":"2024-03-18T16:34:00.666357+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:34:00.666357+0000","last_undegraded":"2024-03-18T16:34:00.666357+0000","last_fullsized":"2024-03-18T16:34:00.666357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:20.279579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.167524+0000","last_change":"2024-03-18T16:21:21.412536+0000","last_active":"2024-03-18T16:33:52.167524+0000","last_peered":"2024-03-18T16:33:52.167524+0000","last_clean":"2024-03-18T16:33:52.167524+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:33:52.167524+0000","last_undegraded":"2024-03-18T16:33:52.167524+0000","last_fullsized":"2024-03-18T16:33:52.167524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:46.962976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.963585+0000","last_change":"2024-03-18T16:21:22.974963+0000","last_active":"2024-03-18T16:33:52.963585+0000","last_peered":"2024-03-18T16:33:52.963585+0000","last_clean":"2024-03-18T16:33:52.963585+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:33:52.963585+0000","last_undegraded":"2024-03-18T16:33:52.963585+0000","last_fullsized":"2024-03-18T16:33:52.963585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:51.820583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:02.169266+0000","last_change":"2024-03-18T16:21:55.151933+0000","last_active":"2024-03-18T16:34:02.169266+0000","last_peered":"2024-03-18T16:34:02.169266+0000","last_clean":"2024-03-18T16:34:02.169266+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:34:02.169266+0000","last_undegraded":"2024-03-18T16:34:02.169266+0000","last_fullsized":"2024-03-18T16:34:02.169266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:38.438575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.534141+0000","last_change":"2024-03-18T16:21:23.056857+0000","last_active":"2024-03-18T16:33:54.534141+0000","last_peered":"2024-03-18T16:33:54.534141+0000","last_clean":"2024-03-18T16:33:54.534141+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:33:54.534141+0000","last_undegraded":"2024-03-18T16:33:54.534141+0000","last_fullsized":"2024-03-18T16:33:54.534141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:13.728335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:55.665554+0000","last_change":"2024-03-18T16:21:22.949289+0000","last_active":"2024-03-18T16:33:55.665554+0000","last_peered":"2024-03-18T16:33:55.665554+0000","last_clean":"2024-03-18T16:33:55.665554+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:33:55.665554+0000","last_undegraded":"2024-03-18T16:33:55.665554+0000","last_fullsized":"2024-03-18T16:33:55.665554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:51:51.357680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:25.669716+0000","last_change":"2024-03-18T16:21:54.845858+0000","last_active":"2024-03-18T16:34:25.669716+0000","last_peered":"2024-03-18T16:34:25.669716+0000","last_clean":"2024-03-18T16:34:25.669716+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:34:25.669716+0000","last_undegraded":"2024-03-18T16:34:25.669716+0000","last_fullsized":"2024-03-18T16:34:25.669716+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204055+0000","last_change":"2024-03-18T16:21:23.059438+0000","last_active":"2024-03-18T16:33:52.204055+0000","last_peered":"2024-03-18T16:33:52.204055+0000","last_clean":"2024-03-18T16:33:52.204055+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:33:52.204055+0000","last_undegraded":"2024-03-18T16:33:52.204055+0000","last_fullsized":"2024-03-18T16:33:52.204055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.171829+0000","last_change":"2024-03-18T16:21:22.849115+0000","last_active":"2024-03-18T16:34:22.171829+0000","last_peered":"2024-03-18T16:34:22.171829+0000","last_clean":"2024-03-18T16:34:22.171829+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:34:22.171829+0000","last_undegraded":"2024-03-18T16:34:22.171829+0000","last_fullsized":"2024-03-18T16:34:22.171829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:06:57.469261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.960363+0000","last_change":"2024-03-18T16:21:22.980253+0000","last_active":"2024-03-18T16:34:22.960363+0000","last_peered":"2024-03-18T16:34:22.960363+0000","last_clean":"2024-03-18T16:34:22.960363+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:34:22.960363+0000","last_undegraded":"2024-03-18T16:34:22.960363+0000","last_fullsized":"2024-03-18T16:34:22.960363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:24:39.938963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536554+0000","last_change":"2024-03-18T16:21:23.042188+0000","last_active":"2024-03-18T16:34:24.536554+0000","last_peered":"2024-03-18T16:34:24.536554+0000","last_clean":"2024-03-18T16:34:24.536554+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:34:24.536554+0000","last_undegraded":"2024-03-18T16:34:24.536554+0000","last_fullsized":"2024-03-18T16:34:24.536554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.171870+0000","last_change":"2024-03-18T16:21:22.851677+0000","last_active":"2024-03-18T16:34:22.171870+0000","last_peered":"2024-03-18T16:34:22.171870+0000","last_clean":"2024-03-18T16:34:22.171870+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:34:22.171870+0000","last_undegraded":"2024-03-18T16:34:22.171870+0000","last_fullsized":"2024-03-18T16:34:22.171870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.484545+0000","last_change":"2024-03-18T16:21:22.947869+0000","last_active":"2024-03-18T16:34:24.484545+0000","last_peered":"2024-03-18T16:34:24.484545+0000","last_clean":"2024-03-18T16:34:24.484545+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:34:24.484545+0000","last_undegraded":"2024-03-18T16:34:24.484545+0000","last_fullsized":"2024-03-18T16:34:24.484545+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.157693+0000","last_change":"2024-03-18T16:21:22.851663+0000","last_active":"2024-03-18T16:32:37.157693+0000","last_peered":"2024-03-18T16:32:37.157693+0000","last_clean":"2024-03-18T16:32:37.157693+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:32:37.157693+0000","last_undegraded":"2024-03-18T16:32:37.157693+0000","last_fullsized":"2024-03-18T16:32:37.157693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:10.660891+0000","last_change":"2024-03-18T16:21:55.268883+0000","last_active":"2024-03-18T16:33:10.660891+0000","last_peered":"2024-03-18T16:33:10.660891+0000","last_clean":"2024-03-18T16:33:10.660891+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:33:10.660891+0000","last_undegraded":"2024-03-18T16:33:10.660891+0000","last_fullsized":"2024-03-18T16:33:10.660891+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:54:31.903287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.196040+0000","last_change":"2024-03-18T16:21:23.059733+0000","last_active":"2024-03-18T16:32:37.196040+0000","last_peered":"2024-03-18T16:32:37.196040+0000","last_clean":"2024-03-18T16:32:37.196040+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:32:37.196040+0000","last_undegraded":"2024-03-18T16:32:37.196040+0000","last_fullsized":"2024-03-18T16:32:37.196040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:39.527349+0000","last_change":"2024-03-18T16:21:23.042457+0000","last_active":"2024-03-18T16:32:39.527349+0000","last_peered":"2024-03-18T16:32:39.527349+0000","last_clean":"2024-03-18T16:32:39.527349+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:32:39.527349+0000","last_undegraded":"2024-03-18T16:32:39.527349+0000","last_fullsized":"2024-03-18T16:32:39.527349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:09.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:12.952871+0000","last_change":"2024-03-18T16:21:54.842423+0000","last_active":"2024-03-18T16:33:12.952871+0000","last_peered":"2024-03-18T16:33:12.952871+0000","last_clean":"2024-03-18T16:33:12.952871+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:33:12.952871+0000","last_undegraded":"2024-03-18T16:33:12.952871+0000","last_fullsized":"2024-03-18T16:33:12.952871+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:01:36.656058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:05.660378+0000","last_change":"2024-03-18T16:21:23.045061+0000","last_active":"2024-03-18T16:33:05.660378+0000","last_peered":"2024-03-18T16:33:05.660378+0000","last_clean":"2024-03-18T16:33:05.660378+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:33:05.660378+0000","last_undegraded":"2024-03-18T16:33:05.660378+0000","last_fullsized":"2024-03-18T16:33:05.660378+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:54.119675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.477157+0000","last_change":"2024-03-18T16:21:22.993632+0000","last_active":"2024-03-18T16:33:04.477157+0000","last_peered":"2024-03-18T16:33:04.477157+0000","last_clean":"2024-03-18T16:33:04.477157+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:33:04.477157+0000","last_undegraded":"2024-03-18T16:33:04.477157+0000","last_fullsized":"2024-03-18T16:33:04.477157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:35:42.249037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.477106+0000","last_change":"2024-03-18T16:21:23.056575+0000","last_active":"2024-03-18T16:33:04.477106+0000","last_peered":"2024-03-18T16:33:04.477106+0000","last_clean":"2024-03-18T16:33:04.477106+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:33:04.477106+0000","last_undegraded":"2024-03-18T16:33:04.477106+0000","last_fullsized":"2024-03-18T16:33:04.477106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:42.689306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":35,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.951745+0000","last_change":"2024-03-18T16:21:22.973283+0000","last_active":"2024-03-18T16:33:02.951745+0000","last_peered":"2024-03-18T16:33:02.951745+0000","last_clean":"2024-03-18T16:33:02.951745+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:33:02.951745+0000","last_undegraded":"2024-03-18T16:33:02.951745+0000","last_fullsized":"2024-03-18T16:33:02.951745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.198570+0000","last_change":"2024-03-18T16:21:23.057150+0000","last_active":"2024-03-18T16:33:02.198570+0000","last_peered":"2024-03-18T16:33:02.198570+0000","last_clean":"2024-03-18T16:33:02.198570+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:33:02.198570+0000","last_undegraded":"2024-03-18T16:33:02.198570+0000","last_fullsized":"2024-03-18T16:33:02.198570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:35:03.408047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.164176+0000","last_change":"2024-03-18T16:21:23.047968+0000","last_active":"2024-03-18T16:33:27.164176+0000","last_peered":"2024-03-18T16:33:27.164176+0000","last_clean":"2024-03-18T16:33:27.164176+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:33:27.164176+0000","last_undegraded":"2024-03-18T16:33:27.164176+0000","last_fullsized":"2024-03-18T16:33:27.164176+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.961347+0000","last_change":"2024-03-18T16:21:23.041794+0000","last_active":"2024-03-18T16:33:27.961347+0000","last_peered":"2024-03-18T16:33:27.961347+0000","last_clean":"2024-03-18T16:33:27.961347+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:33:27.961347+0000","last_undegraded":"2024-03-18T16:33:27.961347+0000","last_fullsized":"2024-03-18T16:33:27.961347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:13.065494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531782+0000","last_change":"2024-03-18T16:21:23.036940+0000","last_active":"2024-03-18T16:33:29.531782+0000","last_peered":"2024-03-18T16:33:29.531782+0000","last_clean":"2024-03-18T16:33:29.531782+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:33:29.531782+0000","last_undegraded":"2024-03-18T16:33:29.531782+0000","last_fullsized":"2024-03-18T16:33:29.531782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:45:38.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531802+0000","last_change":"2024-03-18T16:21:23.042140+0000","last_active":"2024-03-18T16:33:29.531802+0000","last_peered":"2024-03-18T16:33:29.531802+0000","last_clean":"2024-03-18T16:33:29.531802+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:33:29.531802+0000","last_undegraded":"2024-03-18T16:33:29.531802+0000","last_fullsized":"2024-03-18T16:33:29.531802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.090376+0000","last_change":"2024-03-18T16:21:22.995117+0000","last_active":"2024-03-18T16:33:27.090376+0000","last_peered":"2024-03-18T16:33:27.090376+0000","last_clean":"2024-03-18T16:33:27.090376+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:33:27.090376+0000","last_undegraded":"2024-03-18T16:33:27.090376+0000","last_fullsized":"2024-03-18T16:33:27.090376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:57.544738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.167390+0000","last_change":"2024-03-18T16:21:22.948123+0000","last_active":"2024-03-18T16:33:52.167390+0000","last_peered":"2024-03-18T16:33:52.167390+0000","last_clean":"2024-03-18T16:33:52.167390+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:33:52.167390+0000","last_undegraded":"2024-03-18T16:33:52.167390+0000","last_fullsized":"2024-03-18T16:33:52.167390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:59:15.173519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092477+0000","last_change":"2024-03-18T16:21:23.049877+0000","last_active":"2024-03-18T16:33:52.092477+0000","last_peered":"2024-03-18T16:33:52.092477+0000","last_clean":"2024-03-18T16:33:52.092477+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:33:52.092477+0000","last_undegraded":"2024-03-18T16:33:52.092477+0000","last_fullsized":"2024-03-18T16:33:52.092477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.481775+0000","last_change":"2024-03-18T16:21:23.049483+0000","last_active":"2024-03-18T16:33:54.481775+0000","last_peered":"2024-03-18T16:33:54.481775+0000","last_clean":"2024-03-18T16:33:54.481775+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:33:54.481775+0000","last_undegraded":"2024-03-18T16:33:54.481775+0000","last_fullsized":"2024-03-18T16:33:54.481775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092498+0000","last_change":"2024-03-18T16:21:22.995726+0000","last_active":"2024-03-18T16:33:52.092498+0000","last_peered":"2024-03-18T16:33:52.092498+0000","last_clean":"2024-03-18T16:33:52.092498+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:33:52.092498+0000","last_undegraded":"2024-03-18T16:33:52.092498+0000","last_fullsized":"2024-03-18T16:33:52.092498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:24.478835+0000","last_change":"2024-03-18T16:21:22.949640+0000","last_active":"2024-03-18T16:33:24.478835+0000","last_peered":"2024-03-18T16:33:24.478835+0000","last_clean":"2024-03-18T16:33:24.478835+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:33:24.478835+0000","last_undegraded":"2024-03-18T16:33:24.478835+0000","last_fullsized":"2024-03-18T16:33:24.478835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:06:21.734293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.204083+0000","last_change":"2024-03-18T16:21:23.050203+0000","last_active":"2024-03-18T16:33:52.204083+0000","last_peered":"2024-03-18T16:33:52.204083+0000","last_clean":"2024-03-18T16:33:52.204083+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:33:52.204083+0000","last_undegraded":"2024-03-18T16:33:52.204083+0000","last_fullsized":"2024-03-18T16:33:52.204083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:27:21.982408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.957297+0000","last_change":"2024-03-18T16:21:23.053003+0000","last_active":"2024-03-18T16:33:52.957297+0000","last_peered":"2024-03-18T16:33:52.957297+0000","last_clean":"2024-03-18T16:33:52.957297+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:33:52.957297+0000","last_undegraded":"2024-03-18T16:33:52.957297+0000","last_fullsized":"2024-03-18T16:33:52.957297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:40.716522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092540+0000","last_change":"2024-03-18T16:21:23.041863+0000","last_active":"2024-03-18T16:33:52.092540+0000","last_peered":"2024-03-18T16:33:52.092540+0000","last_clean":"2024-03-18T16:33:52.092540+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:33:52.092540+0000","last_undegraded":"2024-03-18T16:33:52.092540+0000","last_fullsized":"2024-03-18T16:33:52.092540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:26.190759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509633,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":421888,"data_stored":729077,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509634,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":344064,"data_stored":137939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509632,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":141275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509634,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":138141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509632,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":413696,"data_stored":727851,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542336,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":138071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":141241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575040,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":140541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:36:50.504 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:36:50.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.712+0000 7f5f2dc00640 1 Processor -- start 2024-03-18T16:36:50.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.712+0000 7f5f2dc00640 1 -- start start 2024-03-18T16:36:50.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.713+0000 7f5f2dc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5f28158420 0x7f5f2815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:50.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.713+0000 7f5f2dc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f280c5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:50.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.713+0000 7f5f2dc00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f280c72d0 con 0x7f5f28141650 2024-03-18T16:36:50.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.713+0000 7f5f2dc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f5f2815c8e0 con 0x7f5f28158420 2024-03-18T16:36:50.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.713+0000 7f5f2dc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5f28159370 con 0x7f5f280c4fa0 2024-03-18T16:36:50.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.714+0000 7f5f2d200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f280c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:50.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.714+0000 7f5f27e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f5f28141650 0x7f5f280c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:58680/0 (socket says 172.21.15.102:58680) 2024-03-18T16:36:50.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.714+0000 7f5f27e00640 1 -- 172.21.15.102:0/1377646586 learned_addr learned my addr 172.21.15.102:0/1377646586 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:50.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.714+0000 7f5f27400640 1 --2- 172.21.15.102:0/1377646586 >> v2:172.21.15.102:3300/0 conn(0x7f5f28158420 0x7f5f2815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.717+0000 7f5f2d200640 1 -- 172.21.15.102:0/1377646586 >> v2:172.21.15.102:3300/0 conn(0x7f5f28158420 msgr2=0x7f5f2815c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.717+0000 7f5f2d200640 1 --2- 172.21.15.102:0/1377646586 >> v2:172.21.15.102:3300/0 conn(0x7f5f28158420 0x7f5f2815c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.717+0000 7f5f2d200640 1 -- 172.21.15.102:0/1377646586 >> v1:172.21.15.154:6789/0 conn(0x7f5f28141650 legacy=0x7f5f280c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:50.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.717+0000 7f5f2d200640 1 -- 172.21.15.102:0/1377646586 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f28158840 con 0x7f5f280c4fa0 2024-03-18T16:36:50.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.719+0000 7f5f2d200640 1 --2- 172.21.15.102:0/1377646586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f280c5430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009d90 tx=0x7f5f2002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=254e25e045f93c50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:50.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.723+0000 7f5f26a00640 1 -- 172.21.15.102:0/1377646586 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f20039070 con 0x7f5f280c4fa0 2024-03-18T16:36:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.723+0000 7f5f26a00640 1 -- 172.21.15.102:0/1377646586 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2002cc80 con 0x7f5f280c4fa0 2024-03-18T16:36:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.723+0000 7f5f26a00640 1 -- 172.21.15.102:0/1377646586 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f20034660 con 0x7f5f280c4fa0 2024-03-18T16:36:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.724+0000 7f5f2dc00640 1 -- 172.21.15.102:0/1377646586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 msgr2=0x7f5f280c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.724+0000 7f5f2dc00640 1 --2- 172.21.15.102:0/1377646586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f280c5430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5f20009d90 tx=0x7f5f2002bed0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.724+0000 7f5f2dc00640 1 -- 172.21.15.102:0/1377646586 shutdown_connections 2024-03-18T16:36:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.724+0000 7f5f2dc00640 1 --2- 172.21.15.102:0/1377646586 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f280c5430 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.724+0000 7f5f2dc00640 1 --2- 172.21.15.102:0/1377646586 >> v2:172.21.15.102:3300/0 conn(0x7f5f28158420 0x7f5f2815c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.724+0000 7f5f2dc00640 1 -- 172.21.15.102:0/1377646586 >> 172.21.15.102:0/1377646586 conn(0x7f5f280cb890 msgr2=0x7f5f280cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.724+0000 7f5f2dc00640 1 -- 172.21.15.102:0/1377646586 shutdown_connections 2024-03-18T16:36:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.724+0000 7f5f2dc00640 1 -- 172.21.15.102:0/1377646586 wait complete. 2024-03-18T16:36:50.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.726+0000 7f5f2dc00640 1 Processor -- start 2024-03-18T16:36:50.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.726+0000 7f5f2dc00640 1 -- start start 2024-03-18T16:36:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.727+0000 7f5f2dc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f28159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.727+0000 7f5f2dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f28141650 0x7f5f28154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.727+0000 7f5f2dc00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f28158420 0x7f5f280cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.727+0000 7f5f2dc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f5f28159090 con 0x7f5f28141650 2024-03-18T16:36:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.727+0000 7f5f2dc00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f28158db0 con 0x7f5f28158420 2024-03-18T16:36:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.727+0000 7f5f2dc00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f5f280c63c0 con 0x7f5f280c4fa0 2024-03-18T16:36:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.728+0000 7f5f27e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f28159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.728+0000 7f5f27e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f28159580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:33548/0 (socket says 172.21.15.102:33548) 2024-03-18T16:36:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.728+0000 7f5f27e00640 1 -- 172.21.15.102:0/2043944053 learned_addr learned my addr 172.21.15.102:0/2043944053 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.728+0000 7f5f2d200640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f28158420 0x7f5f280cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.728+0000 7f5f27400640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f28141650 0x7f5f28154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:50.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.731+0000 7f5f27e00640 1 -- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f28158420 msgr2=0x7f5f280cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:50.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.731+0000 7f5f27e00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f28158420 0x7f5f280cbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.731+0000 7f5f27e00640 1 -- 172.21.15.102:0/2043944053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f28141650 msgr2=0x7f5f28154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:50.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.731+0000 7f5f27e00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f28141650 0x7f5f28154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:50.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.731+0000 7f5f27e00640 1 -- 172.21.15.102:0/2043944053 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f281f0eb0 con 0x7f5f280c4fa0 2024-03-18T16:36:50.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.734+0000 7f5f27e00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f28159580 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800c430 tx=0x7f5f1800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:50.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.738+0000 7f5f1fe00640 1 -- 172.21.15.102:0/2043944053 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f18017070 con 0x7f5f280c4fa0 2024-03-18T16:36:50.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.738+0000 7f5f1fe00640 1 -- 172.21.15.102:0/2043944053 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f18005ce0 con 0x7f5f280c4fa0 2024-03-18T16:36:50.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.738+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f281e5080 con 0x7f5f280c4fa0 2024-03-18T16:36:50.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.738+0000 7f5f1fe00640 1 -- 172.21.15.102:0/2043944053 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f180072f0 con 0x7f5f280c4fa0 2024-03-18T16:36:50.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.738+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f2815b880 con 0x7f5f280c4fa0 2024-03-18T16:36:50.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.739+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f28074fd0 con 0x7f5f280c4fa0 2024-03-18T16:36:50.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.994+0000 7f5f1fe00640 1 -- 172.21.15.102:0/2043944053 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5f1801a070 con 0x7f5f280c4fa0 2024-03-18T16:36:50.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.995+0000 7f5f1fe00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5f0806bc50 0x7f5f0806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:50.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.996+0000 7f5f27400640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5f0806bc50 0x7f5f0806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:50.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.996+0000 7f5f1fe00640 1 -- 172.21.15.102:0/2043944053 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f5f180567c0 con 0x7f5f280c4fa0 2024-03-18T16:36:50.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:50.997+0000 7f5f27400640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5f0806bc50 0x7f5f0806e110 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5f281e4d90 tx=0x7f5f14009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.007+0000 7f5f1fe00640 1 -- 172.21.15.102:0/2043944053 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5f18021020 con 0x7f5f280c4fa0 2024-03-18T16:36:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.226+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f280b7f00 con 0x7f5f280c4fa0 2024-03-18T16:36:51.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.271+0000 7f5f1fe00640 1 -- 172.21.15.102:0/2043944053 <== mon.2 v2:172.21.15.154:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+19038 (secure 0 0 0) 0x7f5f18058720 con 0x7f5f280c4fa0 2024-03-18T16:36:51.273 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:36:51.273 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":30,"fsid":"0179d728-7c99-4c29-b345-3950a2c4b164","created":"2024-03-18T16:20:46.385026+0000","modified":"2024-03-18T16:23:19.592489+0000","last_up_change":"2024-03-18T16:21:05.847994+0000","last_in_change":"2024-03-18T16:20:56.257571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:21:05.204661+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:21:19.524753+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:21:21.031843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:21:26.818690+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d7aad584-b031-4470-8d3b-fe2fd9bd7d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6809","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6811","nonce":3520461741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6813","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6815","nonce":3520461741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6821","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6823","nonce":3520461741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6817","nonce":3520461741},{"type":"v1","addr":"172.21.15.102:6819","nonce":3520461741}]},"public_addr":"172.21.15.102:6811/3520461741","cluster_addr":"172.21.15.102:6815/3520461741","heartbeat_back_addr":"172.21.15.102:6823/3520461741","heartbeat_front_addr":"172.21.15.102:6819/3520461741","state":["exists","up"]},{"osd":1,"uuid":"84532350-698b-486e-9dae-4975d277af0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6801","nonce":1867629855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6803","nonce":1867629855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6807","nonce":1867629855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":1867629855},{"type":"v1","addr":"172.21.15.102:6805","nonce":1867629855}]},"public_addr":"172.21.15.102:6801/1867629855","cluster_addr":"172.21.15.102:6803/1867629855","heartbeat_back_addr":"172.21.15.102:6807/1867629855","heartbeat_front_addr":"172.21.15.102:6805/1867629855","state":["exists","up"]},{"osd":2,"uuid":"a0705f8f-0b79-4084-b22e-3a4df2150f22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6825","nonce":2928547335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6827","nonce":2928547335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6831","nonce":2928547335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2928547335},{"type":"v1","addr":"172.21.15.102:6829","nonce":2928547335}]},"public_addr":"172.21.15.102:6825/2928547335","cluster_addr":"172.21.15.102:6827/2928547335","heartbeat_back_addr":"172.21.15.102:6831/2928547335","heartbeat_front_addr":"172.21.15.102:6829/2928547335","state":["exists","up"]},{"osd":3,"uuid":"014e4711-688e-4cc2-a8c7-85d8b6a9c5f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6810","nonce":1215178908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6814","nonce":1215178908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6822","nonce":1215178908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1215178908},{"type":"v1","addr":"172.21.15.102:6818","nonce":1215178908}]},"public_addr":"172.21.15.102:6810/1215178908","cluster_addr":"172.21.15.102:6814/1215178908","heartbeat_back_addr":"172.21.15.102:6822/1215178908","heartbeat_front_addr":"172.21.15.102:6818/1215178908","state":["exists","up"]},{"osd":4,"uuid":"67312bab-afbc-4612-9ab8-f0e5f907cb09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6827","nonce":2919562209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6829","nonce":2919562209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6833","nonce":2919562209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":2919562209},{"type":"v1","addr":"172.21.15.154:6831","nonce":2919562209}]},"public_addr":"172.21.15.154:6827/2919562209","cluster_addr":"172.21.15.154:6829/2919562209","heartbeat_back_addr":"172.21.15.154:6833/2919562209","heartbeat_front_addr":"172.21.15.154:6831/2919562209","state":["exists","up"]},{"osd":5,"uuid":"5e5e7bc5-c518-4f91-b6ce-3754524f5a56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6803","nonce":1669301852}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6805","nonce":1669301852}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6809","nonce":1669301852}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1669301852},{"type":"v1","addr":"172.21.15.154:6807","nonce":1669301852}]},"public_addr":"172.21.15.154:6803/1669301852","cluster_addr":"172.21.15.154:6805/1669301852","heartbeat_back_addr":"172.21.15.154:6809/1669301852","heartbeat_front_addr":"172.21.15.154:6807/1669301852","state":["exists","up"]},{"osd":6,"uuid":"fe2b01af-9db4-4759-a97f-628cafc8aaf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6811","nonce":1929522630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6813","nonce":1929522630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6817","nonce":1929522630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":1929522630},{"type":"v1","addr":"172.21.15.154:6815","nonce":1929522630}]},"public_addr":"172.21.15.154:6811/1929522630","cluster_addr":"172.21.15.154:6813/1929522630","heartbeat_back_addr":"172.21.15.154:6817/1929522630","heartbeat_front_addr":"172.21.15.154:6815/1929522630","state":["exists","up"]},{"osd":7,"uuid":"f9e0d662-1a25-4e47-a5c2-43ffb11478d8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6819","nonce":2630476699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6821","nonce":2630476699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6825","nonce":2630476699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":2630476699},{"type":"v1","addr":"172.21.15.154:6823","nonce":2630476699}]},"public_addr":"172.21.15.154:6819/2630476699","cluster_addr":"172.21.15.154:6821/2630476699","heartbeat_back_addr":"172.21.15.154:6825/2630476699","heartbeat_front_addr":"172.21.15.154:6823/2630476699","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.196718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.105052+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.966578+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.077198+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.063169+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.933115+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:01.981182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:21:02.061130+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.2d","mappings":[{"from":7,"to":1}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.14","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6834/3423054813":"2024-03-19T16:23:19.571305+0000","172.21.15.102:6832/3423054813":"2024-03-19T16:23:19.571305+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:36:51.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5f0806bc50 msgr2=0x7f5f0806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5f0806bc50 0x7f5f0806e110 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5f281e4d90 tx=0x7f5f14009190 comp rx=0 tx=0).stop 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 msgr2=0x7f5f28159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f28159580 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800c430 tx=0x7f5f1800c8f0 comp rx=0 tx=0).stop 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 shutdown_connections 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f5f0806bc50 0x7f5f0806e110 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f28158420 0x7f5f280cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f28141650 0x7f5f28154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 --2- 172.21.15.102:0/2043944053 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f5f280c4fa0 0x7f5f28159580 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 >> 172.21.15.102:0/2043944053 conn(0x7f5f280cb890 msgr2=0x7f5f28156da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 shutdown_connections 2024-03-18T16:36:51.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:51.275+0000 7f5f2dc00640 1 -- 172.21.15.102:0/2043944053 wait complete. 2024-03-18T16:36:52.296 INFO:tasks.ceph:Scrubbing osd.0 2024-03-18T16:36:52.296 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-18T16:36:52.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.480+0000 7f173a200640 1 Processor -- start 2024-03-18T16:36:52.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.480+0000 7f173a200640 1 -- start start 2024-03-18T16:36:52.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.482+0000 7f173a200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1734070fc0 0x7f173415e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.482+0000 7f173a200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f173415ae00 0x7f1734070480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.482+0000 7f173a200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f173406beb0 con 0x7f17341546e0 2024-03-18T16:36:52.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.482+0000 7f173a200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f17340c2610 con 0x7f1734070fc0 2024-03-18T16:36:52.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.482+0000 7f173a200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f17340cd470 con 0x7f173415ae00 2024-03-18T16:36:52.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.483+0000 7f1739800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f173415ae00 0x7f1734070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.483+0000 7f1733e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1734070fc0 0x7f173415e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.483+0000 7f1738e00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f17341546e0 0x7f17340c9310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:39020/0 (socket says 172.21.15.102:39020) 2024-03-18T16:36:52.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.483+0000 7f1738e00640 1 -- 172.21.15.102:0/3856774146 learned_addr learned my addr 172.21.15.102:0/3856774146 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:52.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.487+0000 7f1733e00640 1 -- 172.21.15.102:0/3856774146 >> v2:172.21.15.154:3300/0 conn(0x7f173415ae00 msgr2=0x7f1734070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:52.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.487+0000 7f1733e00640 1 --2- 172.21.15.102:0/3856774146 >> v2:172.21.15.154:3300/0 conn(0x7f173415ae00 0x7f1734070480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.487+0000 7f1733e00640 1 -- 172.21.15.102:0/3856774146 >> v1:172.21.15.154:6789/0 conn(0x7f17341546e0 legacy=0x7f17340c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.487+0000 7f1733e00640 1 -- 172.21.15.102:0/3856774146 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17340cceb0 con 0x7f1734070fc0 2024-03-18T16:36:52.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.490+0000 7f1733e00640 1 --2- 172.21.15.102:0/3856774146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1734070fc0 0x7f173415e5b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f17340b2f60 tx=0x7f172002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b3b39132d8d490 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:52.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.495+0000 7f1733400640 1 -- 172.21.15.102:0/3856774146 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17200315a0 con 0x7f1734070fc0 2024-03-18T16:36:52.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.495+0000 7f1733400640 1 -- 172.21.15.102:0/3856774146 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1720037070 con 0x7f1734070fc0 2024-03-18T16:36:52.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.495+0000 7f1733400640 1 -- 172.21.15.102:0/3856774146 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720031960 con 0x7f1734070fc0 2024-03-18T16:36:52.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.496+0000 7f173a200640 1 -- 172.21.15.102:0/3856774146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1734070fc0 msgr2=0x7f173415e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.496+0000 7f173a200640 1 --2- 172.21.15.102:0/3856774146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1734070fc0 0x7f173415e5b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f17340b2f60 tx=0x7f172002b840 comp rx=0 tx=0).stop 2024-03-18T16:36:52.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.496+0000 7f173a200640 1 -- 172.21.15.102:0/3856774146 shutdown_connections 2024-03-18T16:36:52.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.496+0000 7f173a200640 1 --2- 172.21.15.102:0/3856774146 >> v2:172.21.15.154:3300/0 conn(0x7f173415ae00 0x7f1734070480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.496+0000 7f173a200640 1 --2- 172.21.15.102:0/3856774146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1734070fc0 0x7f173415e5b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.496+0000 7f173a200640 1 -- 172.21.15.102:0/3856774146 >> 172.21.15.102:0/3856774146 conn(0x7f1734067160 msgr2=0x7f17340728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:52.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.496+0000 7f173a200640 1 -- 172.21.15.102:0/3856774146 shutdown_connections 2024-03-18T16:36:52.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.496+0000 7f173a200640 1 -- 172.21.15.102:0/3856774146 wait complete. 2024-03-18T16:36:52.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.499+0000 7f173a200640 1 Processor -- start 2024-03-18T16:36:52.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.499+0000 7f173a200640 1 -- start start 2024-03-18T16:36:52.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.500+0000 7f173a200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1734070fc0 0x7f17340c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.500+0000 7f173a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17341546e0 0x7f17341536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.500+0000 7f173a200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f173415ae00 0x7f17341e5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.500+0000 7f173a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f173415ddb0 con 0x7f17341546e0 2024-03-18T16:36:52.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.500+0000 7f173a200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f173415dad0 con 0x7f1734070fc0 2024-03-18T16:36:52.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.500+0000 7f173a200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f1734154ac0 con 0x7f173415ae00 2024-03-18T16:36:52.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.501+0000 7f1738e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1734070fc0 0x7f17340c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.501+0000 7f1739800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f173415ae00 0x7f17341e5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.501+0000 7f1733e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17341546e0 0x7f17341536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.501+0000 7f1739800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f173415ae00 0x7f17341e5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38092/0 (socket says 172.21.15.102:38092) 2024-03-18T16:36:52.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.502+0000 7f1739800640 1 -- 172.21.15.102:0/2002672097 learned_addr learned my addr 172.21.15.102:0/2002672097 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:52.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.505+0000 7f1739800640 1 -- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1734070fc0 msgr2=0x7f17340c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.505+0000 7f1739800640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1734070fc0 0x7f17340c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.506+0000 7f1739800640 1 -- 172.21.15.102:0/2002672097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17341546e0 msgr2=0x7f17341536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.506+0000 7f1739800640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17341546e0 0x7f17341536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.506+0000 7f1739800640 1 -- 172.21.15.102:0/2002672097 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17341e4d20 con 0x7f173415ae00 2024-03-18T16:36:52.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.508+0000 7f1739800640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f173415ae00 0x7f17341e5950 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f172c00efc0 tx=0x7f172c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:52.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.511+0000 7f1730c00640 1 -- 172.21.15.102:0/2002672097 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c018020 con 0x7f173415ae00 2024-03-18T16:36:52.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.511+0000 7f1730c00640 1 -- 172.21.15.102:0/2002672097 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f172c014480 con 0x7f173415ae00 2024-03-18T16:36:52.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.511+0000 7f1730c00640 1 -- 172.21.15.102:0/2002672097 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c014870 con 0x7f173415ae00 2024-03-18T16:36:52.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.511+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17341e5000 con 0x7f173415ae00 2024-03-18T16:36:52.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.512+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f17341e55c0 con 0x7f173415ae00 2024-03-18T16:36:52.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.513+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f173404f540 con 0x7f173415ae00 2024-03-18T16:36:52.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.534+0000 7f1730c00640 1 -- 172.21.15.102:0/2002672097 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f172c005d10 con 0x7f173415ae00 2024-03-18T16:36:52.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.535+0000 7f1730c00640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f171406bc20 0x7f171406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.536+0000 7f1738e00640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f171406bc20 0x7f171406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.536+0000 7f1730c00640 1 -- 172.21.15.102:0/2002672097 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f172c058640 con 0x7f173415ae00 2024-03-18T16:36:52.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.537+0000 7f1738e00640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f171406bc20 0x7f171406e0e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1728007fd0 tx=0x7f1728007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.537+0000 7f1730c00640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7f1714075740 0x7f1714077be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.537+0000 7f1730c00640 1 -- 172.21.15.102:0/2002672097 --> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17140782b0 con 0x7f1714075740 2024-03-18T16:36:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.537+0000 7f1730c00640 1 -- 172.21.15.102:0/2002672097 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f172c056840 con 0x7f173415ae00 2024-03-18T16:36:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.537+0000 7f1733e00640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7f1714075740 0x7f1714077be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.538+0000 7f1733e00640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7f1714075740 0x7f1714077be0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:52.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.539+0000 7f1730c00640 1 -- 172.21.15.102:0/2002672097 <== osd.0 v2:172.21.15.102:6809/3520461741 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f17140782b0 con 0x7f1714075740 2024-03-18T16:36:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.555+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 --> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f17340c76c0 con 0x7f1714075740 2024-03-18T16:36:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.556+0000 7f1730c00640 1 -- 172.21.15.102:0/2002672097 <== osd.0 v2:172.21.15.102:6809/3520461741 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f17340c76c0 con 0x7f1714075740 2024-03-18T16:36:52.557 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:36:52.557 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T16:36:52.558 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:36:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.556+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7f1714075740 msgr2=0x7f1714077be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.556+0000 7f173a200640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.102:6809/3520461741,v1:172.21.15.102:6811/3520461741] conn(0x7f1714075740 0x7f1714077be0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.557+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f171406bc20 msgr2=0x7f171406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.557+0000 7f173a200640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f171406bc20 0x7f171406e0e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1728007fd0 tx=0x7f1728007e80 comp rx=0 tx=0).stop 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.557+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f173415ae00 msgr2=0x7f17341e5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.557+0000 7f173a200640 1 --2- 172.21.15.102:0/2002672097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f173415ae00 0x7f17341e5950 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f172c00efc0 tx=0x7f172c00c490 comp rx=0 tx=0).stop 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.557+0000 7f1739800640 1 -- 172.21.15.102:0/2002672097 reap_dead start 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.557+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 shutdown_connections 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.557+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 >> 172.21.15.102:0/2002672097 conn(0x7f1734067160 msgr2=0x7f17340cbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.557+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 shutdown_connections 2024-03-18T16:36:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.557+0000 7f173a200640 1 -- 172.21.15.102:0/2002672097 wait complete. 2024-03-18T16:36:52.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-18T16:36:52.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.749+0000 7f7cfe600640 1 Processor -- start 2024-03-18T16:36:52.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.749+0000 7f7cfe600640 1 -- start start 2024-03-18T16:36:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.750+0000 7f7cfe600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7cf80ca960 0x7f7cf806c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.750+0000 7f7cfe600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7cf80cad40 0x7f7cf815e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.750+0000 7f7cfe600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.750+0000 7f7cfe600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf80c8680 con 0x7f7cf80cad40 2024-03-18T16:36:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.750+0000 7f7cfe600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf815e670 con 0x7f7cf80ca960 2024-03-18T16:36:52.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.750+0000 7f7cfe600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf806c9c0 con 0x7f7cf815a1a0 2024-03-18T16:36:52.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.751+0000 7f7cfdc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.751+0000 7f7cfd200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7cf80ca960 0x7f7cf806c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.751+0000 7f7cf7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7cf80cad40 0x7f7cf815e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.752+0000 7f7cfdc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38102/0 (socket says 172.21.15.102:38102) 2024-03-18T16:36:52.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.752+0000 7f7cfdc00640 1 -- 172.21.15.102:0/895680987 learned_addr learned my addr 172.21.15.102:0/895680987 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:52.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.754+0000 7f7cfdc00640 1 -- 172.21.15.102:0/895680987 >> v2:172.21.15.154:3300/0 conn(0x7f7cf80ca960 msgr2=0x7f7cf806c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:52.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.754+0000 7f7cfdc00640 1 --2- 172.21.15.102:0/895680987 >> v2:172.21.15.154:3300/0 conn(0x7f7cf80ca960 0x7f7cf806c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.754+0000 7f7cfdc00640 1 -- 172.21.15.102:0/895680987 >> v2:172.21.15.102:3300/0 conn(0x7f7cf80cad40 msgr2=0x7f7cf815e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:52.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.754+0000 7f7cfdc00640 1 --2- 172.21.15.102:0/895680987 >> v2:172.21.15.102:3300/0 conn(0x7f7cf80cad40 0x7f7cf815e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.754+0000 7f7cfdc00640 1 -- 172.21.15.102:0/895680987 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf806cca0 con 0x7f7cf815a1a0 2024-03-18T16:36:52.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.756+0000 7f7cfdc00640 1 --2- 172.21.15.102:0/895680987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80705a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80cb250 tx=0x7f7cf002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=346458f0fe85c1b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.760+0000 7f7cf7400640 1 -- 172.21.15.102:0/895680987 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf002cb00 con 0x7f7cf815a1a0 2024-03-18T16:36:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.760+0000 7f7cf7400640 1 -- 172.21.15.102:0/895680987 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf002cca0 con 0x7f7cf815a1a0 2024-03-18T16:36:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.760+0000 7f7cf7400640 1 -- 172.21.15.102:0/895680987 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0034610 con 0x7f7cf815a1a0 2024-03-18T16:36:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.761+0000 7f7cfe600640 1 -- 172.21.15.102:0/895680987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 msgr2=0x7f7cf80705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.761+0000 7f7cfe600640 1 --2- 172.21.15.102:0/895680987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80705a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80cb250 tx=0x7f7cf002bf60 comp rx=0 tx=0).stop 2024-03-18T16:36:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.761+0000 7f7cfe600640 1 -- 172.21.15.102:0/895680987 shutdown_connections 2024-03-18T16:36:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.761+0000 7f7cfe600640 1 --2- 172.21.15.102:0/895680987 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80705a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.761+0000 7f7cfe600640 1 --2- 172.21.15.102:0/895680987 >> v2:172.21.15.102:3300/0 conn(0x7f7cf80cad40 0x7f7cf815e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.761+0000 7f7cfe600640 1 --2- 172.21.15.102:0/895680987 >> v2:172.21.15.154:3300/0 conn(0x7f7cf80ca960 0x7f7cf806c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.761+0000 7f7cfe600640 1 -- 172.21.15.102:0/895680987 >> 172.21.15.102:0/895680987 conn(0x7f7cf8067190 msgr2=0x7f7cf80675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.761+0000 7f7cfe600640 1 -- 172.21.15.102:0/895680987 shutdown_connections 2024-03-18T16:36:52.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.761+0000 7f7cfe600640 1 -- 172.21.15.102:0/895680987 wait complete. 2024-03-18T16:36:52.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.764+0000 7f7cfe600640 1 Processor -- start 2024-03-18T16:36:52.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.764+0000 7f7cfe600640 1 -- start start 2024-03-18T16:36:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.765+0000 7f7cfe600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cf80ca960 0x7f7cf813d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.765+0000 7f7cfe600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80cad40 0x7f7cf813b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.765+0000 7f7cfe600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.765+0000 7f7cfe600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf815dc10 con 0x7f7cf80ca960 2024-03-18T16:36:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.765+0000 7f7cfe600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf815d930 con 0x7f7cf80cad40 2024-03-18T16:36:52.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.765+0000 7f7cfe600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf8070ae0 con 0x7f7cf815a1a0 2024-03-18T16:36:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.766+0000 7f7cf7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80cad40 0x7f7cf813b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.766+0000 7f7cfdc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.766+0000 7f7cfd200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cf80ca960 0x7f7cf813d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.766+0000 7f7cfdc00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38112/0 (socket says 172.21.15.102:38112) 2024-03-18T16:36:52.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.766+0000 7f7cfdc00640 1 -- 172.21.15.102:0/4221206534 learned_addr learned my addr 172.21.15.102:0/4221206534 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.770+0000 7f7cfdc00640 1 -- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80cad40 msgr2=0x7f7cf813b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.770+0000 7f7cfdc00640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80cad40 0x7f7cf813b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.770+0000 7f7cfdc00640 1 -- 172.21.15.102:0/4221206534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cf80ca960 msgr2=0x7f7cf813d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.770+0000 7f7cfdc00640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cf80ca960 0x7f7cf813d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:52.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.770+0000 7f7cfdc00640 1 -- 172.21.15.102:0/4221206534 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf81efd30 con 0x7f7cf815a1a0 2024-03-18T16:36:52.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.773+0000 7f7cfdc00640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80cb730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0036040 tx=0x7f7cf0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.777+0000 7f7cf4c00640 1 -- 172.21.15.102:0/4221206534 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0043020 con 0x7f7cf815a1a0 2024-03-18T16:36:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.777+0000 7f7cf4c00640 1 -- 172.21.15.102:0/4221206534 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf002ce50 con 0x7f7cf815a1a0 2024-03-18T16:36:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.777+0000 7f7cf4c00640 1 -- 172.21.15.102:0/4221206534 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0009510 con 0x7f7cf815a1a0 2024-03-18T16:36:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.777+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf81f3420 con 0x7f7cf815a1a0 2024-03-18T16:36:52.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.778+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf81f0290 con 0x7f7cf815a1a0 2024-03-18T16:36:52.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.779+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf8074fd0 con 0x7f7cf815a1a0 2024-03-18T16:36:52.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.800+0000 7f7cf4c00640 1 -- 172.21.15.102:0/4221206534 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7cf0046020 con 0x7f7cf815a1a0 2024-03-18T16:36:52.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.801+0000 7f7cf4c00640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7cd806bc50 0x7f7cd806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:52.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.802+0000 7f7cfd200640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7cd806bc50 0x7f7cd806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:52.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.802+0000 7f7cfd200640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7cd806bc50 0x7f7cd806e110 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009a20 tx=0x7f7cec009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:52.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:52.803+0000 7f7cf4c00640 1 -- 172.21.15.102:0/4221206534 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f7cf00798a0 con 0x7f7cf815a1a0 2024-03-18T16:36:53.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.058+0000 7f7cf4c00640 1 -- 172.21.15.102:0/4221206534 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7cf004d030 con 0x7f7cf815a1a0 2024-03-18T16:36:53.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.271+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f7cf81433e0 con 0x7f7cd806bc50 2024-03-18T16:36:53.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.271+0000 7f7cf4c00640 1 -- 172.21.15.102:0/4221206534 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7cf81433e0 con 0x7f7cd806bc50 2024-03-18T16:36:53.273 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 0 to deep-scrub 2024-03-18T16:36:53.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7cd806bc50 msgr2=0x7f7cd806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7cd806bc50 0x7f7cd806e110 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009a20 tx=0x7f7cec009190 comp rx=0 tx=0).stop 2024-03-18T16:36:53.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 msgr2=0x7f7cf80cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80cb730 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0036040 tx=0x7f7cf0004a40 comp rx=0 tx=0).stop 2024-03-18T16:36:53.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 shutdown_connections 2024-03-18T16:36:53.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f7cd806bc50 0x7f7cd806e110 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f7cf815a1a0 0x7f7cf80cb730 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80cad40 0x7f7cf813b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 --2- 172.21.15.102:0/4221206534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cf80ca960 0x7f7cf813d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 >> 172.21.15.102:0/4221206534 conn(0x7f7cf8067190 msgr2=0x7f7cf806f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:53.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 shutdown_connections 2024-03-18T16:36:53.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.275+0000 7f7cfe600640 1 -- 172.21.15.102:0/4221206534 wait complete. 2024-03-18T16:36:53.301 INFO:tasks.ceph:Scrubbing osd.1 2024-03-18T16:36:53.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-18T16:36:53.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.479+0000 7f6daa200640 1 Processor -- start 2024-03-18T16:36:53.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.480+0000 7f6daa200640 1 -- start start 2024-03-18T16:36:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.481+0000 7f6daa200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6da40ca5c0 0x7f6da40ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.481+0000 7f6daa200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6da40ce2f0 con 0x7f6da40ce640 2024-03-18T16:36:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.481+0000 7f6daa200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6da40cca30 con 0x7f6da415a800 2024-03-18T16:36:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.481+0000 7f6daa200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f6da40cd640 con 0x7f6da40ca5c0 2024-03-18T16:36:53.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.482+0000 7f6da8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6da40ca5c0 0x7f6da40ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.482+0000 7f6da8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6da40ca5c0 0x7f6da40ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33720/0 (socket says 172.21.15.102:33720) 2024-03-18T16:36:53.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.482+0000 7f6da8e00640 1 -- 172.21.15.102:0/1433191163 learned_addr learned my addr 172.21.15.102:0/1433191163 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:53.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.486+0000 7f6da8e00640 1 -- 172.21.15.102:0/1433191163 >> v1:172.21.15.154:6789/0 conn(0x7f6da415a800 legacy=0x7f6da415ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.486+0000 7f6da8e00640 1 -- 172.21.15.102:0/1433191163 >> v1:172.21.15.102:6789/0 conn(0x7f6da40ce640 legacy=0x7f6da40cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.486+0000 7f6da8e00640 1 -- 172.21.15.102:0/1433191163 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da40ccd10 con 0x7f6da40ca5c0 2024-03-18T16:36:53.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.491+0000 7f6da8e00640 1 --2- 172.21.15.102:0/1433191163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da40ca5c0 0x7f6da40ca9a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009870 tx=0x7f6d9802bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5812861b6ffc3763 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:53.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.497+0000 7f6da3400640 1 -- 172.21.15.102:0/1433191163 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9802d870 con 0x7f6da40ca5c0 2024-03-18T16:36:53.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.497+0000 7f6da3400640 1 -- 172.21.15.102:0/1433191163 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d9802da10 con 0x7f6da40ca5c0 2024-03-18T16:36:53.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.497+0000 7f6da3400640 1 -- 172.21.15.102:0/1433191163 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9802dde0 con 0x7f6da40ca5c0 2024-03-18T16:36:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.498+0000 7f6daa200640 1 -- 172.21.15.102:0/1433191163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da40ca5c0 msgr2=0x7f6da40ca9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.498+0000 7f6daa200640 1 --2- 172.21.15.102:0/1433191163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da40ca5c0 0x7f6da40ca9a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009870 tx=0x7f6d9802bab0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.498+0000 7f6daa200640 1 -- 172.21.15.102:0/1433191163 shutdown_connections 2024-03-18T16:36:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.498+0000 7f6daa200640 1 --2- 172.21.15.102:0/1433191163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da40ca5c0 0x7f6da40ca9a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.498+0000 7f6daa200640 1 -- 172.21.15.102:0/1433191163 >> 172.21.15.102:0/1433191163 conn(0x7f6da40d0060 msgr2=0x7f6da40d0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.498+0000 7f6daa200640 1 -- 172.21.15.102:0/1433191163 shutdown_connections 2024-03-18T16:36:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.498+0000 7f6daa200640 1 -- 172.21.15.102:0/1433191163 wait complete. 2024-03-18T16:36:53.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.501+0000 7f6daa200640 1 Processor -- start 2024-03-18T16:36:53.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.501+0000 7f6daa200640 1 -- start start 2024-03-18T16:36:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.502+0000 7f6daa200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da40ca5c0 0x7f6da41e1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.502+0000 7f6daa200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6da40ce640 0x7f6da4157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.502+0000 7f6daa200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da415a800 0x7f6da41e57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.502+0000 7f6daa200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f6da40ced40 con 0x7f6da40ca5c0 2024-03-18T16:36:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.502+0000 7f6daa200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6da40cd2d0 con 0x7f6da415a800 2024-03-18T16:36:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.502+0000 7f6daa200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f6da40ceec0 con 0x7f6da40ce640 2024-03-18T16:36:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.503+0000 7f6da9800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da415a800 0x7f6da41e57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.503+0000 7f6da8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da40ca5c0 0x7f6da41e1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.504+0000 7f6da9800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da415a800 0x7f6da41e57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56262/0 (socket says 172.21.15.102:56262) 2024-03-18T16:36:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.504+0000 7f6da9800640 1 -- 172.21.15.102:0/2569361394 learned_addr learned my addr 172.21.15.102:0/2569361394 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:53.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.508+0000 7f6da3e00640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6da40ce640 0x7f6da4157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.508+0000 7f6da9800640 1 -- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6da40ce640 msgr2=0x7f6da4157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.508+0000 7f6da9800640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f6da40ce640 0x7f6da4157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.508+0000 7f6da9800640 1 -- 172.21.15.102:0/2569361394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da40ca5c0 msgr2=0x7f6da41e1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.508+0000 7f6da9800640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6da40ca5c0 0x7f6da41e1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.508+0000 7f6da9800640 1 -- 172.21.15.102:0/2569361394 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da4159c40 con 0x7f6da415a800 2024-03-18T16:36:53.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.510+0000 7f6da9800640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da415a800 0x7f6da41e57e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c00c6e0 tx=0x7f6d9c00cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:53.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.512+0000 7f6da0c00640 1 -- 172.21.15.102:0/2569361394 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c003b70 con 0x7f6da415a800 2024-03-18T16:36:53.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.512+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da4159f20 con 0x7f6da415a800 2024-03-18T16:36:53.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.513+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da4157340 con 0x7f6da415a800 2024-03-18T16:36:53.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.514+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6da40ca0a0 con 0x7f6da415a800 2024-03-18T16:36:53.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.545+0000 7f6da0c00640 1 -- 172.21.15.102:0/2569361394 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d9c003d10 con 0x7f6da415a800 2024-03-18T16:36:53.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.545+0000 7f6da0c00640 1 -- 172.21.15.102:0/2569361394 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c005280 con 0x7f6da415a800 2024-03-18T16:36:53.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.548+0000 7f6da0c00640 1 -- 172.21.15.102:0/2569361394 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6d9c0054e0 con 0x7f6da415a800 2024-03-18T16:36:53.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.549+0000 7f6da0c00640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d8406bbd0 0x7f6d8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.549+0000 7f6da8e00640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d8406bbd0 0x7f6d8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.550+0000 7f6da8e00640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d8406bbd0 0x7f6d8406e090 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6d98036000 tx=0x7f6d98008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:53.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.551+0000 7f6da0c00640 1 -- 172.21.15.102:0/2569361394 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f6d9c0057d0 con 0x7f6da415a800 2024-03-18T16:36:53.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.552+0000 7f6da0c00640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f6d840755f0 0x7f6d84077a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.552+0000 7f6da0c00640 1 -- 172.21.15.102:0/2569361394 --> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d84078160 con 0x7f6d840755f0 2024-03-18T16:36:53.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.552+0000 7f6da0c00640 1 -- 172.21.15.102:0/2569361394 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6d9c0559d0 con 0x7f6da415a800 2024-03-18T16:36:53.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.552+0000 7f6da3e00640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f6d840755f0 0x7f6d84077a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.553+0000 7f6da3e00640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f6d840755f0 0x7f6d84077a90 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:53.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.554+0000 7f6da0c00640 1 -- 172.21.15.102:0/2569361394 <== osd.1 v2:172.21.15.102:6800/1867629855 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d84078160 con 0x7f6d840755f0 2024-03-18T16:36:53.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.583+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 --> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6da404f540 con 0x7f6d840755f0 2024-03-18T16:36:53.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.583+0000 7f6da0c00640 1 -- 172.21.15.102:0/2569361394 <== osd.1 v2:172.21.15.102:6800/1867629855 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6da404f540 con 0x7f6d840755f0 2024-03-18T16:36:53.585 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:36:53.585 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T16:36:53.585 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f6d840755f0 msgr2=0x7f6d84077a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.102:6800/1867629855,v1:172.21.15.102:6801/1867629855] conn(0x7f6d840755f0 0x7f6d84077a90 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d8406bbd0 msgr2=0x7f6d8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f6d8406bbd0 0x7f6d8406e090 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6d98036000 tx=0x7f6d98008030 comp rx=0 tx=0).stop 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da415a800 msgr2=0x7f6da41e57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 --2- 172.21.15.102:0/2569361394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6da415a800 0x7f6da41e57e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c00c6e0 tx=0x7f6d9c00cba0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6da9800640 1 -- 172.21.15.102:0/2569361394 reap_dead start 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 shutdown_connections 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 >> 172.21.15.102:0/2569361394 conn(0x7f6da40d0060 msgr2=0x7f6da40d0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 shutdown_connections 2024-03-18T16:36:53.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.584+0000 7f6daa200640 1 -- 172.21.15.102:0/2569361394 wait complete. 2024-03-18T16:36:53.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-18T16:36:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.776+0000 7feffb000640 1 Processor -- start 2024-03-18T16:36:53.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.776+0000 7feffb000640 1 -- start start 2024-03-18T16:36:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.777+0000 7feffb000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feff415a6f0 0x7feff415ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.777+0000 7feffb000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feff415baa0 0x7feff415bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.777+0000 7feffb000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feff40ca500 con 0x7feff415a310 2024-03-18T16:36:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.777+0000 7feffb000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7feff40cd410 con 0x7feff415a6f0 2024-03-18T16:36:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.777+0000 7feffb000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7feff415b080 con 0x7feff415baa0 2024-03-18T16:36:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.778+0000 7feffa600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feff415baa0 0x7feff415bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.778+0000 7feff9200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feff415a6f0 0x7feff415ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.778+0000 7feff9c00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7feff415a310 0x7feff406d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:39040/0 (socket says 172.21.15.102:39040) 2024-03-18T16:36:53.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.779+0000 7feff9c00640 1 -- 172.21.15.102:0/260715487 learned_addr learned my addr 172.21.15.102:0/260715487 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:53.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.781+0000 7feff9200640 1 -- 172.21.15.102:0/260715487 >> v2:172.21.15.154:3300/0 conn(0x7feff415baa0 msgr2=0x7feff415bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:53.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.781+0000 7feff9200640 1 --2- 172.21.15.102:0/260715487 >> v2:172.21.15.154:3300/0 conn(0x7feff415baa0 0x7feff415bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.781+0000 7feff9200640 1 -- 172.21.15.102:0/260715487 >> v1:172.21.15.154:6789/0 conn(0x7feff415a310 legacy=0x7feff406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.781+0000 7feff9200640 1 -- 172.21.15.102:0/260715487 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff415c430 con 0x7feff415a6f0 2024-03-18T16:36:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.784+0000 7feff9200640 1 --2- 172.21.15.102:0/260715487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feff415a6f0 0x7feff415ab40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7feff4067f40 tx=0x7fefe002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bbaab0b899af47 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:53.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.789+0000 7feff3e00640 1 -- 172.21.15.102:0/260715487 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0035020 con 0x7feff415a6f0 2024-03-18T16:36:53.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.789+0000 7feff3e00640 1 -- 172.21.15.102:0/260715487 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefe0031830 con 0x7feff415a6f0 2024-03-18T16:36:53.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.789+0000 7feff3e00640 1 -- 172.21.15.102:0/260715487 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0031c00 con 0x7feff415a6f0 2024-03-18T16:36:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.790+0000 7feffb000640 1 -- 172.21.15.102:0/260715487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feff415a6f0 msgr2=0x7feff415ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.790+0000 7feffb000640 1 --2- 172.21.15.102:0/260715487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feff415a6f0 0x7feff415ab40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7feff4067f40 tx=0x7fefe002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.790+0000 7feffb000640 1 -- 172.21.15.102:0/260715487 shutdown_connections 2024-03-18T16:36:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.790+0000 7feffb000640 1 --2- 172.21.15.102:0/260715487 >> v2:172.21.15.154:3300/0 conn(0x7feff415baa0 0x7feff415bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.790+0000 7feffb000640 1 --2- 172.21.15.102:0/260715487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feff415a6f0 0x7feff415ab40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.790+0000 7feffb000640 1 -- 172.21.15.102:0/260715487 >> 172.21.15.102:0/260715487 conn(0x7feff408c1a0 msgr2=0x7feff4066f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.790+0000 7feffb000640 1 -- 172.21.15.102:0/260715487 shutdown_connections 2024-03-18T16:36:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.790+0000 7feffb000640 1 -- 172.21.15.102:0/260715487 wait complete. 2024-03-18T16:36:53.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.793+0000 7feffb000640 1 Processor -- start 2024-03-18T16:36:53.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.793+0000 7feffb000640 1 -- start start 2024-03-18T16:36:53.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.795+0000 7feffb000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff415a310 0x7feff41f4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.795+0000 7feffb000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff415a6f0 0x7feff4150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.795+0000 7feffb000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feff415baa0 0x7feff40c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.795+0000 7feffb000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7feff40cce50 con 0x7feff415baa0 2024-03-18T16:36:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.795+0000 7feffb000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feff40caa80 con 0x7feff415a310 2024-03-18T16:36:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.795+0000 7feffb000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7feff40cd130 con 0x7feff415a6f0 2024-03-18T16:36:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.795+0000 7feff9c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff415a310 0x7feff41f4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.795+0000 7feffa600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feff415baa0 0x7feff40c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.795+0000 7feff9200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff415a6f0 0x7feff4150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:53.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.796+0000 7feff9c00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff415a310 0x7feff41f4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56276/0 (socket says 172.21.15.102:56276) 2024-03-18T16:36:53.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.796+0000 7feff9c00640 1 -- 172.21.15.102:0/2195239539 learned_addr learned my addr 172.21.15.102:0/2195239539 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:53.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.799+0000 7feff9c00640 1 -- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff415a6f0 msgr2=0x7feff4150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.799+0000 7feff9c00640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff415a6f0 0x7feff4150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.799+0000 7feff9c00640 1 -- 172.21.15.102:0/2195239539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feff415baa0 msgr2=0x7feff40c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:53.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.799+0000 7feff9c00640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feff415baa0 0x7feff40c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:53.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.799+0000 7feff9c00640 1 -- 172.21.15.102:0/2195239539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff40c3420 con 0x7feff415a310 2024-03-18T16:36:53.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.802+0000 7feff9c00640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff415a310 0x7feff41f4b20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fefe800c460 tx=0x7fefe800c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.805+0000 7feff1600640 1 -- 172.21.15.102:0/2195239539 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8017070 con 0x7feff415a310 2024-03-18T16:36:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.805+0000 7feff1600640 1 -- 172.21.15.102:0/2195239539 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefe8005ce0 con 0x7feff415a310 2024-03-18T16:36:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.805+0000 7feff1600640 1 -- 172.21.15.102:0/2195239539 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe80072b0 con 0x7feff415a310 2024-03-18T16:36:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.805+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff41f0f00 con 0x7feff415a310 2024-03-18T16:36:53.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.806+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff41f14c0 con 0x7feff415a310 2024-03-18T16:36:53.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:53.807+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff40cd980 con 0x7feff415a310 2024-03-18T16:36:54.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.066+0000 7feff1600640 1 -- 172.21.15.102:0/2195239539 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fefe801a070 con 0x7feff415a310 2024-03-18T16:36:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.068+0000 7feff1600640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefd406bc20 0x7fefd406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.069+0000 7feff1600640 1 -- 172.21.15.102:0/2195239539 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fefe808a530 con 0x7feff415a310 2024-03-18T16:36:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.069+0000 7feff9200640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefd406bc20 0x7fefd406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.070+0000 7feff9200640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefd406bc20 0x7fefd406e0e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fefe0036000 tx=0x7fefe0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:54.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.103+0000 7feff1600640 1 -- 172.21.15.102:0/2195239539 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feff40cd980 con 0x7feff415a310 2024-03-18T16:36:54.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.399+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7feff40ca1d0 con 0x7fefd406bc20 2024-03-18T16:36:54.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.400+0000 7feff1600640 1 -- 172.21.15.102:0/2195239539 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7feff40ca1d0 con 0x7fefd406bc20 2024-03-18T16:36:54.402 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 1 to deep-scrub 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefd406bc20 msgr2=0x7fefd406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefd406bc20 0x7fefd406e0e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fefe0036000 tx=0x7fefe0008030 comp rx=0 tx=0).stop 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff415a310 msgr2=0x7feff41f4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff415a310 0x7feff41f4b20 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fefe800c460 tx=0x7fefe800c920 comp rx=0 tx=0).stop 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 shutdown_connections 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fefd406bc20 0x7fefd406e0e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feff415baa0 0x7feff40c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7feff415a6f0 0x7feff4150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 --2- 172.21.15.102:0/2195239539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff415a310 0x7feff41f4b20 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 >> 172.21.15.102:0/2195239539 conn(0x7feff408c1a0 msgr2=0x7feff415d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 shutdown_connections 2024-03-18T16:36:54.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.403+0000 7feffb000640 1 -- 172.21.15.102:0/2195239539 wait complete. 2024-03-18T16:36:54.422 INFO:tasks.ceph:Scrubbing osd.2 2024-03-18T16:36:54.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-18T16:36:54.555 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:36:54.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:36:54.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.605+0000 7f9516200640 1 Processor -- start 2024-03-18T16:36:54.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.606+0000 7f9516200640 1 -- start start 2024-03-18T16:36:54.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.608+0000 7f9516200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9510141700 0x7f95100723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.608+0000 7f9516200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951006ba70 0x7f951006be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.608+0000 7f9516200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9510158390 con 0x7f95100c4fb0 2024-03-18T16:36:54.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.608+0000 7f9516200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9510141ae0 con 0x7f9510141700 2024-03-18T16:36:54.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.608+0000 7f9516200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f9510158670 con 0x7f951006ba70 2024-03-18T16:36:54.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.609+0000 7f950f400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9510141700 0x7f95100723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.609+0000 7f950f400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9510141700 0x7f95100723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56286/0 (socket says 172.21.15.102:56286) 2024-03-18T16:36:54.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.609+0000 7f950f400640 1 -- 172.21.15.102:0/1413489139 learned_addr learned my addr 172.21.15.102:0/1413489139 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:54.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.610+0000 7f950ea00640 1 --2- 172.21.15.102:0/1413489139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951006ba70 0x7f951006be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.611+0000 7f950f400640 1 -- 172.21.15.102:0/1413489139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951006ba70 msgr2=0x7f951006be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:54.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.611+0000 7f950f400640 1 --2- 172.21.15.102:0/1413489139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951006ba70 0x7f951006be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.611+0000 7f950f400640 1 -- 172.21.15.102:0/1413489139 >> v1:172.21.15.102:6789/0 conn(0x7f95100c4fb0 legacy=0x7f951015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.611+0000 7f950f400640 1 -- 172.21.15.102:0/1413489139 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9510158950 con 0x7f9510141700 2024-03-18T16:36:54.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.614+0000 7f950f400640 1 --2- 172.21.15.102:0/1413489139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510141700 0x7f95100723b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9500009870 tx=0x7f950002ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74e3f9ea72cafd6e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:54.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.618+0000 7f950e000640 1 -- 172.21.15.102:0/1413489139 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950002f830 con 0x7f9510141700 2024-03-18T16:36:54.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.618+0000 7f950e000640 1 -- 172.21.15.102:0/1413489139 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f950002f9d0 con 0x7f9510141700 2024-03-18T16:36:54.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.618+0000 7f950e000640 1 -- 172.21.15.102:0/1413489139 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950002fda0 con 0x7f9510141700 2024-03-18T16:36:54.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.619+0000 7f9516200640 1 -- 172.21.15.102:0/1413489139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510141700 msgr2=0x7f95100723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.619+0000 7f9516200640 1 --2- 172.21.15.102:0/1413489139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510141700 0x7f95100723b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9500009870 tx=0x7f950002ba10 comp rx=0 tx=0).stop 2024-03-18T16:36:54.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.619+0000 7f9516200640 1 -- 172.21.15.102:0/1413489139 shutdown_connections 2024-03-18T16:36:54.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.619+0000 7f9516200640 1 --2- 172.21.15.102:0/1413489139 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f951006ba70 0x7f951006be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.619+0000 7f9516200640 1 --2- 172.21.15.102:0/1413489139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510141700 0x7f95100723b0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.619+0000 7f9516200640 1 -- 172.21.15.102:0/1413489139 >> 172.21.15.102:0/1413489139 conn(0x7f95100c83a0 msgr2=0x7f95100c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:54.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.619+0000 7f9516200640 1 -- 172.21.15.102:0/1413489139 shutdown_connections 2024-03-18T16:36:54.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.619+0000 7f9516200640 1 -- 172.21.15.102:0/1413489139 wait complete. 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.622+0000 7f9516200640 1 Processor -- start 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.622+0000 7f9516200640 1 -- start start 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.623+0000 7f9516200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f951006ba70 0x7f9510156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.624+0000 7f9516200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95100c4fb0 0x7f951006f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.624+0000 7f9516200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9510141700 0x7f951015a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.624+0000 7f9516200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f951015c260 con 0x7f95100c4fb0 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.624+0000 7f9516200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9510158f10 con 0x7f951006ba70 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.624+0000 7f9516200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f951015c3e0 con 0x7f9510141700 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.624+0000 7f950f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f951006ba70 0x7f9510156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.624+0000 7f950fe00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9510141700 0x7f951015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.625+0000 7f950f400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f951006ba70 0x7f9510156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56292/0 (socket says 172.21.15.102:56292) 2024-03-18T16:36:54.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.625+0000 7f950ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95100c4fb0 0x7f951006f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.625+0000 7f950f400640 1 -- 172.21.15.102:0/2078259514 learned_addr learned my addr 172.21.15.102:0/2078259514 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:54.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.628+0000 7f950f400640 1 -- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9510141700 msgr2=0x7f951015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.628+0000 7f950f400640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f9510141700 0x7f951015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.628+0000 7f950f400640 1 -- 172.21.15.102:0/2078259514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95100c4fb0 msgr2=0x7f951006f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.628+0000 7f950f400640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95100c4fb0 0x7f951006f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.628+0000 7f950f400640 1 -- 172.21.15.102:0/2078259514 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951015b750 con 0x7f951006ba70 2024-03-18T16:36:54.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.630+0000 7f950f400640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f951006ba70 0x7f9510156380 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f950002ba50 tx=0x7f9500004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:54.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.634+0000 7f9507400640 1 -- 172.21.15.102:0/2078259514 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500037dc0 con 0x7f951006ba70 2024-03-18T16:36:54.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.635+0000 7f9507400640 1 -- 172.21.15.102:0/2078259514 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f950003e530 con 0x7f951006ba70 2024-03-18T16:36:54.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.635+0000 7f9507400640 1 -- 172.21.15.102:0/2078259514 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950003e920 con 0x7f951006ba70 2024-03-18T16:36:54.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.635+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95101594e0 con 0x7f951006ba70 2024-03-18T16:36:54.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.635+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9510159aa0 con 0x7f951006ba70 2024-03-18T16:36:54.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.636+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9510074fd0 con 0x7f951006ba70 2024-03-18T16:36:54.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.668+0000 7f9507400640 1 -- 172.21.15.102:0/2078259514 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f950003eac0 con 0x7f951006ba70 2024-03-18T16:36:54.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.669+0000 7f9507400640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f006bc20 0x7f94f006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.670+0000 7f950ea00640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f006bc20 0x7f94f006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.671+0000 7f950ea00640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f006bc20 0x7f94f006e0e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f95100c29f0 tx=0x7f94fc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:54.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.671+0000 7f9507400640 1 -- 172.21.15.102:0/2078259514 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f950002f980 con 0x7f951006ba70 2024-03-18T16:36:54.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.672+0000 7f9507400640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f94f00756c0 0x7f94f0077b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.672+0000 7f9507400640 1 -- 172.21.15.102:0/2078259514 --> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94f0078230 con 0x7f94f00756c0 2024-03-18T16:36:54.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.672+0000 7f950fe00640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f94f00756c0 0x7f94f0077b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.673+0000 7f9507400640 1 -- 172.21.15.102:0/2078259514 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f950007bbe0 con 0x7f951006ba70 2024-03-18T16:36:54.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.673+0000 7f950fe00640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f94f00756c0 0x7f94f0077b60 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:54.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.674+0000 7f9507400640 1 -- 172.21.15.102:0/2078259514 <== osd.2 v2:172.21.15.102:6824/2928547335 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f94f0078230 con 0x7f94f00756c0 2024-03-18T16:36:54.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.703+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 --> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9510152600 con 0x7f94f00756c0 2024-03-18T16:36:54.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.704+0000 7f9507400640 1 -- 172.21.15.102:0/2078259514 <== osd.2 v2:172.21.15.102:6824/2928547335 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9510152600 con 0x7f94f00756c0 2024-03-18T16:36:54.705 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:36:54.705 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T16:36:54.705 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:36:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.704+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f94f00756c0 msgr2=0x7f94f0077b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.704+0000 7f9516200640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.102:6824/2928547335,v1:172.21.15.102:6825/2928547335] conn(0x7f94f00756c0 0x7f94f0077b60 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.704+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f006bc20 msgr2=0x7f94f006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.704+0000 7f9516200640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f94f006bc20 0x7f94f006e0e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f95100c29f0 tx=0x7f94fc008040 comp rx=0 tx=0).stop 2024-03-18T16:36:54.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.704+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f951006ba70 msgr2=0x7f9510156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.704+0000 7f9516200640 1 --2- 172.21.15.102:0/2078259514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f951006ba70 0x7f9510156380 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f950002ba50 tx=0x7f9500004730 comp rx=0 tx=0).stop 2024-03-18T16:36:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.704+0000 7f950fe00640 1 -- 172.21.15.102:0/2078259514 reap_dead start 2024-03-18T16:36:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.705+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 shutdown_connections 2024-03-18T16:36:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.705+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 >> 172.21.15.102:0/2078259514 conn(0x7f95100c83a0 msgr2=0x7f95100c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.705+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 shutdown_connections 2024-03-18T16:36:54.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.705+0000 7f9516200640 1 -- 172.21.15.102:0/2078259514 wait complete. 2024-03-18T16:36:54.723 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-18T16:36:54.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.940+0000 7fe3eec00640 1 Processor -- start 2024-03-18T16:36:54.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.940+0000 7fe3eec00640 1 -- start start 2024-03-18T16:36:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.941+0000 7fe3eec00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80cfe70 0x7fe3e815ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.941+0000 7fe3eec00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3e80cabd0 con 0x7fe3e80d0250 2024-03-18T16:36:54.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.941+0000 7fe3eec00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3e80caea0 con 0x7fe3e80cc2f0 2024-03-18T16:36:54.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.941+0000 7fe3eec00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e80cb180 con 0x7fe3e80cfe70 2024-03-18T16:36:54.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.942+0000 7fe3ece00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe3e80d0250 0x7fe3e80c65e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36448/0 (socket says 172.21.15.102:36448) 2024-03-18T16:36:54.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.942+0000 7fe3ece00640 1 -- 172.21.15.102:0/1459262372 learned_addr learned my addr 172.21.15.102:0/1459262372 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:54.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.942+0000 7fe3ed800640 1 --2- 172.21.15.102:0/1459262372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80cfe70 0x7fe3e815ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.944+0000 7fe3e7e00640 1 -- 172.21.15.102:0/1459262372 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129266960 0 0) 0x7fe3e80caea0 con 0x7fe3e80cc2f0 2024-03-18T16:36:54.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.944+0000 7fe3e7e00640 1 -- 172.21.15.102:0/1459262372 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3d0003520 con 0x7fe3e80cc2f0 2024-03-18T16:36:54.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.946+0000 7fe3ed800640 1 -- 172.21.15.102:0/1459262372 >> v1:172.21.15.154:6789/0 conn(0x7fe3e80cc2f0 legacy=0x7fe3e80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.946+0000 7fe3ed800640 1 -- 172.21.15.102:0/1459262372 >> v1:172.21.15.102:6789/0 conn(0x7fe3e80d0250 legacy=0x7fe3e80c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.946+0000 7fe3ed800640 1 -- 172.21.15.102:0/1459262372 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e80cb460 con 0x7fe3e80cfe70 2024-03-18T16:36:54.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.949+0000 7fe3ed800640 1 --2- 172.21.15.102:0/1459262372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80cfe70 0x7fe3e815ae40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0097d0 tx=0x7fe3dc02b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5c840eb851f14d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:54.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.953+0000 7fe3e7e00640 1 -- 172.21.15.102:0/1459262372 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc034070 con 0x7fe3e80cfe70 2024-03-18T16:36:54.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.953+0000 7fe3e7e00640 1 -- 172.21.15.102:0/1459262372 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3dc030890 con 0x7fe3e80cfe70 2024-03-18T16:36:54.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.953+0000 7fe3e7e00640 1 -- 172.21.15.102:0/1459262372 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc030c60 con 0x7fe3e80cfe70 2024-03-18T16:36:54.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.954+0000 7fe3eec00640 1 -- 172.21.15.102:0/1459262372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80cfe70 msgr2=0x7fe3e815ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.954+0000 7fe3eec00640 1 --2- 172.21.15.102:0/1459262372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80cfe70 0x7fe3e815ae40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc0097d0 tx=0x7fe3dc02b9d0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.954+0000 7fe3eec00640 1 -- 172.21.15.102:0/1459262372 shutdown_connections 2024-03-18T16:36:54.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.954+0000 7fe3eec00640 1 --2- 172.21.15.102:0/1459262372 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80cfe70 0x7fe3e815ae40 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.954+0000 7fe3eec00640 1 -- 172.21.15.102:0/1459262372 >> 172.21.15.102:0/1459262372 conn(0x7fe3e808c3e0 msgr2=0x7fe3e806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:54.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.954+0000 7fe3eec00640 1 -- 172.21.15.102:0/1459262372 shutdown_connections 2024-03-18T16:36:54.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.954+0000 7fe3eec00640 1 -- 172.21.15.102:0/1459262372 wait complete. 2024-03-18T16:36:54.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.956+0000 7fe3eec00640 1 Processor -- start 2024-03-18T16:36:54.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.956+0000 7fe3eec00640 1 -- start start 2024-03-18T16:36:54.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.957+0000 7fe3eec00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3e80cc2f0 0x7fe3e81f0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.957+0000 7fe3eec00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3e80cfe70 0x7fe3e80c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.957+0000 7fe3eec00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80d0250 0x7fe3e8150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:54.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.957+0000 7fe3eec00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e80cbd00 con 0x7fe3e80cfe70 2024-03-18T16:36:54.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.957+0000 7fe3eec00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e80cba20 con 0x7fe3e80cc2f0 2024-03-18T16:36:54.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.957+0000 7fe3eec00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e80c8330 con 0x7fe3e80d0250 2024-03-18T16:36:54.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.958+0000 7fe3ed800640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3e80cc2f0 0x7fe3e81f0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.958+0000 7fe3ee200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80d0250 0x7fe3e8150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.958+0000 7fe3ece00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3e80cfe70 0x7fe3e80c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:54.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.958+0000 7fe3ee200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80d0250 0x7fe3e8150550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38152/0 (socket says 172.21.15.102:38152) 2024-03-18T16:36:54.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.958+0000 7fe3ee200640 1 -- 172.21.15.102:0/2686368461 learned_addr learned my addr 172.21.15.102:0/2686368461 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:54.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.962+0000 7fe3ee200640 1 -- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3e80cc2f0 msgr2=0x7fe3e81f0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.962+0000 7fe3ee200640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3e80cc2f0 0x7fe3e81f0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.962+0000 7fe3ee200640 1 -- 172.21.15.102:0/2686368461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3e80cfe70 msgr2=0x7fe3e80c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:54.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.962+0000 7fe3ee200640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3e80cfe70 0x7fe3e80c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:54.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.962+0000 7fe3ee200640 1 -- 172.21.15.102:0/2686368461 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e81efde0 con 0x7fe3e80d0250 2024-03-18T16:36:54.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.965+0000 7fe3ee200640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80d0250 0x7fe3e8150550 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c7c0 tx=0x7fe3e000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:54.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.967+0000 7fe3e5600640 1 -- 172.21.15.102:0/2686368461 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0003b70 con 0x7fe3e80d0250 2024-03-18T16:36:54.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.967+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e81f13e0 con 0x7fe3e80d0250 2024-03-18T16:36:54.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.967+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e81e0b20 con 0x7fe3e80d0250 2024-03-18T16:36:54.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.969+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e8074fd0 con 0x7fe3e80d0250 2024-03-18T16:36:54.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.969+0000 7fe3e5600640 1 -- 172.21.15.102:0/2686368461 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e0003d10 con 0x7fe3e80d0250 2024-03-18T16:36:54.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:54.969+0000 7fe3e5600640 1 -- 172.21.15.102:0/2686368461 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e000aa50 con 0x7fe3e80d0250 2024-03-18T16:36:55.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.233+0000 7fe3e5600640 1 -- 172.21.15.102:0/2686368461 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe3e000abf0 con 0x7fe3e80d0250 2024-03-18T16:36:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.235+0000 7fe3e5600640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3c806bbd0 0x7fe3c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.236+0000 7fe3e5600640 1 -- 172.21.15.102:0/2686368461 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fe3e0057960 con 0x7fe3e80d0250 2024-03-18T16:36:55.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.236+0000 7fe3ed800640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3c806bbd0 0x7fe3c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:55.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.237+0000 7fe3ed800640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3c806bbd0 0x7fe3c806e090 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc036000 tx=0x7fe3dc03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:55.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.246+0000 7fe3e5600640 1 -- 172.21.15.102:0/2686368461 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe3e00598c0 con 0x7fe3e80d0250 2024-03-18T16:36:55.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.510+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fe3e80671b0 con 0x7fe3c806bbd0 2024-03-18T16:36:55.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.511+0000 7fe3e5600640 1 -- 172.21.15.102:0/2686368461 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe3e80671b0 con 0x7fe3c806bbd0 2024-03-18T16:36:55.513 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 2 to deep-scrub 2024-03-18T16:36:55.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3c806bbd0 msgr2=0x7fe3c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:55.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3c806bbd0 0x7fe3c806e090 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc036000 tx=0x7fe3dc03b040 comp rx=0 tx=0).stop 2024-03-18T16:36:55.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80d0250 msgr2=0x7fe3e8150550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:55.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80d0250 0x7fe3e8150550 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c7c0 tx=0x7fe3e000cc80 comp rx=0 tx=0).stop 2024-03-18T16:36:55.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 shutdown_connections 2024-03-18T16:36:55.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fe3c806bbd0 0x7fe3c806e090 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fe3e80d0250 0x7fe3e8150550 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3e80cfe70 0x7fe3e80c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 --2- 172.21.15.102:0/2686368461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3e80cc2f0 0x7fe3e81f0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 >> 172.21.15.102:0/2686368461 conn(0x7fe3e808c3e0 msgr2=0x7fe3e80c63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:55.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 shutdown_connections 2024-03-18T16:36:55.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.514+0000 7fe3eec00640 1 -- 172.21.15.102:0/2686368461 wait complete. 2024-03-18T16:36:55.541 INFO:tasks.ceph:Scrubbing osd.3 2024-03-18T16:36:55.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-18T16:36:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.716+0000 7fd31f200640 1 Processor -- start 2024-03-18T16:36:55.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.716+0000 7fd31f200640 1 -- start start 2024-03-18T16:36:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.717+0000 7fd31f200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd320152290 0x7fd32006de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.717+0000 7fd31f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd32015b6a0 0x7fd3201587e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.717+0000 7fd31f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd32015bb30 con 0x7fd32006e340 2024-03-18T16:36:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.717+0000 7fd31f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fd32015c650 con 0x7fd32015b6a0 2024-03-18T16:36:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.717+0000 7fd31f200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd320158350 con 0x7fd320152290 2024-03-18T16:36:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.718+0000 7fd31d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd32015b6a0 0x7fd3201587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:55.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.718+0000 7fd31de00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd320152290 0x7fd32006de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:55.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.718+0000 7fd31e800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd32006e340 0x7fd3200c50b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36454/0 (socket says 172.21.15.102:36454) 2024-03-18T16:36:55.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.718+0000 7fd31e800640 1 -- 172.21.15.102:0/2694051274 learned_addr learned my addr 172.21.15.102:0/2694051274 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.722+0000 7fd31de00640 1 -- 172.21.15.102:0/2694051274 >> v2:172.21.15.102:3300/0 conn(0x7fd32015b6a0 msgr2=0x7fd3201587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.722+0000 7fd31de00640 1 --2- 172.21.15.102:0/2694051274 >> v2:172.21.15.102:3300/0 conn(0x7fd32015b6a0 0x7fd3201587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.722+0000 7fd31de00640 1 -- 172.21.15.102:0/2694051274 >> v1:172.21.15.102:6789/0 conn(0x7fd32006e340 legacy=0x7fd3200c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:55.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.722+0000 7fd31de00640 1 -- 172.21.15.102:0/2694051274 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd32015bdd0 con 0x7fd320152290 2024-03-18T16:36:55.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.725+0000 7fd31de00640 1 --2- 172.21.15.102:0/2694051274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd320152290 0x7fd32006de00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd3200cca30 tx=0x7fd31002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=166772356959c34 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:55.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.729+0000 7fd31ca00640 1 -- 172.21.15.102:0/2694051274 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310035020 con 0x7fd320152290 2024-03-18T16:36:55.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.730+0000 7fd31ca00640 1 -- 172.21.15.102:0/2694051274 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd310031830 con 0x7fd320152290 2024-03-18T16:36:55.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.730+0000 7fd31ca00640 1 -- 172.21.15.102:0/2694051274 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310031c00 con 0x7fd320152290 2024-03-18T16:36:55.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.731+0000 7fd31f200640 1 -- 172.21.15.102:0/2694051274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd320152290 msgr2=0x7fd32006de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:55.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.731+0000 7fd31f200640 1 --2- 172.21.15.102:0/2694051274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd320152290 0x7fd32006de00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd3200cca30 tx=0x7fd31002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.731+0000 7fd31f200640 1 -- 172.21.15.102:0/2694051274 shutdown_connections 2024-03-18T16:36:55.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.731+0000 7fd31f200640 1 --2- 172.21.15.102:0/2694051274 >> v2:172.21.15.102:3300/0 conn(0x7fd32015b6a0 0x7fd3201587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.731+0000 7fd31f200640 1 --2- 172.21.15.102:0/2694051274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd320152290 0x7fd32006de00 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.731+0000 7fd31f200640 1 -- 172.21.15.102:0/2694051274 >> 172.21.15.102:0/2694051274 conn(0x7fd3200c84a0 msgr2=0x7fd3200c8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:55.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.731+0000 7fd31f200640 1 -- 172.21.15.102:0/2694051274 shutdown_connections 2024-03-18T16:36:55.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.731+0000 7fd31f200640 1 -- 172.21.15.102:0/2694051274 wait complete. 2024-03-18T16:36:55.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.733+0000 7fd31f200640 1 Processor -- start 2024-03-18T16:36:55.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.734+0000 7fd31f200640 1 -- start start 2024-03-18T16:36:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.735+0000 7fd31f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd32006e340 0x7fd3201e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.735+0000 7fd31f200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd320152290 0x7fd320157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.735+0000 7fd31f200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3201e93c0 0x7fd320149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.735+0000 7fd31f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd320072550 con 0x7fd32006e340 2024-03-18T16:36:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.735+0000 7fd31f200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd32015c370 con 0x7fd320152290 2024-03-18T16:36:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.735+0000 7fd31f200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd320152670 con 0x7fd3201e93c0 2024-03-18T16:36:55.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.736+0000 7fd31d400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd320152290 0x7fd320157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:55.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.736+0000 7fd31d400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd320152290 0x7fd320157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56302/0 (socket says 172.21.15.102:56302) 2024-03-18T16:36:55.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.736+0000 7fd31d400640 1 -- 172.21.15.102:0/2935777246 learned_addr learned my addr 172.21.15.102:0/2935777246 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:55.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.737+0000 7fd31e800640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3201e93c0 0x7fd320149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:55.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.737+0000 7fd31de00640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd32006e340 0x7fd3201e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:55.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.741+0000 7fd31de00640 1 -- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3201e93c0 msgr2=0x7fd320149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:55.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.741+0000 7fd31de00640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd3201e93c0 0x7fd320149b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.741+0000 7fd31de00640 1 -- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd320152290 msgr2=0x7fd320157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:55.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.741+0000 7fd31de00640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd320152290 0x7fd320157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.741+0000 7fd31de00640 1 -- 172.21.15.102:0/2935777246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3201f4610 con 0x7fd32006e340 2024-03-18T16:36:55.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.744+0000 7fd31de00640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd32006e340 0x7fd3201e78b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd310033000 tx=0x7fd3100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:55.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.748+0000 7fd316000640 1 -- 172.21.15.102:0/2935777246 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310041070 con 0x7fd32006e340 2024-03-18T16:36:55.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.748+0000 7fd316000640 1 -- 172.21.15.102:0/2935777246 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31003b940 con 0x7fd32006e340 2024-03-18T16:36:55.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.748+0000 7fd316000640 1 -- 172.21.15.102:0/2935777246 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310035040 con 0x7fd32006e340 2024-03-18T16:36:55.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.748+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3201f48f0 con 0x7fd32006e340 2024-03-18T16:36:55.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.749+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3201f4eb0 con 0x7fd32006e340 2024-03-18T16:36:55.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.750+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd3200c7e20 con 0x7fd32006e340 2024-03-18T16:36:55.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.776+0000 7fd316000640 1 -- 172.21.15.102:0/2935777246 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd31004c060 con 0x7fd32006e340 2024-03-18T16:36:55.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.777+0000 7fd316000640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd30006bc20 0x7fd30006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:55.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.777+0000 7fd31d400640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd30006bc20 0x7fd30006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:55.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.778+0000 7fd31d400640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd30006bc20 0x7fd30006e0e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd3201516e0 tx=0x7fd30c009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:55.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.780+0000 7fd316000640 1 -- 172.21.15.102:0/2935777246 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd3100790a0 con 0x7fd32006e340 2024-03-18T16:36:55.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.780+0000 7fd316000640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fd300075740 0x7fd300077be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:55.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.780+0000 7fd316000640 1 -- 172.21.15.102:0/2935777246 --> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3000782b0 con 0x7fd300075740 2024-03-18T16:36:55.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.780+0000 7fd316000640 1 -- 172.21.15.102:0/2935777246 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd3100772a0 con 0x7fd32006e340 2024-03-18T16:36:55.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.781+0000 7fd31e800640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fd300075740 0x7fd300077be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:55.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.781+0000 7fd31e800640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fd300075740 0x7fd300077be0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:55.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.783+0000 7fd316000640 1 -- 172.21.15.102:0/2935777246 <== osd.3 v2:172.21.15.102:6808/1215178908 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd3000782b0 con 0x7fd300075740 2024-03-18T16:36:55.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.811+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 --> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd320067c20 con 0x7fd300075740 2024-03-18T16:36:55.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.812+0000 7fd316000640 1 -- 172.21.15.102:0/2935777246 <== osd.3 v2:172.21.15.102:6808/1215178908 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd320067c20 con 0x7fd300075740 2024-03-18T16:36:55.814 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:36:55.814 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T16:36:55.814 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:36:55.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.813+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fd300075740 msgr2=0x7fd300077be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:55.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.813+0000 7fd31f200640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.102:6808/1215178908,v1:172.21.15.102:6810/1215178908] conn(0x7fd300075740 0x7fd300077be0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.813+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd30006bc20 msgr2=0x7fd30006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.813+0000 7fd31f200640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd30006bc20 0x7fd30006e0e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd3201516e0 tx=0x7fd30c009190 comp rx=0 tx=0).stop 2024-03-18T16:36:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.813+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd32006e340 msgr2=0x7fd3201e78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.813+0000 7fd31f200640 1 --2- 172.21.15.102:0/2935777246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd32006e340 0x7fd3201e78b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd310033000 tx=0x7fd3100046c0 comp rx=0 tx=0).stop 2024-03-18T16:36:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.813+0000 7fd31e800640 1 -- 172.21.15.102:0/2935777246 reap_dead start 2024-03-18T16:36:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.813+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 shutdown_connections 2024-03-18T16:36:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.813+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 >> 172.21.15.102:0/2935777246 conn(0x7fd3200c84a0 msgr2=0x7fd3201587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.814+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 shutdown_connections 2024-03-18T16:36:55.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:55.814+0000 7fd31f200640 1 -- 172.21.15.102:0/2935777246 wait complete. 2024-03-18T16:36:55.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-18T16:36:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.009+0000 7fd521600640 1 Processor -- start 2024-03-18T16:36:56.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.009+0000 7fd521600640 1 -- start start 2024-03-18T16:36:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.010+0000 7fd521600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd51c15b4d0 0x7fd51c0cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.011+0000 7fd521600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51c0c3150 con 0x7fd51c1589f0 2024-03-18T16:36:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.011+0000 7fd521600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51c0c3430 con 0x7fd51c152120 2024-03-18T16:36:56.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.011+0000 7fd521600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd51c158560 con 0x7fd51c15b4d0 2024-03-18T16:36:56.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.012+0000 7fd520c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd51c1589f0 0x7fd51c0704c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36460/0 (socket says 172.21.15.102:36460) 2024-03-18T16:36:56.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.012+0000 7fd520c00640 1 -- 172.21.15.102:0/2331928659 learned_addr learned my addr 172.21.15.102:0/2331928659 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:56.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.012+0000 7fd51b400640 1 --2- 172.21.15.102:0/2331928659 >> v2:172.21.15.154:3300/0 conn(0x7fd51c15b4d0 0x7fd51c0cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.018+0000 7fd51b400640 1 -- 172.21.15.102:0/2331928659 >> v1:172.21.15.154:6789/0 conn(0x7fd51c152120 legacy=0x7fd51c0c2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.018+0000 7fd51b400640 1 -- 172.21.15.102:0/2331928659 >> v1:172.21.15.102:6789/0 conn(0x7fd51c1589f0 legacy=0x7fd51c0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.018+0000 7fd51b400640 1 -- 172.21.15.102:0/2331928659 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51c0c3710 con 0x7fd51c15b4d0 2024-03-18T16:36:56.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.023+0000 7fd51b400640 1 --2- 172.21.15.102:0/2331928659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c15b4d0 0x7fd51c0cacc0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd5080097a0 tx=0x7fd50802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d19acd88ce067d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:56.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.027+0000 7fd51aa00640 1 -- 172.21.15.102:0/2331928659 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50802f810 con 0x7fd51c15b4d0 2024-03-18T16:36:56.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.027+0000 7fd51aa00640 1 -- 172.21.15.102:0/2331928659 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd50802f9b0 con 0x7fd51c15b4d0 2024-03-18T16:36:56.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.027+0000 7fd51aa00640 1 -- 172.21.15.102:0/2331928659 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50802fd80 con 0x7fd51c15b4d0 2024-03-18T16:36:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.028+0000 7fd521600640 1 -- 172.21.15.102:0/2331928659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c15b4d0 msgr2=0x7fd51c0cacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.028+0000 7fd521600640 1 --2- 172.21.15.102:0/2331928659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c15b4d0 0x7fd51c0cacc0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd5080097a0 tx=0x7fd50802b9f0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.028+0000 7fd521600640 1 -- 172.21.15.102:0/2331928659 shutdown_connections 2024-03-18T16:36:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.028+0000 7fd521600640 1 --2- 172.21.15.102:0/2331928659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c15b4d0 0x7fd51c0cacc0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.028+0000 7fd521600640 1 -- 172.21.15.102:0/2331928659 >> 172.21.15.102:0/2331928659 conn(0x7fd51c067190 msgr2=0x7fd51c0675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.028+0000 7fd521600640 1 -- 172.21.15.102:0/2331928659 shutdown_connections 2024-03-18T16:36:56.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.028+0000 7fd521600640 1 -- 172.21.15.102:0/2331928659 wait complete. 2024-03-18T16:36:56.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.031+0000 7fd521600640 1 Processor -- start 2024-03-18T16:36:56.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.031+0000 7fd521600640 1 -- start start 2024-03-18T16:36:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.032+0000 7fd521600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd51c152120 0x7fd51c0cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.032+0000 7fd521600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c1589f0 0x7fd51c0c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.032+0000 7fd521600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c15b4d0 0x7fd51c0c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.032+0000 7fd521600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c158e00 con 0x7fd51c1589f0 2024-03-18T16:36:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.032+0000 7fd521600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c157ee0 con 0x7fd51c15b4d0 2024-03-18T16:36:56.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.032+0000 7fd521600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fd51c152500 con 0x7fd51c152120 2024-03-18T16:36:56.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.033+0000 7fd51be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd51c152120 0x7fd51c0cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.033+0000 7fd51b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c1589f0 0x7fd51c0c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.034+0000 7fd51be00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd51c152120 0x7fd51c0cdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38168/0 (socket says 172.21.15.102:38168) 2024-03-18T16:36:56.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.034+0000 7fd51be00640 1 -- 172.21.15.102:0/1931431097 learned_addr learned my addr 172.21.15.102:0/1931431097 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:56.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.034+0000 7fd520c00640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c15b4d0 0x7fd51c0c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.037+0000 7fd51be00640 1 -- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c15b4d0 msgr2=0x7fd51c0c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.037+0000 7fd51be00640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c15b4d0 0x7fd51c0c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.037+0000 7fd51be00640 1 -- 172.21.15.102:0/1931431097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c1589f0 msgr2=0x7fd51c0c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.037+0000 7fd51be00640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c1589f0 0x7fd51c0c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.037+0000 7fd51be00640 1 -- 172.21.15.102:0/1931431097 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51c0cc2c0 con 0x7fd51c152120 2024-03-18T16:36:56.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.040+0000 7fd51be00640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd51c152120 0x7fd51c0cdf20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00c400 tx=0x7fd50c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:56.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.044+0000 7fd513e00640 1 -- 172.21.15.102:0/1931431097 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0052d0 con 0x7fd51c152120 2024-03-18T16:36:56.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.044+0000 7fd513e00640 1 -- 172.21.15.102:0/1931431097 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd50c005490 con 0x7fd51c152120 2024-03-18T16:36:56.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.044+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd51c0cc5a0 con 0x7fd51c152120 2024-03-18T16:36:56.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.044+0000 7fd513e00640 1 -- 172.21.15.102:0/1931431097 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c007260 con 0x7fd51c152120 2024-03-18T16:36:56.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.044+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd51c0ccb60 con 0x7fd51c152120 2024-03-18T16:36:56.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.046+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd51c074fd0 con 0x7fd51c152120 2024-03-18T16:36:56.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.067+0000 7fd513e00640 1 -- 172.21.15.102:0/1931431097 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd50c007400 con 0x7fd51c152120 2024-03-18T16:36:56.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.068+0000 7fd513e00640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd4fc06bbd0 0x7fd4fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.068+0000 7fd513e00640 1 -- 172.21.15.102:0/1931431097 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fd50c056970 con 0x7fd51c152120 2024-03-18T16:36:56.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.068+0000 7fd51b400640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd4fc06bbd0 0x7fd4fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.069+0000 7fd51b400640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd4fc06bbd0 0x7fd4fc06e090 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd508036000 tx=0x7fd508008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:56.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.317+0000 7fd513e00640 1 -- 172.21.15.102:0/1931431097 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd50c0588d0 con 0x7fd51c152120 2024-03-18T16:36:56.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.521+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fd51c0b2ec0 con 0x7fd4fc06bbd0 2024-03-18T16:36:56.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.522+0000 7fd513e00640 1 -- 172.21.15.102:0/1931431097 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd51c0b2ec0 con 0x7fd4fc06bbd0 2024-03-18T16:36:56.523 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 3 to deep-scrub 2024-03-18T16:36:56.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.524+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd4fc06bbd0 msgr2=0x7fd4fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.524+0000 7fd521600640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd4fc06bbd0 0x7fd4fc06e090 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd508036000 tx=0x7fd508008030 comp rx=0 tx=0).stop 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.524+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd51c152120 msgr2=0x7fd51c0cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.524+0000 7fd521600640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd51c152120 0x7fd51c0cdf20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00c400 tx=0x7fd50c00c8c0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.525+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 shutdown_connections 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.525+0000 7fd521600640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fd4fc06bbd0 0x7fd4fc06e090 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.525+0000 7fd521600640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c15b4d0 0x7fd51c0c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.525+0000 7fd521600640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c1589f0 0x7fd51c0c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.525+0000 7fd521600640 1 --2- 172.21.15.102:0/1931431097 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fd51c152120 0x7fd51c0cdf20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.525+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 >> 172.21.15.102:0/1931431097 conn(0x7fd51c067190 msgr2=0x7fd51c155cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.525+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 shutdown_connections 2024-03-18T16:36:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.525+0000 7fd521600640 1 -- 172.21.15.102:0/1931431097 wait complete. 2024-03-18T16:36:56.542 INFO:tasks.ceph:Scrubbing osd.4 2024-03-18T16:36:56.542 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-18T16:36:56.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.716+0000 7fcc18a00640 1 Processor -- start 2024-03-18T16:36:56.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.716+0000 7fcc18a00640 1 -- start start 2024-03-18T16:36:56.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.717+0000 7fcc18a00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcc1415af10 0x7fcc140ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.717+0000 7fcc18a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc140ce2f0 con 0x7fcc140ce640 2024-03-18T16:36:56.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.717+0000 7fcc18a00640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc140cca30 con 0x7fcc140ca5c0 2024-03-18T16:36:56.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.717+0000 7fcc18a00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcc140cd640 con 0x7fcc1415af10 2024-03-18T16:36:56.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.718+0000 7fcc12a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcc140ce640 0x7fcc1415a800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:36476/0 (socket says 172.21.15.102:36476) 2024-03-18T16:36:56.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.718+0000 7fcc12a00640 1 -- 172.21.15.102:0/758197768 learned_addr learned my addr 172.21.15.102:0/758197768 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:56.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.719+0000 7fcc13e00640 1 --2- 172.21.15.102:0/758197768 >> v2:172.21.15.154:3300/0 conn(0x7fcc1415af10 0x7fcc140ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.722+0000 7fcc12000640 1 -- 172.21.15.102:0/758197768 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3621333552 0 0) 0x7fcc140ce2f0 con 0x7fcc140ce640 2024-03-18T16:36:56.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.722+0000 7fcc12000640 1 -- 172.21.15.102:0/758197768 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf8003520 con 0x7fcc140ce640 2024-03-18T16:36:56.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.722+0000 7fcc13e00640 1 -- 172.21.15.102:0/758197768 >> v1:172.21.15.154:6789/0 conn(0x7fcc140ca5c0 legacy=0x7fcc140cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.722+0000 7fcc13e00640 1 -- 172.21.15.102:0/758197768 >> v1:172.21.15.102:6789/0 conn(0x7fcc140ce640 legacy=0x7fcc1415a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.722+0000 7fcc13e00640 1 -- 172.21.15.102:0/758197768 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc140ccd10 con 0x7fcc1415af10 2024-03-18T16:36:56.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.727+0000 7fcc13e00640 1 --2- 172.21.15.102:0/758197768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 0x7fcc140ca9a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c02bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2992f918b9f43886 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:56.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.732+0000 7fcc12000640 1 -- 172.21.15.102:0/758197768 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c039070 con 0x7fcc1415af10 2024-03-18T16:36:56.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.732+0000 7fcc12000640 1 -- 172.21.15.102:0/758197768 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c02cd80 con 0x7fcc1415af10 2024-03-18T16:36:56.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.732+0000 7fcc12000640 1 -- 172.21.15.102:0/758197768 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c034660 con 0x7fcc1415af10 2024-03-18T16:36:56.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.732+0000 7fcc18a00640 1 -- 172.21.15.102:0/758197768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 msgr2=0x7fcc140ca9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.732+0000 7fcc18a00640 1 --2- 172.21.15.102:0/758197768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 0x7fcc140ca9a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c02bfd0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.733+0000 7fcc18a00640 1 -- 172.21.15.102:0/758197768 shutdown_connections 2024-03-18T16:36:56.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.733+0000 7fcc18a00640 1 --2- 172.21.15.102:0/758197768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 0x7fcc140ca9a0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.733+0000 7fcc18a00640 1 -- 172.21.15.102:0/758197768 >> 172.21.15.102:0/758197768 conn(0x7fcc140d0060 msgr2=0x7fcc140d0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:56.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.733+0000 7fcc18a00640 1 -- 172.21.15.102:0/758197768 shutdown_connections 2024-03-18T16:36:56.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.733+0000 7fcc18a00640 1 -- 172.21.15.102:0/758197768 wait complete. 2024-03-18T16:36:56.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.735+0000 7fcc18a00640 1 Processor -- start 2024-03-18T16:36:56.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.735+0000 7fcc18a00640 1 -- start start 2024-03-18T16:36:56.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.735+0000 7fcc18a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc140ca5c0 0x7fcc140b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.736+0000 7fcc18a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc140ce640 0x7fcc14157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.736+0000 7fcc18a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 0x7fcc14157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.736+0000 7fcc18a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fcc140ced40 con 0x7fcc140ca5c0 2024-03-18T16:36:56.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.736+0000 7fcc18a00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcc140cd2d0 con 0x7fcc1415af10 2024-03-18T16:36:56.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.736+0000 7fcc18a00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fcc140ceec0 con 0x7fcc140ce640 2024-03-18T16:36:56.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.736+0000 7fcc13e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 0x7fcc14157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.736+0000 7fcc12a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc140ce640 0x7fcc14157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.737+0000 7fcc13e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 0x7fcc14157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56350/0 (socket says 172.21.15.102:56350) 2024-03-18T16:36:56.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.737+0000 7fcc13e00640 1 -- 172.21.15.102:0/1047881399 learned_addr learned my addr 172.21.15.102:0/1047881399 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:56.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.737+0000 7fcc13400640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc140ca5c0 0x7fcc140b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.740+0000 7fcc13e00640 1 -- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc140ce640 msgr2=0x7fcc14157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.740+0000 7fcc13e00640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fcc140ce640 0x7fcc14157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.740+0000 7fcc13e00640 1 -- 172.21.15.102:0/1047881399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc140ca5c0 msgr2=0x7fcc140b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.740+0000 7fcc13e00640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc140ca5c0 0x7fcc140b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.740+0000 7fcc13e00640 1 -- 172.21.15.102:0/1047881399 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc14157c70 con 0x7fcc1415af10 2024-03-18T16:36:56.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.742+0000 7fcc13e00640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 0x7fcc14157660 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009eb0 tx=0x7fcc0c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.747+0000 7fcc0b400640 1 -- 172.21.15.102:0/1047881399 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c039050 con 0x7fcc1415af10 2024-03-18T16:36:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.747+0000 7fcc0b400640 1 -- 172.21.15.102:0/1047881399 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c045070 con 0x7fcc1415af10 2024-03-18T16:36:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.747+0000 7fcc0b400640 1 -- 172.21.15.102:0/1047881399 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c009450 con 0x7fcc1415af10 2024-03-18T16:36:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.747+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc140b19c0 con 0x7fcc1415af10 2024-03-18T16:36:56.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.747+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc140b4ed0 con 0x7fcc1415af10 2024-03-18T16:36:56.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.748+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc14158110 con 0x7fcc1415af10 2024-03-18T16:36:56.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.773+0000 7fcc0b400640 1 -- 172.21.15.102:0/1047881399 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcc0c03e850 con 0x7fcc1415af10 2024-03-18T16:36:56.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.774+0000 7fcc0b400640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf406bbd0 0x7fcbf406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.775+0000 7fcc13400640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf406bbd0 0x7fcbf406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.775+0000 7fcc0b400640 1 -- 172.21.15.102:0/1047881399 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fcc0c07a670 con 0x7fcc1415af10 2024-03-18T16:36:56.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.775+0000 7fcc0b400640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7fcbf4075670 0x7fcbf4077b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:56.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.775+0000 7fcc0b400640 1 -- 172.21.15.102:0/1047881399 --> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc0c00ae60 con 0x7fcbf4075670 2024-03-18T16:36:56.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.776+0000 7fcc13400640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf406bbd0 0x7fcbf406e090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcc04001780 tx=0x7fcc04009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:56.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.776+0000 7fcc12a00640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7fcbf4075670 0x7fcbf4077b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:56.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.776+0000 7fcc0b400640 1 -- 172.21.15.102:0/1047881399 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcc0c078870 con 0x7fcc1415af10 2024-03-18T16:36:56.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.776+0000 7fcc12a00640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7fcbf4075670 0x7fcbf4077b10 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:56.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.778+0000 7fcc0b400640 1 -- 172.21.15.102:0/1047881399 <== osd.4 v2:172.21.15.154:6826/2919562209 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc0c00ae60 con 0x7fcbf4075670 2024-03-18T16:36:56.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.806+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 --> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcc140d08b0 con 0x7fcbf4075670 2024-03-18T16:36:56.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.807+0000 7fcc0b400640 1 -- 172.21.15.102:0/1047881399 <== osd.4 v2:172.21.15.154:6826/2919562209 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcc140d08b0 con 0x7fcbf4075670 2024-03-18T16:36:56.808 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:36:56.808 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T16:36:56.808 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:36:56.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.807+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7fcbf4075670 msgr2=0x7fcbf4077b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.807+0000 7fcc18a00640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6826/2919562209,v1:172.21.15.154:6827/2919562209] conn(0x7fcbf4075670 0x7fcbf4077b10 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.808+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf406bbd0 msgr2=0x7fcbf406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.808+0000 7fcc18a00640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fcbf406bbd0 0x7fcbf406e090 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcc04001780 tx=0x7fcc04009040 comp rx=0 tx=0).stop 2024-03-18T16:36:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.808+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 msgr2=0x7fcc14157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.808+0000 7fcc18a00640 1 --2- 172.21.15.102:0/1047881399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcc1415af10 0x7fcc14157660 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009eb0 tx=0x7fcc0c004a40 comp rx=0 tx=0).stop 2024-03-18T16:36:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.808+0000 7fcc13e00640 1 -- 172.21.15.102:0/1047881399 reap_dead start 2024-03-18T16:36:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.808+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 shutdown_connections 2024-03-18T16:36:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.808+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 >> 172.21.15.102:0/1047881399 conn(0x7fcc140d0060 msgr2=0x7fcc140c7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.808+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 shutdown_connections 2024-03-18T16:36:56.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:56.808+0000 7fcc18a00640 1 -- 172.21.15.102:0/1047881399 wait complete. 2024-03-18T16:36:56.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-18T16:36:57.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.011+0000 7f04e9200640 1 Processor -- start 2024-03-18T16:36:57.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.011+0000 7f04e9200640 1 -- start start 2024-03-18T16:36:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.012+0000 7f04e9200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f04e40ca960 0x7f04e406c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.012+0000 7f04e9200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e40cad40 0x7f04e415e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.012+0000 7f04e9200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04e415e670 con 0x7f04e415a1a0 2024-03-18T16:36:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.012+0000 7f04e9200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f04e40c8680 con 0x7f04e40ca960 2024-03-18T16:36:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.012+0000 7f04e9200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f04e4070cb0 con 0x7f04e40cad40 2024-03-18T16:36:57.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.013+0000 7f04e2a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e40cad40 0x7f04e415e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.014+0000 7f04e2a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e40cad40 0x7f04e415e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38192/0 (socket says 172.21.15.102:38192) 2024-03-18T16:36:57.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.014+0000 7f04e2a00640 1 -- 172.21.15.102:0/4147578797 learned_addr learned my addr 172.21.15.102:0/4147578797 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:57.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.017+0000 7f04e2a00640 1 -- 172.21.15.102:0/4147578797 >> v2:172.21.15.102:3300/0 conn(0x7f04e40ca960 msgr2=0x7f04e406c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:57.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.017+0000 7f04e2a00640 1 --2- 172.21.15.102:0/4147578797 >> v2:172.21.15.102:3300/0 conn(0x7f04e40ca960 0x7f04e406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.017+0000 7f04e2a00640 1 -- 172.21.15.102:0/4147578797 >> v1:172.21.15.102:6789/0 conn(0x7f04e415a1a0 legacy=0x7f04e40705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:57.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.017+0000 7f04e2a00640 1 -- 172.21.15.102:0/4147578797 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e406c9c0 con 0x7f04e40cad40 2024-03-18T16:36:57.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.020+0000 7f04e2a00640 1 --2- 172.21.15.102:0/4147578797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e40cad40 0x7f04e415e130 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009870 tx=0x7f04d002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0a812d9f23e77f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:57.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.023+0000 7f04e2000640 1 -- 172.21.15.102:0/4147578797 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0034070 con 0x7f04e40cad40 2024-03-18T16:36:57.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.025+0000 7f04e2000640 1 -- 172.21.15.102:0/4147578797 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d0030840 con 0x7f04e40cad40 2024-03-18T16:36:57.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.025+0000 7f04e2000640 1 -- 172.21.15.102:0/4147578797 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0030c10 con 0x7f04e40cad40 2024-03-18T16:36:57.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.026+0000 7f04e9200640 1 -- 172.21.15.102:0/4147578797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e40cad40 msgr2=0x7f04e415e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.026+0000 7f04e9200640 1 --2- 172.21.15.102:0/4147578797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e40cad40 0x7f04e415e130 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f04d0009870 tx=0x7f04d002b980 comp rx=0 tx=0).stop 2024-03-18T16:36:57.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.026+0000 7f04e9200640 1 -- 172.21.15.102:0/4147578797 shutdown_connections 2024-03-18T16:36:57.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.026+0000 7f04e9200640 1 --2- 172.21.15.102:0/4147578797 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e40cad40 0x7f04e415e130 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.026+0000 7f04e9200640 1 --2- 172.21.15.102:0/4147578797 >> v2:172.21.15.102:3300/0 conn(0x7f04e40ca960 0x7f04e406c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.026+0000 7f04e9200640 1 -- 172.21.15.102:0/4147578797 >> 172.21.15.102:0/4147578797 conn(0x7f04e4067190 msgr2=0x7f04e40675a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:57.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.026+0000 7f04e9200640 1 -- 172.21.15.102:0/4147578797 shutdown_connections 2024-03-18T16:36:57.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.026+0000 7f04e9200640 1 -- 172.21.15.102:0/4147578797 wait complete. 2024-03-18T16:36:57.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.029+0000 7f04e9200640 1 Processor -- start 2024-03-18T16:36:57.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.029+0000 7f04e9200640 1 -- start start 2024-03-18T16:36:57.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.030+0000 7f04e9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e40ca960 0x7f04e40cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.030+0000 7f04e9200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e40cad40 0x7f04e4154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.030+0000 7f04e9200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e415a1a0 0x7f04e4155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.030+0000 7f04e9200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f04e415d930 con 0x7f04e40ca960 2024-03-18T16:36:57.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.030+0000 7f04e9200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f04e406cf80 con 0x7f04e40cad40 2024-03-18T16:36:57.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.030+0000 7f04e9200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f04e415dc10 con 0x7f04e415a1a0 2024-03-18T16:36:57.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.031+0000 7f04e3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e415a1a0 0x7f04e4155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.031+0000 7f04e2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e40cad40 0x7f04e4154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.031+0000 7f04e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e40ca960 0x7f04e40cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.031+0000 7f04e3e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e415a1a0 0x7f04e4155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38194/0 (socket says 172.21.15.102:38194) 2024-03-18T16:36:57.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.031+0000 7f04e2a00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e40cad40 0x7f04e4154b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56362/0 (socket says 172.21.15.102:56362) 2024-03-18T16:36:57.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.031+0000 7f04e3e00640 1 -- 172.21.15.102:0/411523882 learned_addr learned my addr 172.21.15.102:0/411523882 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.036+0000 7f04e2a00640 1 -- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e415a1a0 msgr2=0x7f04e4155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.036+0000 7f04e2a00640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e415a1a0 0x7f04e4155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.036+0000 7f04e2a00640 1 -- 172.21.15.102:0/411523882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e40ca960 msgr2=0x7f04e40cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.036+0000 7f04e2a00640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e40ca960 0x7f04e40cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.036+0000 7f04e2a00640 1 -- 172.21.15.102:0/411523882 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e41effe0 con 0x7f04e40cad40 2024-03-18T16:36:57.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.041+0000 7f04e2a00640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e40cad40 0x7f04e4154b10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f04d0033000 tx=0x7f04d0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:57.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.045+0000 7f04db400640 1 -- 172.21.15.102:0/411523882 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0048070 con 0x7f04e40cad40 2024-03-18T16:36:57.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.045+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e4156320 con 0x7f04e40cad40 2024-03-18T16:36:57.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.046+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e4156b60 con 0x7f04e40cad40 2024-03-18T16:36:57.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.047+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e4074fd0 con 0x7f04e40cad40 2024-03-18T16:36:57.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.047+0000 7f04db400640 1 -- 172.21.15.102:0/411523882 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04d0039d90 con 0x7f04e40cad40 2024-03-18T16:36:57.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.047+0000 7f04db400640 1 -- 172.21.15.102:0/411523882 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04d0052060 con 0x7f04e40cad40 2024-03-18T16:36:57.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.321+0000 7f04db400640 1 -- 172.21.15.102:0/411523882 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f04d004b070 con 0x7f04e40cad40 2024-03-18T16:36:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.322+0000 7f04db400640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04c406bbd0 0x7f04c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.322+0000 7f04e3400640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04c406bbd0 0x7f04c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.323+0000 7f04e3400640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04c406bbd0 0x7f04c406e090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009990 tx=0x7f04d4009110 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:57.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.324+0000 7f04db400640 1 -- 172.21.15.102:0/411523882 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f04d0079450 con 0x7f04e40cad40 2024-03-18T16:36:57.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.335+0000 7f04db400640 1 -- 172.21.15.102:0/411523882 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f04e4156320 con 0x7f04e40cad40 2024-03-18T16:36:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.534+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f04e41434a0 con 0x7f04c406bbd0 2024-03-18T16:36:57.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.535+0000 7f04db400640 1 -- 172.21.15.102:0/411523882 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f04e41434a0 con 0x7f04c406bbd0 2024-03-18T16:36:57.537 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 4 to deep-scrub 2024-03-18T16:36:57.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.537+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04c406bbd0 msgr2=0x7f04c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.537+0000 7f04e9200640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04c406bbd0 0x7f04c406e090 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f04d4009990 tx=0x7f04d4009110 comp rx=0 tx=0).stop 2024-03-18T16:36:57.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.537+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e40cad40 msgr2=0x7f04e4154b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.537+0000 7f04e9200640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e40cad40 0x7f04e4154b10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f04d0033000 tx=0x7f04d0004730 comp rx=0 tx=0).stop 2024-03-18T16:36:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.538+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 shutdown_connections 2024-03-18T16:36:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.538+0000 7f04e9200640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f04c406bbd0 0x7f04c406e090 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.538+0000 7f04e9200640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f04e415a1a0 0x7f04e4155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.538+0000 7f04e9200640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04e40cad40 0x7f04e4154b10 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.538+0000 7f04e9200640 1 --2- 172.21.15.102:0/411523882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04e40ca960 0x7f04e40cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.538+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 >> 172.21.15.102:0/411523882 conn(0x7f04e4067190 msgr2=0x7f04e41411e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.538+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 shutdown_connections 2024-03-18T16:36:57.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.538+0000 7f04e9200640 1 -- 172.21.15.102:0/411523882 wait complete. 2024-03-18T16:36:57.555 INFO:tasks.ceph:Scrubbing osd.5 2024-03-18T16:36:57.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-18T16:36:57.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.759+0000 7f57a1000640 1 Processor -- start 2024-03-18T16:36:57.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.759+0000 7f57a1000640 1 -- start start 2024-03-18T16:36:57.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.760+0000 7f57a1000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f579c0ca8a0 0x7f579c15b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.760+0000 7f57a1000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f579c15b5b0 con 0x7f579c0cac80 2024-03-18T16:36:57.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.760+0000 7f57a1000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f579c0cd080 con 0x7f579c15e5f0 2024-03-18T16:36:57.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.760+0000 7f57a1000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f579c06df80 con 0x7f579c0ca8a0 2024-03-18T16:36:57.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.761+0000 7f5799e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f579c0ca8a0 0x7f579c15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.762+0000 7f579a800640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f579c15e5f0 0x7f579c06d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:39072/0 (socket says 172.21.15.102:39072) 2024-03-18T16:36:57.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.762+0000 7f579a800640 1 -- 172.21.15.102:0/3634551375 learned_addr learned my addr 172.21.15.102:0/3634551375 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.766+0000 7f5799e00640 1 -- 172.21.15.102:0/3634551375 >> v1:172.21.15.154:6789/0 conn(0x7f579c15e5f0 legacy=0x7f579c06d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.766+0000 7f5799e00640 1 -- 172.21.15.102:0/3634551375 >> v1:172.21.15.102:6789/0 conn(0x7f579c0cac80 legacy=0x7f579c0cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.766+0000 7f5799e00640 1 -- 172.21.15.102:0/3634551375 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f579c0ccda0 con 0x7f579c0ca8a0 2024-03-18T16:36:57.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.772+0000 7f5799e00640 1 --2- 172.21.15.102:0/3634551375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f579c0ca8a0 0x7f579c15b070 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f578c0097a0 tx=0x7f578c02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d27a44b0e881d68d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:57.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.776+0000 7f5798a00640 1 -- 172.21.15.102:0/3634551375 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578c02f810 con 0x7f579c0ca8a0 2024-03-18T16:36:57.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.776+0000 7f5798a00640 1 -- 172.21.15.102:0/3634551375 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f578c02f9b0 con 0x7f579c0ca8a0 2024-03-18T16:36:57.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.776+0000 7f5798a00640 1 -- 172.21.15.102:0/3634551375 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578c02fd80 con 0x7f579c0ca8a0 2024-03-18T16:36:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.777+0000 7f57a1000640 1 -- 172.21.15.102:0/3634551375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f579c0ca8a0 msgr2=0x7f579c15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.777+0000 7f57a1000640 1 --2- 172.21.15.102:0/3634551375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f579c0ca8a0 0x7f579c15b070 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f578c0097a0 tx=0x7f578c02b9f0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.777+0000 7f57a1000640 1 -- 172.21.15.102:0/3634551375 shutdown_connections 2024-03-18T16:36:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.777+0000 7f57a1000640 1 --2- 172.21.15.102:0/3634551375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f579c0ca8a0 0x7f579c15b070 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.777+0000 7f57a1000640 1 -- 172.21.15.102:0/3634551375 >> 172.21.15.102:0/3634551375 conn(0x7f579c0b19c0 msgr2=0x7f579c0b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.778+0000 7f57a1000640 1 -- 172.21.15.102:0/3634551375 shutdown_connections 2024-03-18T16:36:57.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.778+0000 7f57a1000640 1 -- 172.21.15.102:0/3634551375 wait complete. 2024-03-18T16:36:57.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.779+0000 7f57a1000640 1 Processor -- start 2024-03-18T16:36:57.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.779+0000 7f57a1000640 1 -- start start 2024-03-18T16:36:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.780+0000 7f57a1000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f579c0ca8a0 0x7f579c0c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.780+0000 7f57a1000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f579c0cac80 0x7f579c14f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.780+0000 7f57a1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f579c15e5f0 0x7f579c0b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.780+0000 7f57a1000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f579c06cf80 con 0x7f579c15e5f0 2024-03-18T16:36:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.780+0000 7f57a1000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f579c15ea20 con 0x7f579c0cac80 2024-03-18T16:36:57.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.780+0000 7f57a1000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f579c06c930 con 0x7f579c0ca8a0 2024-03-18T16:36:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.781+0000 7f5799400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f579c0cac80 0x7f579c14f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.781+0000 7f5799e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f579c0ca8a0 0x7f579c0c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.781+0000 7f579a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f579c15e5f0 0x7f579c0b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.781+0000 7f5799400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f579c0cac80 0x7f579c14f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56376/0 (socket says 172.21.15.102:56376) 2024-03-18T16:36:57.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.781+0000 7f5799400640 1 -- 172.21.15.102:0/2641545312 learned_addr learned my addr 172.21.15.102:0/2641545312 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:57.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.785+0000 7f5799e00640 1 -- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f579c0cac80 msgr2=0x7f579c14f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.785+0000 7f5799e00640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f579c0cac80 0x7f579c14f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.785+0000 7f5799e00640 1 -- 172.21.15.102:0/2641545312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f579c15e5f0 msgr2=0x7f579c0b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.785+0000 7f5799e00640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f579c15e5f0 0x7f579c0b0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.785+0000 7f5799e00640 1 -- 172.21.15.102:0/2641545312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f579c0c7c00 con 0x7f579c0ca8a0 2024-03-18T16:36:57.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.788+0000 7f5799e00640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f579c0ca8a0 0x7f579c0c76c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f578c033000 tx=0x7f578c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.792+0000 7f5792000640 1 -- 172.21.15.102:0/2641545312 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578c046070 con 0x7f579c0ca8a0 2024-03-18T16:36:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.792+0000 7f5792000640 1 -- 172.21.15.102:0/2641545312 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f578c037c60 con 0x7f579c0ca8a0 2024-03-18T16:36:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.792+0000 7f5792000640 1 -- 172.21.15.102:0/2641545312 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578c02fe60 con 0x7f579c0ca8a0 2024-03-18T16:36:57.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.792+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f579c15a330 con 0x7f579c0ca8a0 2024-03-18T16:36:57.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.792+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f579c0c9360 con 0x7f579c0ca8a0 2024-03-18T16:36:57.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.793+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f579c0c8e30 con 0x7f579c0ca8a0 2024-03-18T16:36:57.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.824+0000 7f5792000640 1 -- 172.21.15.102:0/2641545312 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f578c049070 con 0x7f579c0ca8a0 2024-03-18T16:36:57.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.825+0000 7f5792000640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f577c06bc50 0x7f577c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.825+0000 7f5792000640 1 -- 172.21.15.102:0/2641545312 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f578c078c10 con 0x7f579c0ca8a0 2024-03-18T16:36:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.826+0000 7f5799400640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f577c06bc50 0x7f577c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.826+0000 7f5792000640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f577c075780 0x7f577c077c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:57.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.826+0000 7f5792000640 1 -- 172.21.15.102:0/2641545312 --> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f578c008eb0 con 0x7f577c075780 2024-03-18T16:36:57.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.826+0000 7f579a800640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f577c075780 0x7f577c077c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:57.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.826+0000 7f5799400640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f577c06bc50 0x7f577c06e110 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f579c15dfb0 tx=0x7f5788009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:57.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.827+0000 7f579a800640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f577c075780 0x7f577c077c20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:57.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.828+0000 7f5792000640 1 -- 172.21.15.102:0/2641545312 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f578c076e10 con 0x7f579c0ca8a0 2024-03-18T16:36:57.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.828+0000 7f5792000640 1 -- 172.21.15.102:0/2641545312 <== osd.5 v2:172.21.15.154:6802/1669301852 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f578c008eb0 con 0x7f577c075780 2024-03-18T16:36:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.856+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 --> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f579c074fd0 con 0x7f577c075780 2024-03-18T16:36:57.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.857+0000 7f5792000640 1 -- 172.21.15.102:0/2641545312 <== osd.5 v2:172.21.15.154:6802/1669301852 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f579c074fd0 con 0x7f577c075780 2024-03-18T16:36:57.858 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:36:57.859 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T16:36:57.859 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f577c075780 msgr2=0x7f577c077c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6802/1669301852,v1:172.21.15.154:6803/1669301852] conn(0x7f577c075780 0x7f577c077c20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f577c06bc50 msgr2=0x7f577c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f577c06bc50 0x7f577c06e110 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f579c15dfb0 tx=0x7f5788009040 comp rx=0 tx=0).stop 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f579c0ca8a0 msgr2=0x7f579c0c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 --2- 172.21.15.102:0/2641545312 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f579c0ca8a0 0x7f579c0c76c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f578c033000 tx=0x7f578c004730 comp rx=0 tx=0).stop 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f579a800640 1 -- 172.21.15.102:0/2641545312 reap_dead start 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 shutdown_connections 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 >> 172.21.15.102:0/2641545312 conn(0x7f579c0b19c0 msgr2=0x7f579c0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 shutdown_connections 2024-03-18T16:36:57.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:57.858+0000 7f57a1000640 1 -- 172.21.15.102:0/2641545312 wait complete. 2024-03-18T16:36:57.878 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-18T16:36:58.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.045+0000 7fdf99200640 1 Processor -- start 2024-03-18T16:36:58.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.045+0000 7fdf99200640 1 -- start start 2024-03-18T16:36:58.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.046+0000 7fdf99200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdf94141650 0x7fdf94158420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.046+0000 7fdf99200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf940c72d0 con 0x7fdf940c4fa0 2024-03-18T16:36:58.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.046+0000 7fdf99200640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf94158ad0 con 0x7fdf9415c3a0 2024-03-18T16:36:58.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.046+0000 7fdf99200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fdf94159370 con 0x7fdf94141650 2024-03-18T16:36:58.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.047+0000 7fdf92a00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fdf9415c3a0 0x7fdf940c6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:39074/0 (socket says 172.21.15.102:39074) 2024-03-18T16:36:58.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.047+0000 7fdf92a00640 1 -- 172.21.15.102:0/1805616403 learned_addr learned my addr 172.21.15.102:0/1805616403 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:58.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.047+0000 7fdf93400640 1 --2- 172.21.15.102:0/1805616403 >> v2:172.21.15.102:3300/0 conn(0x7fdf94141650 0x7fdf94158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.050+0000 7fdf92000640 1 -- 172.21.15.102:0/1805616403 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696058095 0 0) 0x7fdf94158ad0 con 0x7fdf9415c3a0 2024-03-18T16:36:58.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.050+0000 7fdf92000640 1 -- 172.21.15.102:0/1805616403 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf78003520 con 0x7fdf9415c3a0 2024-03-18T16:36:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.051+0000 7fdf93400640 1 -- 172.21.15.102:0/1805616403 >> v1:172.21.15.154:6789/0 conn(0x7fdf9415c3a0 legacy=0x7fdf940c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.051+0000 7fdf93400640 1 -- 172.21.15.102:0/1805616403 >> v1:172.21.15.102:6789/0 conn(0x7fdf940c4fa0 legacy=0x7fdf94162cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.051+0000 7fdf93400640 1 -- 172.21.15.102:0/1805616403 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf94158db0 con 0x7fdf94141650 2024-03-18T16:36:58.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.057+0000 7fdf93400640 1 --2- 172.21.15.102:0/1805616403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf94141650 0x7fdf94158420 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdf840097a0 tx=0x7fdf8402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec3e979651e8115 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:58.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.062+0000 7fdf92000640 1 -- 172.21.15.102:0/1805616403 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf84034070 con 0x7fdf94141650 2024-03-18T16:36:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.062+0000 7fdf92000640 1 -- 172.21.15.102:0/1805616403 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf84030610 con 0x7fdf94141650 2024-03-18T16:36:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.062+0000 7fdf92000640 1 -- 172.21.15.102:0/1805616403 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf840309e0 con 0x7fdf94141650 2024-03-18T16:36:58.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.063+0000 7fdf99200640 1 -- 172.21.15.102:0/1805616403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf94141650 msgr2=0x7fdf94158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.063+0000 7fdf99200640 1 --2- 172.21.15.102:0/1805616403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf94141650 0x7fdf94158420 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdf840097a0 tx=0x7fdf8402b960 comp rx=0 tx=0).stop 2024-03-18T16:36:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.063+0000 7fdf99200640 1 -- 172.21.15.102:0/1805616403 shutdown_connections 2024-03-18T16:36:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.063+0000 7fdf99200640 1 --2- 172.21.15.102:0/1805616403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf94141650 0x7fdf94158420 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.063+0000 7fdf99200640 1 -- 172.21.15.102:0/1805616403 >> 172.21.15.102:0/1805616403 conn(0x7fdf940cb890 msgr2=0x7fdf940cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.063+0000 7fdf99200640 1 -- 172.21.15.102:0/1805616403 shutdown_connections 2024-03-18T16:36:58.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.063+0000 7fdf99200640 1 -- 172.21.15.102:0/1805616403 wait complete. 2024-03-18T16:36:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.065+0000 7fdf99200640 1 Processor -- start 2024-03-18T16:36:58.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.065+0000 7fdf99200640 1 -- start start 2024-03-18T16:36:58.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.066+0000 7fdf99200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf940c4fa0 0x7fdf941558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.066+0000 7fdf99200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf94141650 0x7fdf941387e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.066+0000 7fdf99200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9415c3a0 0x7fdf9413e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.066+0000 7fdf99200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fdf940c66a0 con 0x7fdf9415c3a0 2024-03-18T16:36:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.066+0000 7fdf99200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf940c63c0 con 0x7fdf940c4fa0 2024-03-18T16:36:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.066+0000 7fdf99200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9415c7c0 con 0x7fdf94141650 2024-03-18T16:36:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.067+0000 7fdf93400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf940c4fa0 0x7fdf941558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.067+0000 7fdf92a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf94141650 0x7fdf941387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.067+0000 7fdf93e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9415c3a0 0x7fdf9413e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.067+0000 7fdf93400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf940c4fa0 0x7fdf941558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56392/0 (socket says 172.21.15.102:56392) 2024-03-18T16:36:58.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.067+0000 7fdf93400640 1 -- 172.21.15.102:0/3040450370 learned_addr learned my addr 172.21.15.102:0/3040450370 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.071+0000 7fdf92a00640 1 -- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf940c4fa0 msgr2=0x7fdf941558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.071+0000 7fdf92a00640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf940c4fa0 0x7fdf941558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.071+0000 7fdf92a00640 1 -- 172.21.15.102:0/3040450370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9415c3a0 msgr2=0x7fdf9413e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.071+0000 7fdf92a00640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9415c3a0 0x7fdf9413e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.071+0000 7fdf92a00640 1 -- 172.21.15.102:0/3040450370 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9413d250 con 0x7fdf94141650 2024-03-18T16:36:58.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.073+0000 7fdf92a00640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf94141650 0x7fdf941387e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000c3f0 tx=0x7fdf80001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:58.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.075+0000 7fdf8b400640 1 -- 172.21.15.102:0/3040450370 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80017070 con 0x7fdf94141650 2024-03-18T16:36:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.075+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf94157e70 con 0x7fdf94141650 2024-03-18T16:36:58.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.075+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9413cd40 con 0x7fdf94141650 2024-03-18T16:36:58.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.077+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf94074fd0 con 0x7fdf94141650 2024-03-18T16:36:58.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.101+0000 7fdf8b400640 1 -- 172.21.15.102:0/3040450370 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf80005ce0 con 0x7fdf94141650 2024-03-18T16:36:58.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.101+0000 7fdf8b400640 1 -- 172.21.15.102:0/3040450370 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80021060 con 0x7fdf94141650 2024-03-18T16:36:58.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.103+0000 7fdf8b400640 1 -- 172.21.15.102:0/3040450370 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdf8001a070 con 0x7fdf94141650 2024-03-18T16:36:58.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.104+0000 7fdf8b400640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdf7406bbd0 0x7fdf7406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.104+0000 7fdf8b400640 1 -- 172.21.15.102:0/3040450370 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fdf80056ac0 con 0x7fdf94141650 2024-03-18T16:36:58.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.104+0000 7fdf93400640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdf7406bbd0 0x7fdf7406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.105+0000 7fdf93400640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdf7406bbd0 0x7fdf7406e090 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fdf84009560 tx=0x7fdf8403b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:58.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.371+0000 7fdf8b400640 1 -- 172.21.15.102:0/3040450370 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdf94157e70 con 0x7fdf94141650 2024-03-18T16:36:58.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.573+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fdf9413cf90 con 0x7fdf7406bbd0 2024-03-18T16:36:58.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.574+0000 7fdf8b400640 1 -- 172.21.15.102:0/3040450370 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fdf9413cf90 con 0x7fdf7406bbd0 2024-03-18T16:36:58.575 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 5 to deep-scrub 2024-03-18T16:36:58.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdf7406bbd0 msgr2=0x7fdf7406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdf7406bbd0 0x7fdf7406e090 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fdf84009560 tx=0x7fdf8403b040 comp rx=0 tx=0).stop 2024-03-18T16:36:58.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf94141650 msgr2=0x7fdf941387e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf94141650 0x7fdf941387e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000c3f0 tx=0x7fdf80001e70 comp rx=0 tx=0).stop 2024-03-18T16:36:58.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 shutdown_connections 2024-03-18T16:36:58.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fdf7406bbd0 0x7fdf7406e090 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf9415c3a0 0x7fdf9413e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fdf94141650 0x7fdf941387e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 --2- 172.21.15.102:0/3040450370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf940c4fa0 0x7fdf941558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 >> 172.21.15.102:0/3040450370 conn(0x7fdf940cb890 msgr2=0x7fdf94072d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:58.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.577+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 shutdown_connections 2024-03-18T16:36:58.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.578+0000 7fdf99200640 1 -- 172.21.15.102:0/3040450370 wait complete. 2024-03-18T16:36:58.598 INFO:tasks.ceph:Scrubbing osd.6 2024-03-18T16:36:58.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-18T16:36:58.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.782+0000 7f0717200640 1 Processor -- start 2024-03-18T16:36:58.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.782+0000 7f0717200640 1 -- start start 2024-03-18T16:36:58.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.783+0000 7f0717200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07100c6df0 0x7f07100c5410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.783+0000 7f0717200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f071015e430 0x7f071006e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.784+0000 7f0717200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07100c5a70 con 0x7f0710154680 2024-03-18T16:36:58.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.784+0000 7f0717200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f07100c5d50 con 0x7f07100c6df0 2024-03-18T16:36:58.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.784+0000 7f0717200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f07100c6030 con 0x7f071015e430 2024-03-18T16:36:58.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.784+0000 7f0715200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f071015e430 0x7f071006e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.785+0000 7f0715200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f071015e430 0x7f071006e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56400/0 (socket says 172.21.15.102:56400) 2024-03-18T16:36:58.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.785+0000 7f0715200640 1 -- 172.21.15.102:0/3434294959 learned_addr learned my addr 172.21.15.102:0/3434294959 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:58.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.788+0000 7f0715200640 1 -- 172.21.15.102:0/3434294959 >> v2:172.21.15.102:3300/0 conn(0x7f07100c6df0 msgr2=0x7f07100c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:58.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.788+0000 7f0715200640 1 --2- 172.21.15.102:0/3434294959 >> v2:172.21.15.102:3300/0 conn(0x7f07100c6df0 0x7f07100c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.788+0000 7f0715200640 1 -- 172.21.15.102:0/3434294959 >> v1:172.21.15.102:6789/0 conn(0x7f0710154680 legacy=0x7f07100c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:58.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.788+0000 7f0715200640 1 -- 172.21.15.102:0/3434294959 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07100c6310 con 0x7f071015e430 2024-03-18T16:36:58.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.792+0000 7f0715200640 1 --2- 172.21.15.102:0/3434294959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f071015e430 0x7f071006e720 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f07100cdbe0 tx=0x7f070802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96044a360f397b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:58.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.796+0000 7f070f400640 1 -- 172.21.15.102:0/3434294959 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070802cb20 con 0x7f071015e430 2024-03-18T16:36:58.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.799+0000 7f070f400640 1 -- 172.21.15.102:0/3434294959 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f070802ccc0 con 0x7f071015e430 2024-03-18T16:36:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.799+0000 7f070f400640 1 -- 172.21.15.102:0/3434294959 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708034610 con 0x7f071015e430 2024-03-18T16:36:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.800+0000 7f0717200640 1 -- 172.21.15.102:0/3434294959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f071015e430 msgr2=0x7f071006e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.800+0000 7f0717200640 1 --2- 172.21.15.102:0/3434294959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f071015e430 0x7f071006e720 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f07100cdbe0 tx=0x7f070802bf80 comp rx=0 tx=0).stop 2024-03-18T16:36:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.800+0000 7f0717200640 1 -- 172.21.15.102:0/3434294959 shutdown_connections 2024-03-18T16:36:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.800+0000 7f0717200640 1 --2- 172.21.15.102:0/3434294959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f071015e430 0x7f071006e720 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.800+0000 7f0717200640 1 --2- 172.21.15.102:0/3434294959 >> v2:172.21.15.102:3300/0 conn(0x7f07100c6df0 0x7f07100c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.800+0000 7f0717200640 1 -- 172.21.15.102:0/3434294959 >> 172.21.15.102:0/3434294959 conn(0x7f07100ccbf0 msgr2=0x7f07100cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.801+0000 7f0717200640 1 -- 172.21.15.102:0/3434294959 shutdown_connections 2024-03-18T16:36:58.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.801+0000 7f0717200640 1 -- 172.21.15.102:0/3434294959 wait complete. 2024-03-18T16:36:58.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.803+0000 7f0717200640 1 Processor -- start 2024-03-18T16:36:58.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.803+0000 7f0717200640 1 -- start start 2024-03-18T16:36:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.805+0000 7f0717200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07100c6df0 0x7f07101f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.805+0000 7f0717200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0710154680 0x7f07101f0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.805+0000 7f0717200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f071015e430 0x7f071015ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.805+0000 7f0717200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f071015e840 con 0x7f071015e430 2024-03-18T16:36:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.805+0000 7f0717200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07100c68d0 con 0x7f07100c6df0 2024-03-18T16:36:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.805+0000 7f0717200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f071015e9c0 con 0x7f0710154680 2024-03-18T16:36:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.805+0000 7f070fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07100c6df0 0x7f07101f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.806+0000 7f0707e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0710154680 0x7f07101f0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.806+0000 7f070fe00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07100c6df0 0x7f07101f46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56408/0 (socket says 172.21.15.102:56408) 2024-03-18T16:36:58.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.806+0000 7f070fe00640 1 -- 172.21.15.102:0/3690957837 learned_addr learned my addr 172.21.15.102:0/3690957837 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:58.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.806+0000 7f0715200640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f071015e430 0x7f071015ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.809+0000 7f070fe00640 1 -- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0710154680 msgr2=0x7f07101f0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.809+0000 7f070fe00640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f0710154680 0x7f07101f0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.809+0000 7f070fe00640 1 -- 172.21.15.102:0/3690957837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f071015e430 msgr2=0x7f071015ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.809+0000 7f070fe00640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f071015e430 0x7f071015ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.809+0000 7f070fe00640 1 -- 172.21.15.102:0/3690957837 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07101f4be0 con 0x7f07100c6df0 2024-03-18T16:36:58.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.812+0000 7f070fe00640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07100c6df0 0x7f07101f46a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f070000b410 tx=0x7f070000b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:58.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.816+0000 7f070cc00640 1 -- 172.21.15.102:0/3690957837 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700015020 con 0x7f07100c6df0 2024-03-18T16:36:58.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.816+0000 7f070cc00640 1 -- 172.21.15.102:0/3690957837 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0700011830 con 0x7f07100c6df0 2024-03-18T16:36:58.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.816+0000 7f070cc00640 1 -- 172.21.15.102:0/3690957837 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700011c20 con 0x7f07100c6df0 2024-03-18T16:36:58.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.816+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071015b0d0 con 0x7f07100c6df0 2024-03-18T16:36:58.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.816+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f071006bda0 con 0x7f07100c6df0 2024-03-18T16:36:58.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.817+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f071006cf20 con 0x7f07100c6df0 2024-03-18T16:36:58.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.841+0000 7f070cc00640 1 -- 172.21.15.102:0/3690957837 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0700018020 con 0x7f07100c6df0 2024-03-18T16:36:58.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.842+0000 7f070cc00640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f06ec06bc20 0x7f06ec06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.842+0000 7f070cc00640 1 -- 172.21.15.102:0/3690957837 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f070008aeb0 con 0x7f07100c6df0 2024-03-18T16:36:58.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.842+0000 7f0707e00640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f06ec06bc20 0x7f06ec06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.842+0000 7f070cc00640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f06ec075750 0x7f06ec077bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:58.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.842+0000 7f070cc00640 1 -- 172.21.15.102:0/3690957837 --> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f070000de50 con 0x7f06ec075750 2024-03-18T16:36:58.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.842+0000 7f070cc00640 1 -- 172.21.15.102:0/3690957837 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f07000550a0 con 0x7f07100c6df0 2024-03-18T16:36:58.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.843+0000 7f0715200640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f06ec075750 0x7f06ec077bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:58.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.843+0000 7f0707e00640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f06ec06bc20 0x7f06ec06e0e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f071006bc00 tx=0x7f06f8009060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:58.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.843+0000 7f0715200640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f06ec075750 0x7f06ec077bf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:58.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.844+0000 7f070cc00640 1 -- 172.21.15.102:0/3690957837 <== osd.6 v2:172.21.15.154:6810/1929522630 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f070000de50 con 0x7f06ec075750 2024-03-18T16:36:58.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.859+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 --> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0710067cb0 con 0x7f06ec075750 2024-03-18T16:36:58.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.860+0000 7f070cc00640 1 -- 172.21.15.102:0/3690957837 <== osd.6 v2:172.21.15.154:6810/1929522630 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0710067cb0 con 0x7f06ec075750 2024-03-18T16:36:58.861 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:36:58.862 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T16:36:58.862 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.860+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f06ec075750 msgr2=0x7f06ec077bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.860+0000 7f0717200640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6810/1929522630,v1:172.21.15.154:6811/1929522630] conn(0x7f06ec075750 0x7f06ec077bf0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.861+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f06ec06bc20 msgr2=0x7f06ec06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.861+0000 7f0717200640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f06ec06bc20 0x7f06ec06e0e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f071006bc00 tx=0x7f06f8009060 comp rx=0 tx=0).stop 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.861+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07100c6df0 msgr2=0x7f07101f46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.861+0000 7f0717200640 1 --2- 172.21.15.102:0/3690957837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07100c6df0 0x7f07101f46a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f070000b410 tx=0x7f070000b8d0 comp rx=0 tx=0).stop 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.861+0000 7f0715200640 1 -- 172.21.15.102:0/3690957837 reap_dead start 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.861+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 shutdown_connections 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.861+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 >> 172.21.15.102:0/3690957837 conn(0x7f07100ccbf0 msgr2=0x7f071006e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.861+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 shutdown_connections 2024-03-18T16:36:58.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:58.861+0000 7f0717200640 1 -- 172.21.15.102:0/3690957837 wait complete. 2024-03-18T16:36:58.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-18T16:36:59.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.074+0000 7f43f7000640 1 Processor -- start 2024-03-18T16:36:59.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.074+0000 7f43f7000640 1 -- start start 2024-03-18T16:36:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.076+0000 7f43f7000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f43f0154690 0x7f43f015e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.076+0000 7f43f7000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f015a710 con 0x7f43f0071e20 2024-03-18T16:36:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.076+0000 7f43f7000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f015a9f0 con 0x7f43f0154a70 2024-03-18T16:36:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.076+0000 7f43f7000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f43f015acd0 con 0x7f43f0154690 2024-03-18T16:36:59.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.077+0000 7f43efe00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f43f0154690 0x7f43f015e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.077+0000 7f43ef400640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f43f0154a70 0x7f43f00c98d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:39078/0 (socket says 172.21.15.102:39078) 2024-03-18T16:36:59.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.077+0000 7f43ef400640 1 -- 172.21.15.102:0/735440910 learned_addr learned my addr 172.21.15.102:0/735440910 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:36:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.080+0000 7f43efe00640 1 -- 172.21.15.102:0/735440910 >> v1:172.21.15.154:6789/0 conn(0x7f43f0154a70 legacy=0x7f43f00c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.080+0000 7f43efe00640 1 -- 172.21.15.102:0/735440910 >> v1:172.21.15.102:6789/0 conn(0x7f43f0071e20 legacy=0x7f43f00ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.080+0000 7f43efe00640 1 -- 172.21.15.102:0/735440910 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f015afb0 con 0x7f43f0154690 2024-03-18T16:36:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.084+0000 7f43efe00640 1 --2- 172.21.15.102:0/735440910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 0x7f43f015e6c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f43e00097a0 tx=0x7f43e002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e44d394702a59fd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:59.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.089+0000 7f43eea00640 1 -- 172.21.15.102:0/735440910 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0035020 con 0x7f43f0154690 2024-03-18T16:36:59.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.089+0000 7f43eea00640 1 -- 172.21.15.102:0/735440910 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e0031830 con 0x7f43f0154690 2024-03-18T16:36:59.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.089+0000 7f43eea00640 1 -- 172.21.15.102:0/735440910 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0031c00 con 0x7f43f0154690 2024-03-18T16:36:59.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.089+0000 7f43f7000640 1 -- 172.21.15.102:0/735440910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 msgr2=0x7f43f015e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.089+0000 7f43f7000640 1 --2- 172.21.15.102:0/735440910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 0x7f43f015e6c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f43e00097a0 tx=0x7f43e002b8d0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.089+0000 7f43f7000640 1 -- 172.21.15.102:0/735440910 shutdown_connections 2024-03-18T16:36:59.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.089+0000 7f43f7000640 1 --2- 172.21.15.102:0/735440910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 0x7f43f015e6c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.089+0000 7f43f7000640 1 -- 172.21.15.102:0/735440910 >> 172.21.15.102:0/735440910 conn(0x7f43f00700d0 msgr2=0x7f43f006b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:59.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.090+0000 7f43f7000640 1 -- 172.21.15.102:0/735440910 shutdown_connections 2024-03-18T16:36:59.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.090+0000 7f43f7000640 1 -- 172.21.15.102:0/735440910 wait complete. 2024-03-18T16:36:59.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.092+0000 7f43f7000640 1 Processor -- start 2024-03-18T16:36:59.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.092+0000 7f43f7000640 1 -- start start 2024-03-18T16:36:59.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.094+0000 7f43f7000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f0071e20 0x7f43f0155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.094+0000 7f43f7000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 0x7f43f0156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.094+0000 7f43f7000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43f0154a70 0x7f43f01e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.094+0000 7f43f7000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f43f00cafb0 con 0x7f43f0071e20 2024-03-18T16:36:59.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.094+0000 7f43f7000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f43f00c9fe0 con 0x7f43f0154690 2024-03-18T16:36:59.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.094+0000 7f43f7000640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f43f00cb130 con 0x7f43f0154a70 2024-03-18T16:36:59.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.094+0000 7f43ef400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 0x7f43f0156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.095+0000 7f43f5000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43f0154a70 0x7f43f01e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.095+0000 7f43efe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f0071e20 0x7f43f0155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.095+0000 7f43ef400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 0x7f43f0156740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56428/0 (socket says 172.21.15.102:56428) 2024-03-18T16:36:59.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.095+0000 7f43ef400640 1 -- 172.21.15.102:0/2624926174 learned_addr learned my addr 172.21.15.102:0/2624926174 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.098+0000 7f43ef400640 1 -- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43f0154a70 msgr2=0x7f43f01e5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.098+0000 7f43ef400640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43f0154a70 0x7f43f01e5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.098+0000 7f43ef400640 1 -- 172.21.15.102:0/2624926174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f0071e20 msgr2=0x7f43f0155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.098+0000 7f43ef400640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f0071e20 0x7f43f0155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.098+0000 7f43ef400640 1 -- 172.21.15.102:0/2624926174 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f01f1130 con 0x7f43f0154690 2024-03-18T16:36:59.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.101+0000 7f43ef400640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 0x7f43f0156740 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00c440 tx=0x7f43dc001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:59.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.105+0000 7f43e7e00640 1 -- 172.21.15.102:0/2624926174 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc017070 con 0x7f43f0154690 2024-03-18T16:36:59.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.105+0000 7f43e7e00640 1 -- 172.21.15.102:0/2624926174 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43dc005ce0 con 0x7f43f0154690 2024-03-18T16:36:59.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.106+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f01f0b70 con 0x7f43f0154690 2024-03-18T16:36:59.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.106+0000 7f43e7e00640 1 -- 172.21.15.102:0/2624926174 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0072f0 con 0x7f43f0154690 2024-03-18T16:36:59.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.106+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f0155190 con 0x7f43f0154690 2024-03-18T16:36:59.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.107+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f015e6c0 con 0x7f43f0154690 2024-03-18T16:36:59.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.131+0000 7f43e7e00640 1 -- 172.21.15.102:0/2624926174 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f43dc01a070 con 0x7f43f0154690 2024-03-18T16:36:59.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.133+0000 7f43e7e00640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43d006bc20 0x7f43d006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.133+0000 7f43efe00640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43d006bc20 0x7f43d006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.133+0000 7f43e7e00640 1 -- 172.21.15.102:0/2624926174 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f43dc056760 con 0x7f43f0154690 2024-03-18T16:36:59.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.134+0000 7f43efe00640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43d006bc20 0x7f43d006e0e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f43e0036000 tx=0x7f43e003c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:59.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.410+0000 7f43e7e00640 1 -- 172.21.15.102:0/2624926174 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f43f015e6c0 con 0x7f43f0154690 2024-03-18T16:36:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.611+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f43f00730f0 con 0x7f43d006bc20 2024-03-18T16:36:59.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.611+0000 7f43e7e00640 1 -- 172.21.15.102:0/2624926174 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f43f00730f0 con 0x7f43d006bc20 2024-03-18T16:36:59.613 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 6 to deep-scrub 2024-03-18T16:36:59.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43d006bc20 msgr2=0x7f43d006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43d006bc20 0x7f43d006e0e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f43e0036000 tx=0x7f43e003c040 comp rx=0 tx=0).stop 2024-03-18T16:36:59.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 msgr2=0x7f43f0156740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 0x7f43f0156740 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f43dc00c440 tx=0x7f43dc001e70 comp rx=0 tx=0).stop 2024-03-18T16:36:59.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 shutdown_connections 2024-03-18T16:36:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f43d006bc20 0x7f43d006e0e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f43f0154a70 0x7f43f01e5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43f0154690 0x7f43f0156740 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 --2- 172.21.15.102:0/2624926174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f43f0071e20 0x7f43f0155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 >> 172.21.15.102:0/2624926174 conn(0x7f43f00700d0 msgr2=0x7f43f00ca8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 shutdown_connections 2024-03-18T16:36:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.615+0000 7f43f7000640 1 -- 172.21.15.102:0/2624926174 wait complete. 2024-03-18T16:36:59.634 INFO:tasks.ceph:Scrubbing osd.7 2024-03-18T16:36:59.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-18T16:36:59.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.815+0000 7f0557200640 1 Processor -- start 2024-03-18T16:36:59.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.815+0000 7f0557200640 1 -- start start 2024-03-18T16:36:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.816+0000 7f0557200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05580ca5b0 0x7f05580ca990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.816+0000 7f0557200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f05580cec80 0x7f05580ce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.816+0000 7f0557200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f055815a6a0 0x7f055815ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.816+0000 7f0557200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f05580cea70 con 0x7f055815a6a0 2024-03-18T16:36:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.817+0000 7f0557200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f05580cc510 con 0x7f05580cec80 2024-03-18T16:36:59.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.817+0000 7f0557200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f055815b070 con 0x7f05580ca5b0 2024-03-18T16:36:59.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.817+0000 7f0556800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f055815a6a0 0x7f055815ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.818+0000 7f0555400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f05580cec80 0x7f05580ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.818+0000 7f0556800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f055815a6a0 0x7f055815ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33880/0 (socket says 172.21.15.102:33880) 2024-03-18T16:36:59.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.818+0000 7f0556800640 1 -- 172.21.15.102:0/772051966 learned_addr learned my addr 172.21.15.102:0/772051966 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:59.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.820+0000 7f0555400640 1 -- 172.21.15.102:0/772051966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05580ca5b0 msgr2=0x7f05580ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:59.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.820+0000 7f0555400640 1 --2- 172.21.15.102:0/772051966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05580ca5b0 0x7f05580ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.820+0000 7f0555400640 1 -- 172.21.15.102:0/772051966 >> v2:172.21.15.102:3300/0 conn(0x7f055815a6a0 msgr2=0x7f055815ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:36:59.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.820+0000 7f0555400640 1 --2- 172.21.15.102:0/772051966 >> v2:172.21.15.102:3300/0 conn(0x7f055815a6a0 0x7f055815ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.820+0000 7f0555400640 1 -- 172.21.15.102:0/772051966 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05580cde00 con 0x7f05580cec80 2024-03-18T16:36:59.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.823+0000 7f0555400640 1 --2- 172.21.15.102:0/772051966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580cec80 0x7f05580ce530 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f05580d0d30 tx=0x7f054402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dd153793c3bbb0c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:59.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.826+0000 7f0554a00640 1 -- 172.21.15.102:0/772051966 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544035020 con 0x7f05580cec80 2024-03-18T16:36:59.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.827+0000 7f0554a00640 1 -- 172.21.15.102:0/772051966 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0544031830 con 0x7f05580cec80 2024-03-18T16:36:59.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.827+0000 7f0554a00640 1 -- 172.21.15.102:0/772051966 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544031c00 con 0x7f05580cec80 2024-03-18T16:36:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.827+0000 7f0557200640 1 -- 172.21.15.102:0/772051966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580cec80 msgr2=0x7f05580ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.827+0000 7f0557200640 1 --2- 172.21.15.102:0/772051966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580cec80 0x7f05580ce530 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f05580d0d30 tx=0x7f054402b8d0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.827+0000 7f0557200640 1 -- 172.21.15.102:0/772051966 shutdown_connections 2024-03-18T16:36:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.827+0000 7f0557200640 1 --2- 172.21.15.102:0/772051966 >> v2:172.21.15.102:3300/0 conn(0x7f055815a6a0 0x7f055815ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.827+0000 7f0557200640 1 --2- 172.21.15.102:0/772051966 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580cec80 0x7f05580ce530 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.827+0000 7f0557200640 1 --2- 172.21.15.102:0/772051966 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05580ca5b0 0x7f05580ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.827+0000 7f0557200640 1 -- 172.21.15.102:0/772051966 >> 172.21.15.102:0/772051966 conn(0x7f05580b1740 msgr2=0x7f05580b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.828+0000 7f0557200640 1 -- 172.21.15.102:0/772051966 shutdown_connections 2024-03-18T16:36:59.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.828+0000 7f0557200640 1 -- 172.21.15.102:0/772051966 wait complete. 2024-03-18T16:36:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.829+0000 7f0557200640 1 Processor -- start 2024-03-18T16:36:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.829+0000 7f0557200640 1 -- start start 2024-03-18T16:36:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.830+0000 7f0557200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580ca5b0 0x7f05580c4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.830+0000 7f0557200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05580cec80 0x7f05581578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.830+0000 7f0557200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815a6a0 0x7f05581552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.830+0000 7f0557200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f05580cca60 con 0x7f055815a6a0 2024-03-18T16:36:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.830+0000 7f0557200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f05580cc780 con 0x7f05580ca5b0 2024-03-18T16:36:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.830+0000 7f0557200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f05580ccd40 con 0x7f05580cec80 2024-03-18T16:36:59.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.831+0000 7f0555e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580ca5b0 0x7f05580c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.831+0000 7f0555e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580ca5b0 0x7f05580c4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56442/0 (socket says 172.21.15.102:56442) 2024-03-18T16:36:59.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.831+0000 7f0555e00640 1 -- 172.21.15.102:0/1977013633 learned_addr learned my addr 172.21.15.102:0/1977013633 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:36:59.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.831+0000 7f0556800640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815a6a0 0x7f05581552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.835+0000 7f0555e00640 1 -- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05580cec80 msgr2=0x7f05581578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.835+0000 7f0555e00640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f05580cec80 0x7f05581578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.835+0000 7f0555e00640 1 -- 172.21.15.102:0/1977013633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815a6a0 msgr2=0x7f05581552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.835+0000 7f0555e00640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815a6a0 0x7f05581552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.835+0000 7f0555e00640 1 -- 172.21.15.102:0/1977013633 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05580c36a0 con 0x7f05580ca5b0 2024-03-18T16:36:59.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.837+0000 7f0555e00640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580ca5b0 0x7f05580c4830 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f054800b4a0 tx=0x7f054800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.841+0000 7f054e000640 1 -- 172.21.15.102:0/1977013633 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548014070 con 0x7f05580ca5b0 2024-03-18T16:36:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.841+0000 7f054e000640 1 -- 172.21.15.102:0/1977013633 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0548010610 con 0x7f05580ca5b0 2024-03-18T16:36:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.841+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05580c3910 con 0x7f05580ca5b0 2024-03-18T16:36:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.841+0000 7f054e000640 1 -- 172.21.15.102:0/1977013633 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0548010a00 con 0x7f05580ca5b0 2024-03-18T16:36:59.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.841+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05580c5ff0 con 0x7f05580ca5b0 2024-03-18T16:36:59.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.842+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0558068290 con 0x7f05580ca5b0 2024-03-18T16:36:59.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.861+0000 7f054e000640 1 -- 172.21.15.102:0/1977013633 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0548017070 con 0x7f05580ca5b0 2024-03-18T16:36:59.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.862+0000 7f054e000640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f053806bc20 0x7f053806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.862+0000 7f0555400640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f053806bc20 0x7f053806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.863+0000 7f0555400640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f053806bc20 0x7f053806e0e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0544036000 tx=0x7f0544030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:59.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.865+0000 7f054e000640 1 -- 172.21.15.102:0/1977013633 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f0548056f00 con 0x7f05580ca5b0 2024-03-18T16:36:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.865+0000 7f054e000640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f05380756c0 0x7f0538077b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:36:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.865+0000 7f054e000640 1 -- 172.21.15.102:0/1977013633 --> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0548006820 con 0x7f05380756c0 2024-03-18T16:36:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.866+0000 7f0556800640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f05380756c0 0x7f0538077b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:36:59.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.866+0000 7f0556800640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f05380756c0 0x7f0538077b60 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:36:59.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.867+0000 7f054e000640 1 -- 172.21.15.102:0/1977013633 <== osd.7 v2:172.21.15.154:6818/2630476699 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0548006820 con 0x7f05380756c0 2024-03-18T16:36:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.883+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 --> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f055804f540 con 0x7f05380756c0 2024-03-18T16:36:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.883+0000 7f054e000640 1 -- 172.21.15.102:0/1977013633 <== osd.7 v2:172.21.15.154:6818/2630476699 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f055804f540 con 0x7f05380756c0 2024-03-18T16:36:59.885 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-18T16:36:59.885 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-18T16:36:59.885 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f05380756c0 msgr2=0x7f0538077b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6818/2630476699,v1:172.21.15.154:6819/2630476699] conn(0x7f05380756c0 0x7f0538077b60 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f053806bc20 msgr2=0x7f053806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f053806bc20 0x7f053806e0e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0544036000 tx=0x7f0544030000 comp rx=0 tx=0).stop 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580ca5b0 msgr2=0x7f05580c4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 --2- 172.21.15.102:0/1977013633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f05580ca5b0 0x7f05580c4830 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f054800b4a0 tx=0x7f054800b960 comp rx=0 tx=0).stop 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0556800640 1 -- 172.21.15.102:0/1977013633 reap_dead start 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 shutdown_connections 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 >> 172.21.15.102:0/1977013633 conn(0x7f05580b1740 msgr2=0x7f0558154220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 shutdown_connections 2024-03-18T16:36:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:36:59.884+0000 7f0557200640 1 -- 172.21.15.102:0/1977013633 wait complete. 2024-03-18T16:36:59.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-18T16:37:00.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.074+0000 7f1476e00640 1 Processor -- start 2024-03-18T16:37:00.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.074+0000 7f1476e00640 1 -- start start 2024-03-18T16:37:00.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.075+0000 7f1476e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700cfe70 0x7f147015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.075+0000 7f1476e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14700d0250 0x7f14700c65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.075+0000 7f1476e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14700cabd0 con 0x7f14700cc2f0 2024-03-18T16:37:00.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.075+0000 7f1476e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f14700caea0 con 0x7f14700d0250 2024-03-18T16:37:00.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.075+0000 7f1476e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f14700cb180 con 0x7f14700cfe70 2024-03-18T16:37:00.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.076+0000 7f1475a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700cfe70 0x7f147015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.076+0000 7f1475000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f14700d0250 0x7f14700c65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.076+0000 7f1475a00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700cfe70 0x7f147015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38254/0 (socket says 172.21.15.102:38254) 2024-03-18T16:37:00.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.076+0000 7f1475a00640 1 -- 172.21.15.102:0/2958884468 learned_addr learned my addr 172.21.15.102:0/2958884468 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:37:00.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.079+0000 7f1475a00640 1 -- 172.21.15.102:0/2958884468 >> v2:172.21.15.102:3300/0 conn(0x7f14700d0250 msgr2=0x7f14700c65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:37:00.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.079+0000 7f1475a00640 1 --2- 172.21.15.102:0/2958884468 >> v2:172.21.15.102:3300/0 conn(0x7f14700d0250 0x7f14700c65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.079+0000 7f1475a00640 1 -- 172.21.15.102:0/2958884468 >> v1:172.21.15.102:6789/0 conn(0x7f14700cc2f0 legacy=0x7f14700cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.079+0000 7f1475a00640 1 -- 172.21.15.102:0/2958884468 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14700cb460 con 0x7f14700cfe70 2024-03-18T16:37:00.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.081+0000 7f1475a00640 1 --2- 172.21.15.102:0/2958884468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700cfe70 0x7f147015ae40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1464009870 tx=0x7f146402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3ca2a746c2c673 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:00.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.083+0000 7f146fe00640 1 -- 172.21.15.102:0/2958884468 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146402f830 con 0x7f14700cfe70 2024-03-18T16:37:00.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.085+0000 7f146fe00640 1 -- 172.21.15.102:0/2958884468 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f146402f9d0 con 0x7f14700cfe70 2024-03-18T16:37:00.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.085+0000 7f146fe00640 1 -- 172.21.15.102:0/2958884468 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146402fda0 con 0x7f14700cfe70 2024-03-18T16:37:00.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.086+0000 7f1476e00640 1 -- 172.21.15.102:0/2958884468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700cfe70 msgr2=0x7f147015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.086+0000 7f1476e00640 1 --2- 172.21.15.102:0/2958884468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700cfe70 0x7f147015ae40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1464009870 tx=0x7f146402ba10 comp rx=0 tx=0).stop 2024-03-18T16:37:00.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.086+0000 7f1476e00640 1 -- 172.21.15.102:0/2958884468 shutdown_connections 2024-03-18T16:37:00.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.086+0000 7f1476e00640 1 --2- 172.21.15.102:0/2958884468 >> v2:172.21.15.102:3300/0 conn(0x7f14700d0250 0x7f14700c65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.086+0000 7f1476e00640 1 --2- 172.21.15.102:0/2958884468 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700cfe70 0x7f147015ae40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.086+0000 7f1476e00640 1 -- 172.21.15.102:0/2958884468 >> 172.21.15.102:0/2958884468 conn(0x7f147008c3e0 msgr2=0x7f147006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:00.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.086+0000 7f1476e00640 1 -- 172.21.15.102:0/2958884468 shutdown_connections 2024-03-18T16:37:00.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.087+0000 7f1476e00640 1 -- 172.21.15.102:0/2958884468 wait complete. 2024-03-18T16:37:00.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.088+0000 7f1476e00640 1 Processor -- start 2024-03-18T16:37:00.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.089+0000 7f1476e00640 1 -- start start 2024-03-18T16:37:00.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.089+0000 7f1476e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14700cc2f0 0x7f14701e1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.089+0000 7f1476e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14700cfe70 0x7f1470156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.090+0000 7f1476e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700d0250 0x7f1470157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.090+0000 7f1476e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f14700cbd00 con 0x7f14700cc2f0 2024-03-18T16:37:00.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.090+0000 7f1476e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f14700cba20 con 0x7f14700cfe70 2024-03-18T16:37:00.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.090+0000 7f1476e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f14700c6b20 con 0x7f14700d0250 2024-03-18T16:37:00.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.090+0000 7f1476400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700d0250 0x7f1470157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.091+0000 7f1475a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14700cc2f0 0x7f14701e1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.091+0000 7f1476400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700d0250 0x7f1470157490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:38262/0 (socket says 172.21.15.102:38262) 2024-03-18T16:37:00.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.091+0000 7f1476400640 1 -- 172.21.15.102:0/579410074 learned_addr learned my addr 172.21.15.102:0/579410074 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:37:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.094+0000 7f1475000640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14700cfe70 0x7f1470156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.094+0000 7f1476400640 1 -- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14700cfe70 msgr2=0x7f1470156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.094+0000 7f1476400640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14700cfe70 0x7f1470156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.094+0000 7f1476400640 1 -- 172.21.15.102:0/579410074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14700cc2f0 msgr2=0x7f14701e1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.094+0000 7f1476400640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14700cc2f0 0x7f14701e1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.094+0000 7f1476400640 1 -- 172.21.15.102:0/579410074 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147015a1b0 con 0x7f14700d0250 2024-03-18T16:37:00.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.097+0000 7f1476400640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700d0250 0x7f1470157490 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f146800c6b0 tx=0x7f146800cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:00.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.101+0000 7f146d600640 1 -- 172.21.15.102:0/579410074 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468003b70 con 0x7f14700d0250 2024-03-18T16:37:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.101+0000 7f146d600640 1 -- 172.21.15.102:0/579410074 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1468003d30 con 0x7f14700d0250 2024-03-18T16:37:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.101+0000 7f146d600640 1 -- 172.21.15.102:0/579410074 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468005310 con 0x7f14700d0250 2024-03-18T16:37:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.101+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14701f3950 con 0x7f14700d0250 2024-03-18T16:37:00.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.101+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f147015a450 con 0x7f14700d0250 2024-03-18T16:37:00.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.103+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1470074fd0 con 0x7f14700d0250 2024-03-18T16:37:00.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.126+0000 7f146d600640 1 -- 172.21.15.102:0/579410074 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f14680054b0 con 0x7f14700d0250 2024-03-18T16:37:00.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.127+0000 7f146d600640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f145006bbd0 0x7f145006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.128+0000 7f1475a00640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f145006bbd0 0x7f145006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.129+0000 7f1475a00640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f145006bbd0 0x7f145006e090 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1464036000 tx=0x7f1464008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:00.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.370+0000 7f146d600640 1 -- 172.21.15.102:0/579410074 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f14680057a0 con 0x7f14700d0250 2024-03-18T16:37:00.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.386+0000 7f146d600640 1 -- 172.21.15.102:0/579410074 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1468059e60 con 0x7f14700d0250 2024-03-18T16:37:00.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.580+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f147015b130 con 0x7f145006bbd0 2024-03-18T16:37:00.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.580+0000 7f146d600640 1 -- 172.21.15.102:0/579410074 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f147015b130 con 0x7f145006bbd0 2024-03-18T16:37:00.582 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 7 to deep-scrub 2024-03-18T16:37:00.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f145006bbd0 msgr2=0x7f145006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f145006bbd0 0x7f145006e090 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1464036000 tx=0x7f1464008030 comp rx=0 tx=0).stop 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700d0250 msgr2=0x7f1470157490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700d0250 0x7f1470157490 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f146800c6b0 tx=0x7f146800cb70 comp rx=0 tx=0).stop 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 shutdown_connections 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f145006bbd0 0x7f145006e090 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f14700d0250 0x7f1470157490 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14700cfe70 0x7f1470156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 --2- 172.21.15.102:0/579410074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14700cc2f0 0x7f14701e1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 >> 172.21.15.102:0/579410074 conn(0x7f147008c3e0 msgr2=0x7f14700c4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 shutdown_connections 2024-03-18T16:37:00.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.583+0000 7f1476e00640 1 -- 172.21.15.102:0/579410074 wait complete. 2024-03-18T16:37:00.601 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:37:00.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.766+0000 7f8632600640 1 Processor -- start 2024-03-18T16:37:00.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.767+0000 7f8632600640 1 -- start start 2024-03-18T16:37:00.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.768+0000 7f8632600640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f862c154520 0x7f862c0c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.768+0000 7f8632600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f862c0c3890 0x7f862c0c8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.768+0000 7f8632600640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f862c15a170 con 0x7f862c0c3cd0 2024-03-18T16:37:00.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.768+0000 7f8632600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f862c154900 con 0x7f862c0c3890 2024-03-18T16:37:00.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.768+0000 7f8632600640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f862c0c4160 con 0x7f862c154520 2024-03-18T16:37:00.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.769+0000 7f862be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f862c0c3890 0x7f862c0c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.769+0000 7f8631200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f862c154520 0x7f862c0c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.769+0000 7f862be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f862c0c3890 0x7f862c0c8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33914/0 (socket says 172.21.15.102:33914) 2024-03-18T16:37:00.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.769+0000 7f8631c00640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f862c0c3cd0 0x7f862c0c9370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.102:39082/0 (socket says 172.21.15.102:39082) 2024-03-18T16:37:00.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.769+0000 7f862be00640 1 -- 172.21.15.102:0/2816868128 learned_addr learned my addr 172.21.15.102:0/2816868128 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:37:00.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.772+0000 7f862be00640 1 -- 172.21.15.102:0/2816868128 >> v2:172.21.15.154:3300/0 conn(0x7f862c154520 msgr2=0x7f862c0c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:37:00.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.772+0000 7f862be00640 1 --2- 172.21.15.102:0/2816868128 >> v2:172.21.15.154:3300/0 conn(0x7f862c154520 0x7f862c0c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.772+0000 7f862be00640 1 -- 172.21.15.102:0/2816868128 >> v1:172.21.15.154:6789/0 conn(0x7f862c0c3cd0 legacy=0x7f862c0c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.772+0000 7f862be00640 1 -- 172.21.15.102:0/2816868128 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862c0c4440 con 0x7f862c0c3890 2024-03-18T16:37:00.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.775+0000 7f862be00640 1 --2- 172.21.15.102:0/2816868128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f862c0c3890 0x7f862c0c8920 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f862c0cb4c0 tx=0x7f861802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd8cfab08f45dec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:00.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.781+0000 7f862b400640 1 -- 172.21.15.102:0/2816868128 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618034070 con 0x7f862c0c3890 2024-03-18T16:37:00.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.781+0000 7f862b400640 1 -- 172.21.15.102:0/2816868128 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8618030860 con 0x7f862c0c3890 2024-03-18T16:37:00.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.781+0000 7f862b400640 1 -- 172.21.15.102:0/2816868128 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618030c30 con 0x7f862c0c3890 2024-03-18T16:37:00.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.782+0000 7f8632600640 1 -- 172.21.15.102:0/2816868128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f862c0c3890 msgr2=0x7f862c0c8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.782+0000 7f8632600640 1 --2- 172.21.15.102:0/2816868128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f862c0c3890 0x7f862c0c8920 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f862c0cb4c0 tx=0x7f861802b9a0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.782+0000 7f8632600640 1 -- 172.21.15.102:0/2816868128 shutdown_connections 2024-03-18T16:37:00.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.782+0000 7f8632600640 1 --2- 172.21.15.102:0/2816868128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f862c0c3890 0x7f862c0c8920 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.782+0000 7f8632600640 1 --2- 172.21.15.102:0/2816868128 >> v2:172.21.15.154:3300/0 conn(0x7f862c154520 0x7f862c0c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.782+0000 7f8632600640 1 -- 172.21.15.102:0/2816868128 >> 172.21.15.102:0/2816868128 conn(0x7f862c066f80 msgr2=0x7f862c067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:00.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.782+0000 7f8632600640 1 -- 172.21.15.102:0/2816868128 shutdown_connections 2024-03-18T16:37:00.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.782+0000 7f8632600640 1 -- 172.21.15.102:0/2816868128 wait complete. 2024-03-18T16:37:00.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.784+0000 7f8632600640 1 Processor -- start 2024-03-18T16:37:00.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.784+0000 7f8632600640 1 -- start start 2024-03-18T16:37:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.785+0000 7f8632600640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f862c0c3890 0x7f862c148940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.785+0000 7f8632600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f862c154520 0x7f862c1527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.785+0000 7f8632600640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f862c142550 0x7f862c152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.785+0000 7f8632600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f862c15a9f0 con 0x7f862c154520 2024-03-18T16:37:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.785+0000 7f8632600640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f862c15a710 con 0x7f862c0c3890 2024-03-18T16:37:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.785+0000 7f8632600640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f862c0ca3f0 con 0x7f862c142550 2024-03-18T16:37:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.786+0000 7f8631200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f862c0c3890 0x7f862c148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.786+0000 7f8631c00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f862c142550 0x7f862c152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.786+0000 7f862be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f862c154520 0x7f862c1527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.786+0000 7f8631200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f862c0c3890 0x7f862c148940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:56480/0 (socket says 172.21.15.102:56480) 2024-03-18T16:37:00.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.786+0000 7f8631200640 1 -- 172.21.15.102:0/893221795 learned_addr learned my addr 172.21.15.102:0/893221795 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:37:00.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.789+0000 7f8631c00640 1 -- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f862c0c3890 msgr2=0x7f862c148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.789+0000 7f8631c00640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f862c0c3890 0x7f862c148940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.789+0000 7f8631c00640 1 -- 172.21.15.102:0/893221795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f862c154520 msgr2=0x7f862c1527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:00.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.789+0000 7f8631c00640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f862c154520 0x7f862c1527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:00.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.789+0000 7f8631c00640 1 -- 172.21.15.102:0/893221795 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862c145600 con 0x7f862c142550 2024-03-18T16:37:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.792+0000 7f8631c00640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f862c142550 0x7f862c152d10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f862400c6f0 tx=0x7f862400cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:00.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.795+0000 7f8628c00640 1 -- 172.21.15.102:0/893221795 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8624017070 con 0x7f862c142550 2024-03-18T16:37:00.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.795+0000 7f8628c00640 1 -- 172.21.15.102:0/893221795 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8624005da0 con 0x7f862c142550 2024-03-18T16:37:00.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.795+0000 7f8628c00640 1 -- 172.21.15.102:0/893221795 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f862400a840 con 0x7f862c142550 2024-03-18T16:37:00.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.795+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f862c1458a0 con 0x7f862c142550 2024-03-18T16:37:00.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.795+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f862c151fd0 con 0x7f862c142550 2024-03-18T16:37:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.797+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f862c153d80 con 0x7f862c142550 2024-03-18T16:37:00.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.816+0000 7f8628c00640 1 -- 172.21.15.102:0/893221795 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f862401a070 con 0x7f862c142550 2024-03-18T16:37:00.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.817+0000 7f8628c00640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f860c06bbd0 0x7f860c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:00.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.817+0000 7f8631200640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f860c06bbd0 0x7f860c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:00.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.818+0000 7f8631200640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f860c06bbd0 0x7f860c06e090 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8620009b60 tx=0x7f8620009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:00.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:00.819+0000 7f8628c00640 1 -- 172.21.15.102:0/893221795 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f86240568b0 con 0x7f862c142550 2024-03-18T16:37:01.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.069+0000 7f8628c00640 1 -- 172.21.15.102:0/893221795 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f862c153d80 con 0x7f862c142550 2024-03-18T16:37:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.328+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f862c074fd0 con 0x7f860c06bbd0 2024-03-18T16:37:01.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.336+0000 7f8628c00640 1 -- 172.21.15.102:0/893221795 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+329710 (secure 0 0 0) 0x7f862c074fd0 con 0x7f860c06bbd0 2024-03-18T16:37:01.342 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:37:01.344 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:37:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f860c06bbd0 msgr2=0x7f860c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f860c06bbd0 0x7f860c06e090 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8620009b60 tx=0x7f8620009190 comp rx=0 tx=0).stop 2024-03-18T16:37:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f862c142550 msgr2=0x7f862c152d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f862c142550 0x7f862c152d10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f862400c6f0 tx=0x7f862400cbb0 comp rx=0 tx=0).stop 2024-03-18T16:37:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 shutdown_connections 2024-03-18T16:37:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f860c06bbd0 0x7f860c06e090 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f862c142550 0x7f862c152d10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f862c154520 0x7f862c1527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:01.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 --2- 172.21.15.102:0/893221795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f862c0c3890 0x7f862c148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:01.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.342+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 >> 172.21.15.102:0/893221795 conn(0x7f862c066f80 msgr2=0x7f862c0c8560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:01.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.343+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 shutdown_connections 2024-03-18T16:37:01.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:01.343+0000 7f8632600640 1 -- 172.21.15.102:0/893221795 wait complete. 2024-03-18T16:37:01.368 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2024-03-18T16:37:01.036925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":335,"ondisk_log_size":335,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219012,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755708,"statfs":{"total":773094113280,"available":772869844992,"internally_reserved":0,"allocated":2949120,"data_stored":2294136,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002757"},"pg_stats":[{"pgid":"2.1f","version":"30'19","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:55.744172+0000","last_change":"2024-03-18T16:36:55.744239+0000","last_active":"2024-03-18T16:36:55.744172+0000","last_peered":"2024-03-18T16:36:55.744172+0000","last_clean":"2024-03-18T16:36:55.744172+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:36:55.744172+0000","last_undegraded":"2024-03-18T16:36:55.744172+0000","last_fullsized":"2024-03-18T16:36:55.744172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-03-18T16:36:55.744050+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-03-18T16:36:55.744050+0000","last_clean_scrub_stamp":"2024-03-18T16:36:55.744050+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:11.820011+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:22.178782+0000","last_change":"2024-03-18T16:21:23.045605+0000","last_active":"2024-03-18T16:35:22.178782+0000","last_peered":"2024-03-18T16:35:22.178782+0000","last_clean":"2024-03-18T16:35:22.178782+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:35:22.178782+0000","last_undegraded":"2024-03-18T16:35:22.178782+0000","last_fullsized":"2024-03-18T16:35:22.178782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:02.183897+0000","last_change":"2024-03-18T16:21:29.146958+0000","last_active":"2024-03-18T16:36:02.183897+0000","last_peered":"2024-03-18T16:36:02.183897+0000","last_clean":"2024-03-18T16:36:02.183897+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:36:02.183897+0000","last_undegraded":"2024-03-18T16:36:02.183897+0000","last_fullsized":"2024-03-18T16:36:02.183897+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:22.178808+0000","last_change":"2024-03-18T16:21:21.413060+0000","last_active":"2024-03-18T16:35:22.178808+0000","last_peered":"2024-03-18T16:35:22.178808+0000","last_clean":"2024-03-18T16:35:22.178808+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:35:22.178808+0000","last_undegraded":"2024-03-18T16:35:22.178808+0000","last_fullsized":"2024-03-18T16:35:22.178808+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:57.218987+0000","last_change":"2024-03-18T16:21:23.050849+0000","last_active":"2024-03-18T16:35:57.218987+0000","last_peered":"2024-03-18T16:35:57.218987+0000","last_clean":"2024-03-18T16:35:57.218987+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:35:57.218987+0000","last_undegraded":"2024-03-18T16:35:57.218987+0000","last_fullsized":"2024-03-18T16:35:57.218987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:57.974029+0000","last_change":"2024-03-18T16:21:29.065755+0000","last_active":"2024-03-18T16:35:57.974029+0000","last_peered":"2024-03-18T16:35:57.974029+0000","last_clean":"2024-03-18T16:35:57.974029+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:35:57.974029+0000","last_undegraded":"2024-03-18T16:35:57.974029+0000","last_fullsized":"2024-03-18T16:35:57.974029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"29'12","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092522+0000","last_change":"2024-03-18T16:21:21.398188+0000","last_active":"2024-03-18T16:33:52.092522+0000","last_peered":"2024-03-18T16:33:52.092522+0000","last_clean":"2024-03-18T16:33:52.092522+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:33:52.092522+0000","last_undegraded":"2024-03-18T16:33:52.092522+0000","last_fullsized":"2024-03-18T16:33:52.092522+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:59:39.319571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:24.413319+0000","last_change":"2024-03-18T16:21:22.946288+0000","last_active":"2024-03-18T16:36:24.413319+0000","last_peered":"2024-03-18T16:36:24.413319+0000","last_clean":"2024-03-18T16:36:24.413319+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:36:24.413319+0000","last_undegraded":"2024-03-18T16:36:24.413319+0000","last_fullsized":"2024-03-18T16:36:24.413319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:27.897091+0000","last_change":"2024-03-18T16:21:29.065891+0000","last_active":"2024-03-18T16:36:27.897091+0000","last_peered":"2024-03-18T16:36:27.897091+0000","last_clean":"2024-03-18T16:36:27.897091+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:36:27.897091+0000","last_undegraded":"2024-03-18T16:36:27.897091+0000","last_fullsized":"2024-03-18T16:36:27.897091+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092509+0000","last_change":"2024-03-18T16:21:21.397071+0000","last_active":"2024-03-18T16:33:52.092509+0000","last_peered":"2024-03-18T16:33:52.092509+0000","last_clean":"2024-03-18T16:33:52.092509+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:33:52.092509+0000","last_undegraded":"2024-03-18T16:33:52.092509+0000","last_fullsized":"2024-03-18T16:33:52.092509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:15:50.688899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:24.413337+0000","last_change":"2024-03-18T16:21:22.946468+0000","last_active":"2024-03-18T16:36:24.413337+0000","last_peered":"2024-03-18T16:36:24.413337+0000","last_clean":"2024-03-18T16:36:24.413337+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:36:24.413337+0000","last_undegraded":"2024-03-18T16:36:24.413337+0000","last_fullsized":"2024-03-18T16:36:24.413337+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:27.897068+0000","last_change":"2024-03-18T16:21:29.073724+0000","last_active":"2024-03-18T16:36:27.897068+0000","last_peered":"2024-03-18T16:36:27.897068+0000","last_clean":"2024-03-18T16:36:27.897068+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:36:27.897068+0000","last_undegraded":"2024-03-18T16:36:27.897068+0000","last_fullsized":"2024-03-18T16:36:27.897068+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536515+0000","last_change":"2024-03-18T16:21:23.041926+0000","last_active":"2024-03-18T16:34:24.536515+0000","last_peered":"2024-03-18T16:34:24.536515+0000","last_clean":"2024-03-18T16:34:24.536515+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:34:24.536515+0000","last_undegraded":"2024-03-18T16:34:24.536515+0000","last_fullsized":"2024-03-18T16:34:24.536515+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:57:03.532253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"29'4","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085707+0000","last_change":"2024-03-18T16:21:21.394801+0000","last_active":"2024-03-18T16:32:37.085707+0000","last_peered":"2024-03-18T16:32:37.085707+0000","last_clean":"2024-03-18T16:32:37.085707+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:32:37.085707+0000","last_undegraded":"2024-03-18T16:32:37.085707+0000","last_fullsized":"2024-03-18T16:32:37.085707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:16.004480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:57.969474+0000","last_change":"2024-03-18T16:21:29.073757+0000","last_active":"2024-03-18T16:34:57.969474+0000","last_peered":"2024-03-18T16:34:57.969474+0000","last_clean":"2024-03-18T16:34:57.969474+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:34:57.969474+0000","last_undegraded":"2024-03-18T16:34:57.969474+0000","last_fullsized":"2024-03-18T16:34:57.969474+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:52.175000+0000","last_change":"2024-03-18T16:21:23.060776+0000","last_active":"2024-03-18T16:34:52.175000+0000","last_peered":"2024-03-18T16:34:52.175000+0000","last_clean":"2024-03-18T16:34:52.175000+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:34:52.175000+0000","last_undegraded":"2024-03-18T16:34:52.175000+0000","last_fullsized":"2024-03-18T16:34:52.175000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"30'4","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:37.866964+0000","last_change":"2024-03-18T16:21:21.412733+0000","last_active":"2024-03-18T16:36:37.866964+0000","last_peered":"2024-03-18T16:36:37.866964+0000","last_clean":"2024-03-18T16:36:37.866964+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:36:37.866964+0000","last_undegraded":"2024-03-18T16:36:37.866964+0000","last_fullsized":"2024-03-18T16:36:37.866964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:57.969452+0000","last_change":"2024-03-18T16:21:29.063980+0000","last_active":"2024-03-18T16:34:57.969452+0000","last_peered":"2024-03-18T16:34:57.969452+0000","last_clean":"2024-03-18T16:34:57.969452+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:34:57.969452+0000","last_undegraded":"2024-03-18T16:34:57.969452+0000","last_fullsized":"2024-03-18T16:34:57.969452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:55.672658+0000","last_change":"2024-03-18T16:21:54.838108+0000","last_active":"2024-03-18T16:34:55.672658+0000","last_peered":"2024-03-18T16:34:55.672658+0000","last_clean":"2024-03-18T16:34:55.672658+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:34:55.672658+0000","last_undegraded":"2024-03-18T16:34:55.672658+0000","last_fullsized":"2024-03-18T16:34:55.672658+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"29'7","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:32.866318+0000","last_change":"2024-03-18T16:21:21.407132+0000","last_active":"2024-03-18T16:36:32.866318+0000","last_peered":"2024-03-18T16:36:32.866318+0000","last_clean":"2024-03-18T16:36:32.866318+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:36:32.866318+0000","last_undegraded":"2024-03-18T16:36:32.866318+0000","last_fullsized":"2024-03-18T16:36:32.866318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:25.675927+0000","last_change":"2024-03-18T16:21:55.269661+0000","last_active":"2024-03-18T16:35:25.675927+0000","last_peered":"2024-03-18T16:35:25.675927+0000","last_clean":"2024-03-18T16:35:25.675927+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:35:25.675927+0000","last_undegraded":"2024-03-18T16:35:25.675927+0000","last_fullsized":"2024-03-18T16:35:25.675927+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:32.897429+0000","last_change":"2024-03-18T16:21:22.840551+0000","last_active":"2024-03-18T16:36:32.897429+0000","last_peered":"2024-03-18T16:36:32.897429+0000","last_clean":"2024-03-18T16:36:32.897429+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:36:32.897429+0000","last_undegraded":"2024-03-18T16:36:32.897429+0000","last_fullsized":"2024-03-18T16:36:32.897429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"29'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:22.095352+0000","last_change":"2024-03-18T16:21:21.400349+0000","last_active":"2024-03-18T16:34:22.095352+0000","last_peered":"2024-03-18T16:34:22.095352+0000","last_clean":"2024-03-18T16:34:22.095352+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:34:22.095352+0000","last_undegraded":"2024-03-18T16:34:22.095352+0000","last_fullsized":"2024-03-18T16:34:22.095352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:09:12.411398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:57.969483+0000","last_change":"2024-03-18T16:21:29.064624+0000","last_active":"2024-03-18T16:34:57.969483+0000","last_peered":"2024-03-18T16:34:57.969483+0000","last_clean":"2024-03-18T16:34:57.969483+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:34:57.969483+0000","last_undegraded":"2024-03-18T16:34:57.969483+0000","last_fullsized":"2024-03-18T16:34:57.969483+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:32.222648+0000","last_change":"2024-03-18T16:21:23.055982+0000","last_active":"2024-03-18T16:36:32.222648+0000","last_peered":"2024-03-18T16:36:32.222648+0000","last_clean":"2024-03-18T16:36:32.222648+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:36:32.222648+0000","last_undegraded":"2024-03-18T16:36:32.222648+0000","last_fullsized":"2024-03-18T16:36:32.222648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"29'7","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536571+0000","last_change":"2024-03-18T16:21:21.400843+0000","last_active":"2024-03-18T16:34:24.536571+0000","last_peered":"2024-03-18T16:34:24.536571+0000","last_clean":"2024-03-18T16:34:24.536571+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:34:24.536571+0000","last_undegraded":"2024-03-18T16:34:24.536571+0000","last_fullsized":"2024-03-18T16:34:24.536571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:39.623702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:02.974446+0000","last_change":"2024-03-18T16:21:29.140888+0000","last_active":"2024-03-18T16:36:02.974446+0000","last_peered":"2024-03-18T16:36:02.974446+0000","last_clean":"2024-03-18T16:36:02.974446+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:36:02.974446+0000","last_undegraded":"2024-03-18T16:36:02.974446+0000","last_fullsized":"2024-03-18T16:36:02.974446+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:32.187196+0000","last_change":"2024-03-18T16:21:22.994455+0000","last_active":"2024-03-18T16:36:32.187196+0000","last_peered":"2024-03-18T16:36:32.187196+0000","last_clean":"2024-03-18T16:36:32.187196+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:36:32.187196+0000","last_undegraded":"2024-03-18T16:36:32.187196+0000","last_fullsized":"2024-03-18T16:36:32.187196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"29'7","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:32.866412+0000","last_change":"2024-03-18T16:21:21.402920+0000","last_active":"2024-03-18T16:36:32.866412+0000","last_peered":"2024-03-18T16:36:32.866412+0000","last_clean":"2024-03-18T16:36:32.866412+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:36:32.866412+0000","last_undegraded":"2024-03-18T16:36:32.866412+0000","last_fullsized":"2024-03-18T16:36:32.866412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:32.179849+0000","last_change":"2024-03-18T16:21:29.145235+0000","last_active":"2024-03-18T16:35:32.179849+0000","last_peered":"2024-03-18T16:35:32.179849+0000","last_clean":"2024-03-18T16:35:32.179849+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:35:32.179849+0000","last_undegraded":"2024-03-18T16:35:32.179849+0000","last_fullsized":"2024-03-18T16:35:32.179849+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:57.972216+0000","last_change":"2024-03-18T16:21:22.975172+0000","last_active":"2024-03-18T16:35:57.972216+0000","last_peered":"2024-03-18T16:35:57.972216+0000","last_clean":"2024-03-18T16:35:57.972216+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:35:57.972216+0000","last_undegraded":"2024-03-18T16:35:57.972216+0000","last_fullsized":"2024-03-18T16:35:57.972216+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092531+0000","last_change":"2024-03-18T16:21:21.399688+0000","last_active":"2024-03-18T16:33:52.092531+0000","last_peered":"2024-03-18T16:33:52.092531+0000","last_clean":"2024-03-18T16:33:52.092531+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:33:52.092531+0000","last_undegraded":"2024-03-18T16:33:52.092531+0000","last_fullsized":"2024-03-18T16:33:52.092531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:11.933810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:07.220034+0000","last_change":"2024-03-18T16:21:29.141555+0000","last_active":"2024-03-18T16:36:07.220034+0000","last_peered":"2024-03-18T16:36:07.220034+0000","last_clean":"2024-03-18T16:36:07.220034+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:36:07.220034+0000","last_undegraded":"2024-03-18T16:36:07.220034+0000","last_fullsized":"2024-03-18T16:36:07.220034+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.691194+0000","last_change":"2024-03-18T16:36:58.691214+0000","last_active":"2024-03-18T16:36:58.691194+0000","last_peered":"2024-03-18T16:36:58.691194+0000","last_clean":"2024-03-18T16:36:58.691194+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:36:58.691194+0000","last_undegraded":"2024-03-18T16:36:58.691194+0000","last_fullsized":"2024-03-18T16:36:58.691194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:58.691130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:58.691130+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.691130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:26.607438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"30'2","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:56.393346+0000","last_change":"2024-03-18T16:36:56.393387+0000","last_active":"2024-03-18T16:36:56.393346+0000","last_peered":"2024-03-18T16:36:56.393346+0000","last_clean":"2024-03-18T16:36:56.393346+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:36:56.393346+0000","last_undegraded":"2024-03-18T16:36:56.393346+0000","last_fullsized":"2024-03-18T16:36:56.393346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-03-18T16:36:56.393245+0000","last_deep_scrub":"30'2","last_deep_scrub_stamp":"2024-03-18T16:36:56.393245+0000","last_clean_scrub_stamp":"2024-03-18T16:36:56.393245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:10.756772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:07.862254+0000","last_change":"2024-03-18T16:21:29.076629+0000","last_active":"2024-03-18T16:36:07.862254+0000","last_peered":"2024-03-18T16:36:07.862254+0000","last_clean":"2024-03-18T16:36:07.862254+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:36:07.862254+0000","last_undegraded":"2024-03-18T16:36:07.862254+0000","last_fullsized":"2024-03-18T16:36:07.862254+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:57.974067+0000","last_change":"2024-03-18T16:21:23.055810+0000","last_active":"2024-03-18T16:35:57.974067+0000","last_peered":"2024-03-18T16:35:57.974067+0000","last_clean":"2024-03-18T16:35:57.974067+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:35:57.974067+0000","last_undegraded":"2024-03-18T16:35:57.974067+0000","last_fullsized":"2024-03-18T16:35:57.974067+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"29'5","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:59.492619+0000","last_change":"2024-03-18T16:21:21.399134+0000","last_active":"2024-03-18T16:35:59.492619+0000","last_peered":"2024-03-18T16:35:59.492619+0000","last_clean":"2024-03-18T16:35:59.492619+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:35:59.492619+0000","last_undegraded":"2024-03-18T16:35:59.492619+0000","last_fullsized":"2024-03-18T16:35:59.492619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:07.862295+0000","last_change":"2024-03-18T16:21:29.067567+0000","last_active":"2024-03-18T16:36:07.862295+0000","last_peered":"2024-03-18T16:36:07.862295+0000","last_clean":"2024-03-18T16:36:07.862295+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:36:07.862295+0000","last_undegraded":"2024-03-18T16:36:07.862295+0000","last_fullsized":"2024-03-18T16:36:07.862295+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:55.352796+0000","last_change":"2024-03-18T16:36:55.352845+0000","last_active":"2024-03-18T16:36:55.352796+0000","last_peered":"2024-03-18T16:36:55.352796+0000","last_clean":"2024-03-18T16:36:55.352796+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:36:55.352796+0000","last_undegraded":"2024-03-18T16:36:55.352796+0000","last_fullsized":"2024-03-18T16:36:55.352796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:55.352684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:55.352684+0000","last_clean_scrub_stamp":"2024-03-18T16:36:55.352684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:14:01.365564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'8","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.105679+0000","last_change":"2024-03-18T16:36:58.105742+0000","last_active":"2024-03-18T16:36:58.105679+0000","last_peered":"2024-03-18T16:36:58.105679+0000","last_clean":"2024-03-18T16:36:58.105679+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:36:58.105679+0000","last_undegraded":"2024-03-18T16:36:58.105679+0000","last_fullsized":"2024-03-18T16:36:58.105679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:36:58.105545+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:36:58.105545+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.105545+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:05:38.967022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:42.867496+0000","last_change":"2024-03-18T16:21:29.076875+0000","last_active":"2024-03-18T16:36:42.867496+0000","last_peered":"2024-03-18T16:36:42.867496+0000","last_clean":"2024-03-18T16:36:42.867496+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:36:42.867496+0000","last_undegraded":"2024-03-18T16:36:42.867496+0000","last_fullsized":"2024-03-18T16:36:42.867496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:57.218894+0000","last_change":"2024-03-18T16:21:23.054497+0000","last_active":"2024-03-18T16:35:57.218894+0000","last_peered":"2024-03-18T16:35:57.218894+0000","last_clean":"2024-03-18T16:35:57.218894+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:35:57.218894+0000","last_undegraded":"2024-03-18T16:35:57.218894+0000","last_fullsized":"2024-03-18T16:35:57.218894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:22.089824+0000","last_change":"2024-03-18T16:21:21.404754+0000","last_active":"2024-03-18T16:33:22.089824+0000","last_peered":"2024-03-18T16:33:22.089824+0000","last_clean":"2024-03-18T16:33:22.089824+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:33:22.089824+0000","last_undegraded":"2024-03-18T16:33:22.089824+0000","last_fullsized":"2024-03-18T16:33:22.089824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:21:27.424729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:42.223693+0000","last_change":"2024-03-18T16:21:29.147316+0000","last_active":"2024-03-18T16:36:42.223693+0000","last_peered":"2024-03-18T16:36:42.223693+0000","last_clean":"2024-03-18T16:36:42.223693+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:36:42.223693+0000","last_undegraded":"2024-03-18T16:36:42.223693+0000","last_fullsized":"2024-03-18T16:36:42.223693+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:57.183354+0000","last_change":"2024-03-18T16:21:22.947635+0000","last_active":"2024-03-18T16:35:57.183354+0000","last_peered":"2024-03-18T16:35:57.183354+0000","last_clean":"2024-03-18T16:35:57.183354+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:35:57.183354+0000","last_undegraded":"2024-03-18T16:35:57.183354+0000","last_fullsized":"2024-03-18T16:35:57.183354+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"29'6","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:59.492651+0000","last_change":"2024-03-18T16:21:21.381306+0000","last_active":"2024-03-18T16:35:59.492651+0000","last_peered":"2024-03-18T16:35:59.492651+0000","last_clean":"2024-03-18T16:35:59.492651+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:35:59.492651+0000","last_undegraded":"2024-03-18T16:35:59.492651+0000","last_fullsized":"2024-03-18T16:35:59.492651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:07.088485+0000","last_change":"2024-03-18T16:21:08.662767+0000","last_active":"2024-03-18T16:33:07.088485+0000","last_peered":"2024-03-18T16:33:07.088485+0000","last_clean":"2024-03-18T16:33:07.088485+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:33:07.088485+0000","last_undegraded":"2024-03-18T16:33:07.088485+0000","last_fullsized":"2024-03-18T16:33:07.088485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:05.847994+0000","last_clean_scrub_stamp":"2024-03-18T16:21:05.847994+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:12:10.494707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:59.534564+0000","last_change":"2024-03-18T16:21:29.145697+0000","last_active":"2024-03-18T16:33:59.534564+0000","last_peered":"2024-03-18T16:33:59.534564+0000","last_clean":"2024-03-18T16:33:59.534564+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:33:59.534564+0000","last_undegraded":"2024-03-18T16:33:59.534564+0000","last_fullsized":"2024-03-18T16:33:59.534564+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:46.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085732+0000","last_change":"2024-03-18T16:21:22.995291+0000","last_active":"2024-03-18T16:32:37.085732+0000","last_peered":"2024-03-18T16:32:37.085732+0000","last_clean":"2024-03-18T16:32:37.085732+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:32:37.085732+0000","last_undegraded":"2024-03-18T16:32:37.085732+0000","last_fullsized":"2024-03-18T16:32:37.085732+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:50:32.918246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"30'9","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:56.047878+0000","last_change":"2024-03-18T16:36:56.047926+0000","last_active":"2024-03-18T16:36:56.047878+0000","last_peered":"2024-03-18T16:36:56.047878+0000","last_clean":"2024-03-18T16:36:56.047878+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:36:56.047878+0000","last_undegraded":"2024-03-18T16:36:56.047878+0000","last_fullsized":"2024-03-18T16:36:56.047878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-03-18T16:36:56.047790+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-03-18T16:36:56.047790+0000","last_clean_scrub_stamp":"2024-03-18T16:36:56.047790+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:55.302327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:29.537048+0000","last_change":"2024-03-18T16:21:29.139432+0000","last_active":"2024-03-18T16:34:29.537048+0000","last_peered":"2024-03-18T16:34:29.537048+0000","last_clean":"2024-03-18T16:34:29.537048+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:34:29.537048+0000","last_undegraded":"2024-03-18T16:34:29.537048+0000","last_fullsized":"2024-03-18T16:34:29.537048+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:39:14.828125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:54.487358+0000","last_change":"2024-03-18T16:21:22.949578+0000","last_active":"2024-03-18T16:34:54.487358+0000","last_peered":"2024-03-18T16:34:54.487358+0000","last_clean":"2024-03-18T16:34:54.487358+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:34:54.487358+0000","last_undegraded":"2024-03-18T16:34:54.487358+0000","last_fullsized":"2024-03-18T16:34:54.487358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"30'5","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.427903+0000","last_change":"2024-03-18T16:36:57.427951+0000","last_active":"2024-03-18T16:36:57.427903+0000","last_peered":"2024-03-18T16:36:57.427903+0000","last_clean":"2024-03-18T16:36:57.427903+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:36:57.427903+0000","last_undegraded":"2024-03-18T16:36:57.427903+0000","last_fullsized":"2024-03-18T16:36:57.427903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5","last_scrub_stamp":"2024-03-18T16:36:57.427798+0000","last_deep_scrub":"30'5","last_deep_scrub_stamp":"2024-03-18T16:36:57.427798+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.427798+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:25.346758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:02.212948+0000","last_change":"2024-03-18T16:21:29.076813+0000","last_active":"2024-03-18T16:35:02.212948+0000","last_peered":"2024-03-18T16:35:02.212948+0000","last_clean":"2024-03-18T16:35:02.212948+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:35:02.212948+0000","last_undegraded":"2024-03-18T16:35:02.212948+0000","last_fullsized":"2024-03-18T16:35:02.212948+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085756+0000","last_change":"2024-03-18T16:21:23.000009+0000","last_active":"2024-03-18T16:32:37.085756+0000","last_peered":"2024-03-18T16:32:37.085756+0000","last_clean":"2024-03-18T16:32:37.085756+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:32:37.085756+0000","last_undegraded":"2024-03-18T16:32:37.085756+0000","last_fullsized":"2024-03-18T16:32:37.085756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:16:39.383342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.109254+0000","last_change":"2024-03-18T16:36:59.109282+0000","last_active":"2024-03-18T16:36:59.109254+0000","last_peered":"2024-03-18T16:36:59.109254+0000","last_clean":"2024-03-18T16:36:59.109254+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:36:59.109254+0000","last_undegraded":"2024-03-18T16:36:59.109254+0000","last_fullsized":"2024-03-18T16:36:59.109254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:36:59.109184+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:36:59.109184+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.109184+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:02:56.365385+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:39.414809+0000","last_change":"2024-03-18T16:21:29.139595+0000","last_active":"2024-03-18T16:36:39.414809+0000","last_peered":"2024-03-18T16:36:39.414809+0000","last_clean":"2024-03-18T16:36:39.414809+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:36:39.414809+0000","last_undegraded":"2024-03-18T16:36:39.414809+0000","last_fullsized":"2024-03-18T16:36:39.414809+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:37.085742+0000","last_change":"2024-03-18T16:21:23.000054+0000","last_active":"2024-03-18T16:32:37.085742+0000","last_peered":"2024-03-18T16:32:37.085742+0000","last_clean":"2024-03-18T16:32:37.085742+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:32:37.085742+0000","last_undegraded":"2024-03-18T16:32:37.085742+0000","last_fullsized":"2024-03-18T16:32:37.085742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:34:41.535042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"29'2","reported_seq":41,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:52.969056+0000","last_change":"2024-03-18T16:21:21.399315+0000","last_active":"2024-03-18T16:34:52.969056+0000","last_peered":"2024-03-18T16:34:52.969056+0000","last_clean":"2024-03-18T16:34:52.969056+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:34:52.969056+0000","last_undegraded":"2024-03-18T16:34:52.969056+0000","last_fullsized":"2024-03-18T16:34:52.969056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:37.897830+0000","last_change":"2024-03-18T16:21:29.065243+0000","last_active":"2024-03-18T16:36:37.897830+0000","last_peered":"2024-03-18T16:36:37.897830+0000","last_clean":"2024-03-18T16:36:37.897830+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:36:37.897830+0000","last_undegraded":"2024-03-18T16:36:37.897830+0000","last_fullsized":"2024-03-18T16:36:37.897830+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.087917+0000","last_change":"2024-03-18T16:21:22.996080+0000","last_active":"2024-03-18T16:33:02.087917+0000","last_peered":"2024-03-18T16:33:02.087917+0000","last_clean":"2024-03-18T16:33:02.087917+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:33:02.087917+0000","last_undegraded":"2024-03-18T16:33:02.087917+0000","last_fullsized":"2024-03-18T16:33:02.087917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:43.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'26","reported_seq":73,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.064571+0000","last_change":"2024-03-18T16:36:57.064626+0000","last_active":"2024-03-18T16:36:57.064571+0000","last_peered":"2024-03-18T16:36:57.064571+0000","last_clean":"2024-03-18T16:36:57.064571+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:36:57.064571+0000","last_undegraded":"2024-03-18T16:36:57.064571+0000","last_fullsized":"2024-03-18T16:36:57.064571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'26","last_scrub_stamp":"2024-03-18T16:36:57.064452+0000","last_deep_scrub":"30'26","last_deep_scrub_stamp":"2024-03-18T16:36:57.064452+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.064452+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:50.653357+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:02.219506+0000","last_change":"2024-03-18T16:21:29.079031+0000","last_active":"2024-03-18T16:36:02.219506+0000","last_peered":"2024-03-18T16:36:02.219506+0000","last_clean":"2024-03-18T16:36:02.219506+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:36:02.219506+0000","last_undegraded":"2024-03-18T16:36:02.219506+0000","last_fullsized":"2024-03-18T16:36:02.219506+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529622+0000","last_change":"2024-03-18T16:21:23.056832+0000","last_active":"2024-03-18T16:33:04.529622+0000","last_peered":"2024-03-18T16:33:04.529622+0000","last_clean":"2024-03-18T16:33:04.529622+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:33:04.529622+0000","last_undegraded":"2024-03-18T16:33:04.529622+0000","last_fullsized":"2024-03-18T16:33:04.529622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:56.124747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529644+0000","last_change":"2024-03-18T16:21:21.404392+0000","last_active":"2024-03-18T16:33:04.529644+0000","last_peered":"2024-03-18T16:33:04.529644+0000","last_clean":"2024-03-18T16:33:04.529644+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:33:04.529644+0000","last_undegraded":"2024-03-18T16:33:04.529644+0000","last_fullsized":"2024-03-18T16:33:04.529644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:12:34.660944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:02.183956+0000","last_change":"2024-03-18T16:21:29.139956+0000","last_active":"2024-03-18T16:36:02.183956+0000","last_peered":"2024-03-18T16:36:02.183956+0000","last_clean":"2024-03-18T16:36:02.183956+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:36:02.183956+0000","last_undegraded":"2024-03-18T16:36:02.183956+0000","last_fullsized":"2024-03-18T16:36:02.183956+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:02.087948+0000","last_change":"2024-03-18T16:21:22.995275+0000","last_active":"2024-03-18T16:33:02.087948+0000","last_peered":"2024-03-18T16:33:02.087948+0000","last_clean":"2024-03-18T16:33:02.087948+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:33:02.087948+0000","last_undegraded":"2024-03-18T16:33:02.087948+0000","last_fullsized":"2024-03-18T16:33:02.087948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:48.024985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"29'5","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:24.489663+0000","last_change":"2024-03-18T16:21:21.412044+0000","last_active":"2024-03-18T16:35:24.489663+0000","last_peered":"2024-03-18T16:35:24.489663+0000","last_clean":"2024-03-18T16:35:24.489663+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:35:24.489663+0000","last_undegraded":"2024-03-18T16:35:24.489663+0000","last_fullsized":"2024-03-18T16:35:24.489663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:32.179935+0000","last_change":"2024-03-18T16:21:55.270056+0000","last_active":"2024-03-18T16:35:32.179935+0000","last_peered":"2024-03-18T16:35:32.179935+0000","last_clean":"2024-03-18T16:35:32.179935+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:35:32.179935+0000","last_undegraded":"2024-03-18T16:35:32.179935+0000","last_fullsized":"2024-03-18T16:35:32.179935+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:22.214988+0000","last_change":"2024-03-18T16:21:23.056116+0000","last_active":"2024-03-18T16:35:22.214988+0000","last_peered":"2024-03-18T16:35:22.214988+0000","last_clean":"2024-03-18T16:35:22.214988+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:35:22.214988+0000","last_undegraded":"2024-03-18T16:35:22.214988+0000","last_fullsized":"2024-03-18T16:35:22.214988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'15","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:54.791306+0000","last_change":"2024-03-18T16:36:54.791362+0000","last_active":"2024-03-18T16:36:54.791306+0000","last_peered":"2024-03-18T16:36:54.791306+0000","last_clean":"2024-03-18T16:36:54.791306+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:36:54.791306+0000","last_undegraded":"2024-03-18T16:36:54.791306+0000","last_fullsized":"2024-03-18T16:36:54.791306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-03-18T16:36:54.791182+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-03-18T16:36:54.791182+0000","last_clean_scrub_stamp":"2024-03-18T16:36:54.791182+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:14:00.804062+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:59.492677+0000","last_change":"2024-03-18T16:21:29.141554+0000","last_active":"2024-03-18T16:35:59.492677+0000","last_peered":"2024-03-18T16:35:59.492677+0000","last_clean":"2024-03-18T16:35:59.492677+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:35:59.492677+0000","last_undegraded":"2024-03-18T16:35:59.492677+0000","last_fullsized":"2024-03-18T16:35:59.492677+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.090336+0000","last_change":"2024-03-18T16:21:22.996694+0000","last_active":"2024-03-18T16:33:27.090336+0000","last_peered":"2024-03-18T16:33:27.090336+0000","last_clean":"2024-03-18T16:33:27.090336+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:33:27.090336+0000","last_undegraded":"2024-03-18T16:33:27.090336+0000","last_fullsized":"2024-03-18T16:33:27.090336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:05:03.704090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"29'14","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:04.529661+0000","last_change":"2024-03-18T16:21:21.412149+0000","last_active":"2024-03-18T16:33:04.529661+0000","last_peered":"2024-03-18T16:33:04.529661+0000","last_clean":"2024-03-18T16:33:04.529661+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:33:04.529661+0000","last_undegraded":"2024-03-18T16:33:04.529661+0000","last_fullsized":"2024-03-18T16:33:04.529661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:14.806102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:32.969033+0000","last_change":"2024-03-18T16:21:29.077350+0000","last_active":"2024-03-18T16:35:32.969033+0000","last_peered":"2024-03-18T16:35:32.969033+0000","last_clean":"2024-03-18T16:35:32.969033+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:35:32.969033+0000","last_undegraded":"2024-03-18T16:35:32.969033+0000","last_fullsized":"2024-03-18T16:35:32.969033+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:22.971307+0000","last_change":"2024-03-18T16:21:22.992928+0000","last_active":"2024-03-18T16:35:22.971307+0000","last_peered":"2024-03-18T16:35:22.971307+0000","last_clean":"2024-03-18T16:35:22.971307+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:35:22.971307+0000","last_undegraded":"2024-03-18T16:35:22.971307+0000","last_fullsized":"2024-03-18T16:35:22.971307+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"29'5","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:52.971644+0000","last_change":"2024-03-18T16:21:21.403749+0000","last_active":"2024-03-18T16:35:52.971644+0000","last_peered":"2024-03-18T16:35:52.971644+0000","last_clean":"2024-03-18T16:35:52.971644+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:35:52.971644+0000","last_undegraded":"2024-03-18T16:35:52.971644+0000","last_fullsized":"2024-03-18T16:35:52.971644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:27.971661+0000","last_change":"2024-03-18T16:21:29.065378+0000","last_active":"2024-03-18T16:35:27.971661+0000","last_peered":"2024-03-18T16:35:27.971661+0000","last_clean":"2024-03-18T16:35:27.971661+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:35:27.971661+0000","last_undegraded":"2024-03-18T16:35:27.971661+0000","last_fullsized":"2024-03-18T16:35:27.971661+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:52.218381+0000","last_change":"2024-03-18T16:21:23.054168+0000","last_active":"2024-03-18T16:35:52.218381+0000","last_peered":"2024-03-18T16:35:52.218381+0000","last_clean":"2024-03-18T16:35:52.218381+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:35:52.218381+0000","last_undegraded":"2024-03-18T16:35:52.218381+0000","last_fullsized":"2024-03-18T16:35:52.218381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"29'4","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:54.492123+0000","last_change":"2024-03-18T16:21:21.398307+0000","last_active":"2024-03-18T16:35:54.492123+0000","last_peered":"2024-03-18T16:35:54.492123+0000","last_clean":"2024-03-18T16:35:54.492123+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:35:54.492123+0000","last_undegraded":"2024-03-18T16:35:54.492123+0000","last_fullsized":"2024-03-18T16:35:54.492123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:32.179885+0000","last_change":"2024-03-18T16:21:29.065855+0000","last_active":"2024-03-18T16:35:32.179885+0000","last_peered":"2024-03-18T16:35:32.179885+0000","last_clean":"2024-03-18T16:35:32.179885+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:35:32.179885+0000","last_undegraded":"2024-03-18T16:35:32.179885+0000","last_fullsized":"2024-03-18T16:35:32.179885+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:54.492113+0000","last_change":"2024-03-18T16:21:22.993606+0000","last_active":"2024-03-18T16:35:54.492113+0000","last_peered":"2024-03-18T16:35:54.492113+0000","last_clean":"2024-03-18T16:35:54.492113+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:35:54.492113+0000","last_undegraded":"2024-03-18T16:35:54.492113+0000","last_fullsized":"2024-03-18T16:35:54.492113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"29'4","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531813+0000","last_change":"2024-03-18T16:21:21.399588+0000","last_active":"2024-03-18T16:33:29.531813+0000","last_peered":"2024-03-18T16:33:29.531813+0000","last_clean":"2024-03-18T16:33:29.531813+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:33:29.531813+0000","last_undegraded":"2024-03-18T16:33:29.531813+0000","last_fullsized":"2024-03-18T16:33:29.531813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:27.028766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:32.179907+0000","last_change":"2024-03-18T16:21:29.144832+0000","last_active":"2024-03-18T16:35:32.179907+0000","last_peered":"2024-03-18T16:35:32.179907+0000","last_clean":"2024-03-18T16:35:32.179907+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:35:32.179907+0000","last_undegraded":"2024-03-18T16:35:32.179907+0000","last_fullsized":"2024-03-18T16:35:32.179907+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:32.216223+0000","last_change":"2024-03-18T16:21:29.148215+0000","last_active":"2024-03-18T16:35:32.216223+0000","last_peered":"2024-03-18T16:35:32.216223+0000","last_clean":"2024-03-18T16:35:32.216223+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:35:32.216223+0000","last_undegraded":"2024-03-18T16:35:32.216223+0000","last_fullsized":"2024-03-18T16:35:32.216223+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:52.971590+0000","last_change":"2024-03-18T16:21:22.980300+0000","last_active":"2024-03-18T16:35:52.971590+0000","last_peered":"2024-03-18T16:35:52.971590+0000","last_clean":"2024-03-18T16:35:52.971590+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:35:52.971590+0000","last_undegraded":"2024-03-18T16:35:52.971590+0000","last_fullsized":"2024-03-18T16:35:52.971590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"29'3","reported_seq":43,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:54.492064+0000","last_change":"2024-03-18T16:21:21.329294+0000","last_active":"2024-03-18T16:35:54.492064+0000","last_peered":"2024-03-18T16:35:54.492064+0000","last_clean":"2024-03-18T16:35:54.492064+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:35:54.492064+0000","last_undegraded":"2024-03-18T16:35:54.492064+0000","last_fullsized":"2024-03-18T16:35:54.492064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:59.534588+0000","last_change":"2024-03-18T16:21:29.076706+0000","last_active":"2024-03-18T16:33:59.534588+0000","last_peered":"2024-03-18T16:33:59.534588+0000","last_clean":"2024-03-18T16:33:59.534588+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:33:59.534588+0000","last_undegraded":"2024-03-18T16:33:59.534588+0000","last_fullsized":"2024-03-18T16:33:59.534588+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:49:57.880669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:52.182770+0000","last_change":"2024-03-18T16:21:23.052989+0000","last_active":"2024-03-18T16:35:52.182770+0000","last_peered":"2024-03-18T16:35:52.182770+0000","last_clean":"2024-03-18T16:35:52.182770+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:35:52.182770+0000","last_undegraded":"2024-03-18T16:35:52.182770+0000","last_fullsized":"2024-03-18T16:35:52.182770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"29'3","reported_seq":42,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:52.182805+0000","last_change":"2024-03-18T16:21:21.413020+0000","last_active":"2024-03-18T16:35:52.182805+0000","last_peered":"2024-03-18T16:35:52.182805+0000","last_clean":"2024-03-18T16:35:52.182805+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:35:52.182805+0000","last_undegraded":"2024-03-18T16:35:52.182805+0000","last_fullsized":"2024-03-18T16:35:52.182805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"29'7","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.534182+0000","last_change":"2024-03-18T16:21:21.397354+0000","last_active":"2024-03-18T16:33:54.534182+0000","last_peered":"2024-03-18T16:33:54.534182+0000","last_clean":"2024-03-18T16:33:54.534182+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:33:54.534182+0000","last_undegraded":"2024-03-18T16:33:54.534182+0000","last_fullsized":"2024-03-18T16:33:54.534182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:46.333255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:22.221604+0000","last_change":"2024-03-18T16:21:23.059441+0000","last_active":"2024-03-18T16:36:22.221604+0000","last_peered":"2024-03-18T16:36:22.221604+0000","last_clean":"2024-03-18T16:36:22.221604+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:36:22.221604+0000","last_undegraded":"2024-03-18T16:36:22.221604+0000","last_fullsized":"2024-03-18T16:36:22.221604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:30.684381+0000","last_change":"2024-03-18T16:21:29.058800+0000","last_active":"2024-03-18T16:36:30.684381+0000","last_peered":"2024-03-18T16:36:30.684381+0000","last_clean":"2024-03-18T16:36:30.684381+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:36:30.684381+0000","last_undegraded":"2024-03-18T16:36:30.684381+0000","last_fullsized":"2024-03-18T16:36:30.684381+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"29'5","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:22.186063+0000","last_change":"2024-03-18T16:21:21.412536+0000","last_active":"2024-03-18T16:36:22.186063+0000","last_peered":"2024-03-18T16:36:22.186063+0000","last_clean":"2024-03-18T16:36:22.186063+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:36:22.186063+0000","last_undegraded":"2024-03-18T16:36:22.186063+0000","last_fullsized":"2024-03-18T16:36:22.186063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:20.023353+0000","last_clean_scrub_stamp":"2024-03-18T16:21:20.023353+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:22.896637+0000","last_change":"2024-03-18T16:21:22.974963+0000","last_active":"2024-03-18T16:36:22.896637+0000","last_peered":"2024-03-18T16:36:22.896637+0000","last_clean":"2024-03-18T16:36:22.896637+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:36:22.896637+0000","last_undegraded":"2024-03-18T16:36:22.896637+0000","last_fullsized":"2024-03-18T16:36:22.896637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:32.187157+0000","last_change":"2024-03-18T16:21:55.151933+0000","last_active":"2024-03-18T16:36:32.187157+0000","last_peered":"2024-03-18T16:36:32.187157+0000","last_clean":"2024-03-18T16:36:32.187157+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:36:32.187157+0000","last_undegraded":"2024-03-18T16:36:32.187157+0000","last_fullsized":"2024-03-18T16:36:32.187157+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:27.706281+0000","last_clean_scrub_stamp":"2024-03-18T16:21:27.706281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:54.534141+0000","last_change":"2024-03-18T16:21:23.056857+0000","last_active":"2024-03-18T16:33:54.534141+0000","last_peered":"2024-03-18T16:33:54.534141+0000","last_clean":"2024-03-18T16:33:54.534141+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:33:54.534141+0000","last_undegraded":"2024-03-18T16:33:54.534141+0000","last_fullsized":"2024-03-18T16:33:54.534141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:31:13.728335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.088938+0000","last_change":"2024-03-18T16:37:00.088973+0000","last_active":"2024-03-18T16:37:00.088938+0000","last_peered":"2024-03-18T16:37:00.088938+0000","last_clean":"2024-03-18T16:37:00.088938+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:37:00.088938+0000","last_undegraded":"2024-03-18T16:37:00.088938+0000","last_fullsized":"2024-03-18T16:37:00.088938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:00.088814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:00.088814+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.088814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:11:15.114511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:30.684340+0000","last_change":"2024-03-18T16:21:54.845858+0000","last_active":"2024-03-18T16:36:30.684340+0000","last_peered":"2024-03-18T16:36:30.684340+0000","last_clean":"2024-03-18T16:36:30.684340+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:36:30.684340+0000","last_undegraded":"2024-03-18T16:36:30.684340+0000","last_fullsized":"2024-03-18T16:36:30.684340+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:57.218941+0000","last_change":"2024-03-18T16:21:23.059438+0000","last_active":"2024-03-18T16:35:57.218941+0000","last_peered":"2024-03-18T16:35:57.218941+0000","last_clean":"2024-03-18T16:35:57.218941+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:35:57.218941+0000","last_undegraded":"2024-03-18T16:35:57.218941+0000","last_fullsized":"2024-03-18T16:35:57.218941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:32.187127+0000","last_change":"2024-03-18T16:21:22.849115+0000","last_active":"2024-03-18T16:36:32.187127+0000","last_peered":"2024-03-18T16:36:32.187127+0000","last_clean":"2024-03-18T16:36:32.187127+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:36:32.187127+0000","last_undegraded":"2024-03-18T16:36:32.187127+0000","last_fullsized":"2024-03-18T16:36:32.187127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:32.866380+0000","last_change":"2024-03-18T16:21:22.980253+0000","last_active":"2024-03-18T16:36:32.866380+0000","last_peered":"2024-03-18T16:36:32.866380+0000","last_clean":"2024-03-18T16:36:32.866380+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:36:32.866380+0000","last_undegraded":"2024-03-18T16:36:32.866380+0000","last_fullsized":"2024-03-18T16:36:32.866380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:24.536554+0000","last_change":"2024-03-18T16:21:23.042188+0000","last_active":"2024-03-18T16:34:24.536554+0000","last_peered":"2024-03-18T16:34:24.536554+0000","last_clean":"2024-03-18T16:34:24.536554+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:34:24.536554+0000","last_undegraded":"2024-03-18T16:34:24.536554+0000","last_fullsized":"2024-03-18T16:34:24.536554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:01:17.682244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:32.187180+0000","last_change":"2024-03-18T16:21:22.851677+0000","last_active":"2024-03-18T16:36:32.187180+0000","last_peered":"2024-03-18T16:36:32.187180+0000","last_clean":"2024-03-18T16:36:32.187180+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:36:32.187180+0000","last_undegraded":"2024-03-18T16:36:32.187180+0000","last_fullsized":"2024-03-18T16:36:32.187180+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:39.414846+0000","last_change":"2024-03-18T16:21:22.947869+0000","last_active":"2024-03-18T16:36:39.414846+0000","last_peered":"2024-03-18T16:36:39.414846+0000","last_clean":"2024-03-18T16:36:39.414846+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:36:39.414846+0000","last_undegraded":"2024-03-18T16:36:39.414846+0000","last_fullsized":"2024-03-18T16:36:39.414846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:52.174963+0000","last_change":"2024-03-18T16:21:22.851663+0000","last_active":"2024-03-18T16:34:52.174963+0000","last_peered":"2024-03-18T16:34:52.174963+0000","last_clean":"2024-03-18T16:34:52.174963+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:34:52.174963+0000","last_undegraded":"2024-03-18T16:34:52.174963+0000","last_fullsized":"2024-03-18T16:34:52.174963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:25.675873+0000","last_change":"2024-03-18T16:21:55.268883+0000","last_active":"2024-03-18T16:35:25.675873+0000","last_peered":"2024-03-18T16:35:25.675873+0000","last_clean":"2024-03-18T16:35:25.675873+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:35:25.675873+0000","last_undegraded":"2024-03-18T16:35:25.675873+0000","last_fullsized":"2024-03-18T16:35:25.675873+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:34:52.211494+0000","last_change":"2024-03-18T16:21:23.059733+0000","last_active":"2024-03-18T16:34:52.211494+0000","last_peered":"2024-03-18T16:34:52.211494+0000","last_clean":"2024-03-18T16:34:52.211494+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:34:52.211494+0000","last_undegraded":"2024-03-18T16:34:52.211494+0000","last_fullsized":"2024-03-18T16:34:52.211494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:32:39.527349+0000","last_change":"2024-03-18T16:21:23.042457+0000","last_active":"2024-03-18T16:32:39.527349+0000","last_peered":"2024-03-18T16:32:39.527349+0000","last_clean":"2024-03-18T16:32:39.527349+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:32:39.527349+0000","last_undegraded":"2024-03-18T16:32:39.527349+0000","last_fullsized":"2024-03-18T16:32:39.527349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:10:09.455282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:27.968421+0000","last_change":"2024-03-18T16:21:54.842423+0000","last_active":"2024-03-18T16:35:27.968421+0000","last_peered":"2024-03-18T16:35:27.968421+0000","last_clean":"2024-03-18T16:35:27.968421+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:35:27.968421+0000","last_undegraded":"2024-03-18T16:35:27.968421+0000","last_fullsized":"2024-03-18T16:35:27.968421+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:25.675908+0000","last_change":"2024-03-18T16:21:23.045061+0000","last_active":"2024-03-18T16:35:25.675908+0000","last_peered":"2024-03-18T16:35:25.675908+0000","last_clean":"2024-03-18T16:35:25.675908+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:35:25.675908+0000","last_undegraded":"2024-03-18T16:35:25.675908+0000","last_fullsized":"2024-03-18T16:35:25.675908+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:24.489695+0000","last_change":"2024-03-18T16:21:22.993632+0000","last_active":"2024-03-18T16:35:24.489695+0000","last_peered":"2024-03-18T16:35:24.489695+0000","last_clean":"2024-03-18T16:35:24.489695+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:35:24.489695+0000","last_undegraded":"2024-03-18T16:35:24.489695+0000","last_fullsized":"2024-03-18T16:35:24.489695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:24.489639+0000","last_change":"2024-03-18T16:21:23.056575+0000","last_active":"2024-03-18T16:35:24.489639+0000","last_peered":"2024-03-18T16:35:24.489639+0000","last_clean":"2024-03-18T16:35:24.489639+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:35:24.489639+0000","last_undegraded":"2024-03-18T16:35:24.489639+0000","last_fullsized":"2024-03-18T16:35:24.489639+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:22.967618+0000","last_change":"2024-03-18T16:21:22.973283+0000","last_active":"2024-03-18T16:35:22.967618+0000","last_peered":"2024-03-18T16:35:22.967618+0000","last_clean":"2024-03-18T16:35:22.967618+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:35:22.967618+0000","last_undegraded":"2024-03-18T16:35:22.967618+0000","last_fullsized":"2024-03-18T16:35:22.967618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:53.738279+0000","last_change":"2024-03-18T16:36:53.738313+0000","last_active":"2024-03-18T16:36:53.738279+0000","last_peered":"2024-03-18T16:36:53.738279+0000","last_clean":"2024-03-18T16:36:53.738279+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:36:53.738279+0000","last_undegraded":"2024-03-18T16:36:53.738279+0000","last_fullsized":"2024-03-18T16:36:53.738279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:53.738175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:53.738175+0000","last_clean_scrub_stamp":"2024-03-18T16:36:53.738175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:57.558446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:52.182837+0000","last_change":"2024-03-18T16:21:23.047968+0000","last_active":"2024-03-18T16:35:52.182837+0000","last_peered":"2024-03-18T16:35:52.182837+0000","last_clean":"2024-03-18T16:35:52.182837+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:35:52.182837+0000","last_undegraded":"2024-03-18T16:35:52.182837+0000","last_fullsized":"2024-03-18T16:35:52.182837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:52.973698+0000","last_change":"2024-03-18T16:21:23.041794+0000","last_active":"2024-03-18T16:35:52.973698+0000","last_peered":"2024-03-18T16:35:52.973698+0000","last_clean":"2024-03-18T16:35:52.973698+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:35:52.973698+0000","last_undegraded":"2024-03-18T16:35:52.973698+0000","last_fullsized":"2024-03-18T16:35:52.973698+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531782+0000","last_change":"2024-03-18T16:21:23.036940+0000","last_active":"2024-03-18T16:33:29.531782+0000","last_peered":"2024-03-18T16:33:29.531782+0000","last_clean":"2024-03-18T16:33:29.531782+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:33:29.531782+0000","last_undegraded":"2024-03-18T16:33:29.531782+0000","last_fullsized":"2024-03-18T16:33:29.531782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:45:38.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:29.531802+0000","last_change":"2024-03-18T16:21:23.042140+0000","last_active":"2024-03-18T16:33:29.531802+0000","last_peered":"2024-03-18T16:33:29.531802+0000","last_clean":"2024-03-18T16:33:29.531802+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:33:29.531802+0000","last_undegraded":"2024-03-18T16:33:29.531802+0000","last_fullsized":"2024-03-18T16:33:29.531802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:47:37.864510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:27.090376+0000","last_change":"2024-03-18T16:21:22.995117+0000","last_active":"2024-03-18T16:33:27.090376+0000","last_peered":"2024-03-18T16:33:27.090376+0000","last_clean":"2024-03-18T16:33:27.090376+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:33:27.090376+0000","last_undegraded":"2024-03-18T16:33:27.090376+0000","last_fullsized":"2024-03-18T16:33:27.090376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:57.544738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:22.186012+0000","last_change":"2024-03-18T16:21:22.948123+0000","last_active":"2024-03-18T16:36:22.186012+0000","last_peered":"2024-03-18T16:36:22.186012+0000","last_clean":"2024-03-18T16:36:22.186012+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:36:22.186012+0000","last_undegraded":"2024-03-18T16:36:22.186012+0000","last_fullsized":"2024-03-18T16:36:22.186012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092477+0000","last_change":"2024-03-18T16:21:23.049877+0000","last_active":"2024-03-18T16:33:52.092477+0000","last_peered":"2024-03-18T16:33:52.092477+0000","last_clean":"2024-03-18T16:33:52.092477+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:33:52.092477+0000","last_undegraded":"2024-03-18T16:33:52.092477+0000","last_fullsized":"2024-03-18T16:33:52.092477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:45:05.352704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:24.413279+0000","last_change":"2024-03-18T16:21:23.049483+0000","last_active":"2024-03-18T16:36:24.413279+0000","last_peered":"2024-03-18T16:36:24.413279+0000","last_clean":"2024-03-18T16:36:24.413279+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:36:24.413279+0000","last_undegraded":"2024-03-18T16:36:24.413279+0000","last_fullsized":"2024-03-18T16:36:24.413279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092498+0000","last_change":"2024-03-18T16:21:22.995726+0000","last_active":"2024-03-18T16:33:52.092498+0000","last_peered":"2024-03-18T16:33:52.092498+0000","last_clean":"2024-03-18T16:33:52.092498+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:33:52.092498+0000","last_undegraded":"2024-03-18T16:33:52.092498+0000","last_fullsized":"2024-03-18T16:33:52.092498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:33:16.880622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:24.489675+0000","last_change":"2024-03-18T16:21:22.949640+0000","last_active":"2024-03-18T16:35:24.489675+0000","last_peered":"2024-03-18T16:35:24.489675+0000","last_clean":"2024-03-18T16:35:24.489675+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:35:24.489675+0000","last_undegraded":"2024-03-18T16:35:24.489675+0000","last_fullsized":"2024-03-18T16:35:24.489675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:56.783318+0000","last_change":"2024-03-18T16:36:56.783359+0000","last_active":"2024-03-18T16:36:56.783318+0000","last_peered":"2024-03-18T16:36:56.783318+0000","last_clean":"2024-03-18T16:36:56.783318+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:36:56.783318+0000","last_undegraded":"2024-03-18T16:36:56.783318+0000","last_fullsized":"2024-03-18T16:36:56.783318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:56.783216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:56.783216+0000","last_clean_scrub_stamp":"2024-03-18T16:36:56.783216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:52.104516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:35:57.972268+0000","last_change":"2024-03-18T16:21:23.053003+0000","last_active":"2024-03-18T16:35:57.972268+0000","last_peered":"2024-03-18T16:35:57.972268+0000","last_clean":"2024-03-18T16:35:57.972268+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:35:57.972268+0000","last_undegraded":"2024-03-18T16:35:57.972268+0000","last_fullsized":"2024-03-18T16:35:57.972268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:33:52.092540+0000","last_change":"2024-03-18T16:21:23.041863+0000","last_active":"2024-03-18T16:33:52.092540+0000","last_peered":"2024-03-18T16:33:52.092540+0000","last_clean":"2024-03-18T16:33:52.092540+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:33:52.092540+0000","last_undegraded":"2024-03-18T16:33:52.092540+0000","last_fullsized":"2024-03-18T16:33:52.092540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:51:26.190759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509635,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":421888,"data_stored":729077,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509636,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":344064,"data_stored":137939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509634,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":352256,"data_stored":141275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509636,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":138141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509634,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":413696,"data_stored":727851,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542339,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":352256,"data_stored":138071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":141241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575043,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":140541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:37:01.376 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.376 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.376 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.376 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.376 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.377 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.377 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.377 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.377 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.377 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.377 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.377 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.377 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.377 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.378 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.378 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.378 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.378 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.378 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.378 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.378 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.378 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.378 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.379 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.379 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.379 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.379 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.379 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.379 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.379 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.379 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.379 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.380 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.381 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-18T16:21:05.847994+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=5, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.381 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.381 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.381 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.381 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.381 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.381 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.381 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.381 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.382 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.382 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.382 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.382 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.382 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.382 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.382 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.382 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.383 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.383 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.383 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.383 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.383 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.383 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.383 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.383 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.383 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.384 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.384 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.384 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.384 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.384 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.384 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.384 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.384 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.384 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.385 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.385 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.385 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.385 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.385 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.385 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.385 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.385 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-03-18T16:21:20.023353+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=20, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.385 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.386 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-03-18T16:21:27.706281+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=27, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.386 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.386 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.386 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.386 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.386 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.386 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.386 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.386 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.387 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.388 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.389 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:01.389 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-18T16:37:21.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:37:21.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.573+0000 7f13e3200640 1 Processor -- start 2024-03-18T16:37:21.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.573+0000 7f13e3200640 1 -- start start 2024-03-18T16:37:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.575+0000 7f13e3200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e40c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.575+0000 7f13e3200640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13e4072810 0x7f13e4072c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.575+0000 7f13e3200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e4073530 con 0x7f13e40c7f30 2024-03-18T16:37:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.575+0000 7f13e3200640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f13e40c5d80 con 0x7f13e4072810 2024-03-18T16:37:21.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.575+0000 7f13e3200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f13e4157900 con 0x7f13e4157d90 2024-03-18T16:37:21.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.575+0000 7f13e1400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e40c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:21.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.576+0000 7f13e2800640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13e4072810 0x7f13e4072c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:21.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.576+0000 7f13e1e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f13e40c7f30 0x7f13e415b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55300/0 (socket says 172.21.15.102:55300) 2024-03-18T16:37:21.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.576+0000 7f13e1e00640 1 -- 172.21.15.102:0/3608757475 learned_addr learned my addr 172.21.15.102:0/3608757475 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-18T16:37:21.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.578+0000 7f13e1400640 1 -- 172.21.15.102:0/3608757475 >> v2:172.21.15.154:3300/0 conn(0x7f13e4072810 msgr2=0x7f13e4072c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:37:21.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.578+0000 7f13e1400640 1 --2- 172.21.15.102:0/3608757475 >> v2:172.21.15.154:3300/0 conn(0x7f13e4072810 0x7f13e4072c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:21.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.578+0000 7f13e1400640 1 -- 172.21.15.102:0/3608757475 >> v1:172.21.15.102:6789/0 conn(0x7f13e40c7f30 legacy=0x7f13e415b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:21.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.578+0000 7f13e1400640 1 -- 172.21.15.102:0/3608757475 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e40c57e0 con 0x7f13e4157d90 2024-03-18T16:37:21.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.581+0000 7f13e1400640 1 --2- 172.21.15.102:0/3608757475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e40c2d90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f13d00097a0 tx=0x7f13d002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4c69ba90313d42f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:21.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.583+0000 7f13e0a00640 1 -- 172.21.15.102:0/3608757475 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0034070 con 0x7f13e4157d90 2024-03-18T16:37:21.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.585+0000 7f13e0a00640 1 -- 172.21.15.102:0/3608757475 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d0030800 con 0x7f13e4157d90 2024-03-18T16:37:21.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.585+0000 7f13e0a00640 1 -- 172.21.15.102:0/3608757475 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0030bd0 con 0x7f13e4157d90 2024-03-18T16:37:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.585+0000 7f13e3200640 1 -- 172.21.15.102:0/3608757475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 msgr2=0x7f13e40c2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.585+0000 7f13e3200640 1 --2- 172.21.15.102:0/3608757475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e40c2d90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f13d00097a0 tx=0x7f13d002b940 comp rx=0 tx=0).stop 2024-03-18T16:37:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.586+0000 7f13e3200640 1 -- 172.21.15.102:0/3608757475 shutdown_connections 2024-03-18T16:37:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.586+0000 7f13e3200640 1 --2- 172.21.15.102:0/3608757475 >> v2:172.21.15.154:3300/0 conn(0x7f13e4072810 0x7f13e4072c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.586+0000 7f13e3200640 1 --2- 172.21.15.102:0/3608757475 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e40c2d90 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.586+0000 7f13e3200640 1 -- 172.21.15.102:0/3608757475 >> 172.21.15.102:0/3608757475 conn(0x7f13e40cb890 msgr2=0x7f13e40cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:21.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.586+0000 7f13e3200640 1 -- 172.21.15.102:0/3608757475 shutdown_connections 2024-03-18T16:37:21.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.586+0000 7f13e3200640 1 -- 172.21.15.102:0/3608757475 wait complete. 2024-03-18T16:37:21.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.587+0000 7f13e3200640 1 Processor -- start 2024-03-18T16:37:21.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.588+0000 7f13e3200640 1 -- start start 2024-03-18T16:37:21.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.588+0000 7f13e3200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4072810 0x7f13e414b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:21.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.588+0000 7f13e3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13e40c7f30 0x7f13e414c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:21.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.588+0000 7f13e3200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e414a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:21.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.588+0000 7f13e3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f13e4157590 con 0x7f13e40c7f30 2024-03-18T16:37:21.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.588+0000 7f13e3200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41572b0 con 0x7f13e4072810 2024-03-18T16:37:21.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.588+0000 7f13e3200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f13e40c2940 con 0x7f13e4157d90 2024-03-18T16:37:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.589+0000 7f13e2800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e414a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.589+0000 7f13e1e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4072810 0x7f13e414b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.589+0000 7f13e1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13e40c7f30 0x7f13e414c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.590+0000 7f13e2800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e414a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:57952/0 (socket says 172.21.15.102:57952) 2024-03-18T16:37:21.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.590+0000 7f13e2800640 1 -- 172.21.15.102:0/1141225243 learned_addr learned my addr 172.21.15.102:0/1141225243 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:37:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.595+0000 7f13e2800640 1 -- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4072810 msgr2=0x7f13e414b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.595+0000 7f13e2800640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4072810 0x7f13e414b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.595+0000 7f13e2800640 1 -- 172.21.15.102:0/1141225243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13e40c7f30 msgr2=0x7f13e414c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.595+0000 7f13e2800640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13e40c7f30 0x7f13e414c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:21.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.595+0000 7f13e2800640 1 -- 172.21.15.102:0/1141225243 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e414cf70 con 0x7f13e4157d90 2024-03-18T16:37:21.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.598+0000 7f13e2800640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e414a250 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00efc0 tx=0x7f13dc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:21.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.602+0000 7f13da000640 1 -- 172.21.15.102:0/1141225243 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc018020 con 0x7f13e4157d90 2024-03-18T16:37:21.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.602+0000 7f13da000640 1 -- 172.21.15.102:0/1141225243 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13dc014480 con 0x7f13e4157d90 2024-03-18T16:37:21.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.602+0000 7f13da000640 1 -- 172.21.15.102:0/1141225243 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc014870 con 0x7f13e4157d90 2024-03-18T16:37:21.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.602+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e41f3980 con 0x7f13e4157d90 2024-03-18T16:37:21.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.602+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e41f3f00 con 0x7f13e4157d90 2024-03-18T16:37:21.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.604+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e4074fd0 con 0x7f13e4157d90 2024-03-18T16:37:21.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.636+0000 7f13da000640 1 -- 172.21.15.102:0/1141225243 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f13dc005d10 con 0x7f13e4157d90 2024-03-18T16:37:21.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.637+0000 7f13da000640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13c406bc20 0x7f13c406e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:21.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.638+0000 7f13e1e00640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13c406bc20 0x7f13c406e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:21.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.638+0000 7f13e1e00640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13c406bc20 0x7f13c406e0e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f13d4007fd0 tx=0x7f13d4007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:21.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.876+0000 7f13da000640 1 -- 172.21.15.102:0/1141225243 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f13dc058240 con 0x7f13e4157d90 2024-03-18T16:37:21.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:21.895+0000 7f13da000640 1 -- 172.21.15.102:0/1141225243 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f13dc022030 con 0x7f13e4157d90 2024-03-18T16:37:22.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.091+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f13e40c64b0 con 0x7f13c406bc20 2024-03-18T16:37:22.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.100+0000 7f13da000640 1 -- 172.21.15.102:0/1141225243 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332842 (secure 0 0 0) 0x7f13e40c64b0 con 0x7f13c406bc20 2024-03-18T16:37:22.106 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:37:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:37:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13c406bc20 msgr2=0x7f13c406e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13c406bc20 0x7f13c406e0e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f13d4007fd0 tx=0x7f13d4007e80 comp rx=0 tx=0).stop 2024-03-18T16:37:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 msgr2=0x7f13e414a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e414a250 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00efc0 tx=0x7f13dc00c490 comp rx=0 tx=0).stop 2024-03-18T16:37:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 shutdown_connections 2024-03-18T16:37:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f13c406bc20 0x7f13c406e0e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f13e4157d90 0x7f13e414a250 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13e40c7f30 0x7f13e414c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 --2- 172.21.15.102:0/1141225243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e4072810 0x7f13e414b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 >> 172.21.15.102:0/1141225243 conn(0x7f13e40cb890 msgr2=0x7f13e4150f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 shutdown_connections 2024-03-18T16:37:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:22.105+0000 7f13e3200640 1 -- 172.21.15.102:0/1141225243 wait complete. 2024-03-18T16:37:22.123 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-03-18T16:37:21.042199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223168,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751552,"statfs":{"total":773094113280,"available":772865589248,"internally_reserved":0,"allocated":2949120,"data_stored":2294136,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003188"},"pg_stats":[{"pgid":"2.1f","version":"30'19","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:55.744172+0000","last_change":"2024-03-18T16:36:55.744239+0000","last_active":"2024-03-18T16:36:55.744172+0000","last_peered":"2024-03-18T16:36:55.744172+0000","last_clean":"2024-03-18T16:36:55.744172+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:36:55.744172+0000","last_undegraded":"2024-03-18T16:36:55.744172+0000","last_fullsized":"2024-03-18T16:36:55.744172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-03-18T16:36:55.744050+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-03-18T16:36:55.744050+0000","last_clean_scrub_stamp":"2024-03-18T16:36:55.744050+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:11.820011+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.261746+0000","last_change":"2024-03-18T16:36:58.261782+0000","last_active":"2024-03-18T16:36:58.261746+0000","last_peered":"2024-03-18T16:36:58.261746+0000","last_clean":"2024-03-18T16:36:58.261746+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:36:58.261746+0000","last_undegraded":"2024-03-18T16:36:58.261746+0000","last_fullsized":"2024-03-18T16:36:58.261746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:58.261640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:58.261640+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.261640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:49.044786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:14.054386+0000","last_change":"2024-03-18T16:37:14.054425+0000","last_active":"2024-03-18T16:37:14.054386+0000","last_peered":"2024-03-18T16:37:14.054386+0000","last_clean":"2024-03-18T16:37:14.054386+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:37:14.054386+0000","last_undegraded":"2024-03-18T16:37:14.054386+0000","last_fullsized":"2024-03-18T16:37:14.054386+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:14.054277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:14.054277+0000","last_clean_scrub_stamp":"2024-03-18T16:37:14.054277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:21:34.384174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":50,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.237317+0000","last_change":"2024-03-18T16:37:03.237356+0000","last_active":"2024-03-18T16:37:03.237317+0000","last_peered":"2024-03-18T16:37:03.237317+0000","last_clean":"2024-03-18T16:37:03.237317+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:37:03.237317+0000","last_undegraded":"2024-03-18T16:37:03.237317+0000","last_fullsized":"2024-03-18T16:37:03.237317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-03-18T16:37:03.237215+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-03-18T16:37:03.237215+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.237215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:39.476865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.823610+0000","last_change":"2024-03-18T16:36:57.823648+0000","last_active":"2024-03-18T16:36:57.823610+0000","last_peered":"2024-03-18T16:36:57.823610+0000","last_clean":"2024-03-18T16:36:57.823610+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:36:57.823610+0000","last_undegraded":"2024-03-18T16:36:57.823610+0000","last_fullsized":"2024-03-18T16:36:57.823610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:57.823501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:57.823501+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.823501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:31.842624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.235049+0000","last_change":"2024-03-18T16:37:08.235087+0000","last_active":"2024-03-18T16:37:08.235049+0000","last_peered":"2024-03-18T16:37:08.235049+0000","last_clean":"2024-03-18T16:37:08.235049+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:37:08.235049+0000","last_undegraded":"2024-03-18T16:37:08.235049+0000","last_fullsized":"2024-03-18T16:37:08.235049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.234930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.234930+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.234930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:53.212320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"30'19","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.331077+0000","last_change":"2024-03-18T16:37:05.331141+0000","last_active":"2024-03-18T16:37:05.331077+0000","last_peered":"2024-03-18T16:37:05.331077+0000","last_clean":"2024-03-18T16:37:05.331077+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:37:05.331077+0000","last_undegraded":"2024-03-18T16:37:05.331077+0000","last_fullsized":"2024-03-18T16:37:05.331077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-03-18T16:37:05.330962+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-03-18T16:37:05.330962+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.330962+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:00.652262+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.649403+0000","last_change":"2024-03-18T16:37:04.649434+0000","last_active":"2024-03-18T16:37:04.649403+0000","last_peered":"2024-03-18T16:37:04.649403+0000","last_clean":"2024-03-18T16:37:04.649403+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:37:04.649403+0000","last_undegraded":"2024-03-18T16:37:04.649403+0000","last_fullsized":"2024-03-18T16:37:04.649403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.649288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.649288+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.649288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:06:47.727287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.263410+0000","last_change":"2024-03-18T16:37:09.263453+0000","last_active":"2024-03-18T16:37:09.263410+0000","last_peered":"2024-03-18T16:37:09.263410+0000","last_clean":"2024-03-18T16:37:09.263410+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:37:09.263410+0000","last_undegraded":"2024-03-18T16:37:09.263410+0000","last_fullsized":"2024-03-18T16:37:09.263410+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.263275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.263275+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.263275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:36.374129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.287032+0000","last_change":"2024-03-18T16:37:04.287084+0000","last_active":"2024-03-18T16:37:04.287032+0000","last_peered":"2024-03-18T16:37:04.287032+0000","last_clean":"2024-03-18T16:37:04.287032+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:37:04.287032+0000","last_undegraded":"2024-03-18T16:37:04.287032+0000","last_fullsized":"2024-03-18T16:37:04.287032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:37:04.286912+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:37:04.286912+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.286912+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:32.205872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.669736+0000","last_change":"2024-03-18T16:37:05.669766+0000","last_active":"2024-03-18T16:37:05.669736+0000","last_peered":"2024-03-18T16:37:05.669736+0000","last_clean":"2024-03-18T16:37:05.669736+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:37:05.669736+0000","last_undegraded":"2024-03-18T16:37:05.669736+0000","last_fullsized":"2024-03-18T16:37:05.669736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.669668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.669668+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.669668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:46.003848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.297217+0000","last_change":"2024-03-18T16:37:10.297240+0000","last_active":"2024-03-18T16:37:10.297217+0000","last_peered":"2024-03-18T16:37:10.297217+0000","last_clean":"2024-03-18T16:37:10.297217+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:37:10.297217+0000","last_undegraded":"2024-03-18T16:37:10.297217+0000","last_fullsized":"2024-03-18T16:37:10.297217+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.297151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.297151+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.297151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:46.536801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.629463+0000","last_change":"2024-03-18T16:37:05.629484+0000","last_active":"2024-03-18T16:37:05.629463+0000","last_peered":"2024-03-18T16:37:05.629463+0000","last_clean":"2024-03-18T16:37:05.629463+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:37:05.629463+0000","last_undegraded":"2024-03-18T16:37:05.629463+0000","last_fullsized":"2024-03-18T16:37:05.629463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.629391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.629391+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.629391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:09.571784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'6","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.287493+0000","last_change":"2024-03-18T16:37:03.287493+0000","last_active":"2024-03-18T16:37:03.287493+0000","last_peered":"2024-03-18T16:37:03.287493+0000","last_clean":"2024-03-18T16:37:03.287493+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:37:03.287493+0000","last_undegraded":"2024-03-18T16:37:03.287493+0000","last_fullsized":"2024-03-18T16:37:03.287493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-03-18T16:37:03.287337+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-03-18T16:37:03.287337+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.287337+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:56:24.467281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.339313+0000","last_change":"2024-03-18T16:37:11.339313+0000","last_active":"2024-03-18T16:37:11.339313+0000","last_peered":"2024-03-18T16:37:11.339313+0000","last_clean":"2024-03-18T16:37:11.339313+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:37:11.339313+0000","last_undegraded":"2024-03-18T16:37:11.339313+0000","last_fullsized":"2024-03-18T16:37:11.339313+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.339212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.339212+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.339212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:08.758806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.125587+0000","last_change":"2024-03-18T16:37:09.125622+0000","last_active":"2024-03-18T16:37:09.125587+0000","last_peered":"2024-03-18T16:37:09.125587+0000","last_clean":"2024-03-18T16:37:09.125587+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:37:09.125587+0000","last_undegraded":"2024-03-18T16:37:09.125587+0000","last_fullsized":"2024-03-18T16:37:09.125587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.125479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.125479+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.125479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:23:01.464282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"30'6","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.506773+0000","last_change":"2024-03-18T16:37:01.506831+0000","last_active":"2024-03-18T16:37:01.506773+0000","last_peered":"2024-03-18T16:37:01.506773+0000","last_clean":"2024-03-18T16:37:01.506773+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:37:01.506773+0000","last_undegraded":"2024-03-18T16:37:01.506773+0000","last_fullsized":"2024-03-18T16:37:01.506773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-03-18T16:37:01.506679+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-03-18T16:37:01.506679+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.506679+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:15.971938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.329573+0000","last_change":"2024-03-18T16:37:04.329573+0000","last_active":"2024-03-18T16:37:04.329573+0000","last_peered":"2024-03-18T16:37:04.329573+0000","last_clean":"2024-03-18T16:37:04.329573+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:37:04.329573+0000","last_undegraded":"2024-03-18T16:37:04.329573+0000","last_fullsized":"2024-03-18T16:37:04.329573+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.329478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.329478+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.329478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:05.494778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.055913+0000","last_change":"2024-03-18T16:37:03.055913+0000","last_active":"2024-03-18T16:37:03.055913+0000","last_peered":"2024-03-18T16:37:03.055913+0000","last_clean":"2024-03-18T16:37:03.055913+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:37:03.055913+0000","last_undegraded":"2024-03-18T16:37:03.055913+0000","last_fullsized":"2024-03-18T16:37:03.055913+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:03.055762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:03.055762+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.055762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:57.364301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"30'11","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.464940+0000","last_change":"2024-03-18T16:37:00.464989+0000","last_active":"2024-03-18T16:37:00.464940+0000","last_peered":"2024-03-18T16:37:00.464940+0000","last_clean":"2024-03-18T16:37:00.464940+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:37:00.464940+0000","last_undegraded":"2024-03-18T16:37:00.464940+0000","last_fullsized":"2024-03-18T16:37:00.464940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-03-18T16:37:00.464840+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-03-18T16:37:00.464840+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.464840+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:55.288952+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":29,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.060960+0000","last_change":"2024-03-18T16:37:06.060998+0000","last_active":"2024-03-18T16:37:06.060960+0000","last_peered":"2024-03-18T16:37:06.060960+0000","last_clean":"2024-03-18T16:37:06.060960+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:37:06.060960+0000","last_undegraded":"2024-03-18T16:37:06.060960+0000","last_fullsized":"2024-03-18T16:37:06.060960+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.060854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.060854+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.060854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:40:20.223068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.375469+0000","last_change":"2024-03-18T16:36:59.375515+0000","last_active":"2024-03-18T16:36:59.375469+0000","last_peered":"2024-03-18T16:36:59.375469+0000","last_clean":"2024-03-18T16:36:59.375469+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:36:59.375469+0000","last_undegraded":"2024-03-18T16:36:59.375469+0000","last_fullsized":"2024-03-18T16:36:59.375469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:59.375329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:59.375329+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.375329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:15.451290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'3","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.357087+0000","last_change":"2024-03-18T16:37:07.357138+0000","last_active":"2024-03-18T16:37:07.357087+0000","last_peered":"2024-03-18T16:37:07.357087+0000","last_clean":"2024-03-18T16:37:07.357087+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:37:07.357087+0000","last_undegraded":"2024-03-18T16:37:07.357087+0000","last_fullsized":"2024-03-18T16:37:07.357087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3","last_scrub_stamp":"2024-03-18T16:37:07.356980+0000","last_deep_scrub":"30'3","last_deep_scrub_stamp":"2024-03-18T16:37:07.356980+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.356980+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:02.181091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.292253+0000","last_change":"2024-03-18T16:37:05.292253+0000","last_active":"2024-03-18T16:37:05.292253+0000","last_peered":"2024-03-18T16:37:05.292253+0000","last_clean":"2024-03-18T16:37:05.292253+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:37:05.292253+0000","last_undegraded":"2024-03-18T16:37:05.292253+0000","last_fullsized":"2024-03-18T16:37:05.292253+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.292101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.292101+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.292101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:09.403145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.778425+0000","last_change":"2024-03-18T16:37:00.778458+0000","last_active":"2024-03-18T16:37:00.778425+0000","last_peered":"2024-03-18T16:37:00.778425+0000","last_clean":"2024-03-18T16:37:00.778425+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:37:00.778425+0000","last_undegraded":"2024-03-18T16:37:00.778425+0000","last_fullsized":"2024-03-18T16:37:00.778425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:00.778351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:00.778351+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.778351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:01.943650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'11","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.669353+0000","last_change":"2024-03-18T16:37:01.669407+0000","last_active":"2024-03-18T16:37:01.669353+0000","last_peered":"2024-03-18T16:37:01.669353+0000","last_clean":"2024-03-18T16:37:01.669353+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:37:01.669353+0000","last_undegraded":"2024-03-18T16:37:01.669353+0000","last_fullsized":"2024-03-18T16:37:01.669353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-03-18T16:37:01.669280+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-03-18T16:37:01.669280+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.669280+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:53.707032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.338416+0000","last_change":"2024-03-18T16:37:12.338455+0000","last_active":"2024-03-18T16:37:12.338416+0000","last_peered":"2024-03-18T16:37:12.338416+0000","last_clean":"2024-03-18T16:37:12.338416+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:37:12.338416+0000","last_undegraded":"2024-03-18T16:37:12.338416+0000","last_fullsized":"2024-03-18T16:37:12.338416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.338340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.338340+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.338340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:54.272465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.272266+0000","last_change":"2024-03-18T16:37:00.272301+0000","last_active":"2024-03-18T16:37:00.272266+0000","last_peered":"2024-03-18T16:37:00.272266+0000","last_clean":"2024-03-18T16:37:00.272266+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:37:00.272266+0000","last_undegraded":"2024-03-18T16:37:00.272266+0000","last_fullsized":"2024-03-18T16:37:00.272266+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:00.272158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:00.272158+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.272158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:29.140232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"30'11","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.472346+0000","last_change":"2024-03-18T16:36:58.472402+0000","last_active":"2024-03-18T16:36:58.472346+0000","last_peered":"2024-03-18T16:36:58.472346+0000","last_clean":"2024-03-18T16:36:58.472346+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:36:58.472346+0000","last_undegraded":"2024-03-18T16:36:58.472346+0000","last_fullsized":"2024-03-18T16:36:58.472346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-03-18T16:36:58.472229+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-03-18T16:36:58.472229+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.472229+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:53.793529+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:13.079212+0000","last_change":"2024-03-18T16:37:13.079250+0000","last_active":"2024-03-18T16:37:13.079212+0000","last_peered":"2024-03-18T16:37:13.079212+0000","last_clean":"2024-03-18T16:37:13.079212+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:37:13.079212+0000","last_undegraded":"2024-03-18T16:37:13.079212+0000","last_fullsized":"2024-03-18T16:37:13.079212+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:13.079107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:13.079107+0000","last_clean_scrub_stamp":"2024-03-18T16:37:13.079107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:18.445982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.513877+0000","last_change":"2024-03-18T16:37:03.513913+0000","last_active":"2024-03-18T16:37:03.513877+0000","last_peered":"2024-03-18T16:37:03.513877+0000","last_clean":"2024-03-18T16:37:03.513877+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:37:03.513877+0000","last_undegraded":"2024-03-18T16:37:03.513877+0000","last_fullsized":"2024-03-18T16:37:03.513877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:03.513739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:03.513739+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.513739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:07.624782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'7","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.368680+0000","last_change":"2024-03-18T16:37:06.368707+0000","last_active":"2024-03-18T16:37:06.368680+0000","last_peered":"2024-03-18T16:37:06.368680+0000","last_clean":"2024-03-18T16:37:06.368680+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:37:06.368680+0000","last_undegraded":"2024-03-18T16:37:06.368680+0000","last_fullsized":"2024-03-18T16:37:06.368680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-03-18T16:37:06.368613+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-03-18T16:37:06.368613+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.368613+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:40.387736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.661783+0000","last_change":"2024-03-18T16:37:07.661835+0000","last_active":"2024-03-18T16:37:07.661783+0000","last_peered":"2024-03-18T16:37:07.661783+0000","last_clean":"2024-03-18T16:37:07.661783+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:37:07.661783+0000","last_undegraded":"2024-03-18T16:37:07.661783+0000","last_fullsized":"2024-03-18T16:37:07.661783+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.661639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.661639+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.661639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:05.081233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.691194+0000","last_change":"2024-03-18T16:36:58.691214+0000","last_active":"2024-03-18T16:36:58.691194+0000","last_peered":"2024-03-18T16:36:58.691194+0000","last_clean":"2024-03-18T16:36:58.691194+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:36:58.691194+0000","last_undegraded":"2024-03-18T16:36:58.691194+0000","last_fullsized":"2024-03-18T16:36:58.691194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:58.691130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:58.691130+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.691130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:26.607438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"30'2","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:56.393346+0000","last_change":"2024-03-18T16:36:56.393387+0000","last_active":"2024-03-18T16:36:56.393346+0000","last_peered":"2024-03-18T16:36:56.393346+0000","last_clean":"2024-03-18T16:36:56.393346+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:36:56.393346+0000","last_undegraded":"2024-03-18T16:36:56.393346+0000","last_fullsized":"2024-03-18T16:36:56.393346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-03-18T16:36:56.393245+0000","last_deep_scrub":"30'2","last_deep_scrub_stamp":"2024-03-18T16:36:56.393245+0000","last_clean_scrub_stamp":"2024-03-18T16:36:56.393245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:10.756772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.544297+0000","last_change":"2024-03-18T16:37:08.544335+0000","last_active":"2024-03-18T16:37:08.544297+0000","last_peered":"2024-03-18T16:37:08.544297+0000","last_clean":"2024-03-18T16:37:08.544297+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:37:08.544297+0000","last_undegraded":"2024-03-18T16:37:08.544297+0000","last_fullsized":"2024-03-18T16:37:08.544297+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.544193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.544193+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.544193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:44.783842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.331718+0000","last_change":"2024-03-18T16:37:03.331737+0000","last_active":"2024-03-18T16:37:03.331718+0000","last_peered":"2024-03-18T16:37:03.331718+0000","last_clean":"2024-03-18T16:37:03.331718+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:37:03.331718+0000","last_undegraded":"2024-03-18T16:37:03.331718+0000","last_fullsized":"2024-03-18T16:37:03.331718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:03.331651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:03.331651+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.331651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:17.796909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"30'7","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.655618+0000","last_change":"2024-03-18T16:37:02.655663+0000","last_active":"2024-03-18T16:37:02.655618+0000","last_peered":"2024-03-18T16:37:02.655618+0000","last_clean":"2024-03-18T16:37:02.655618+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:37:02.655618+0000","last_undegraded":"2024-03-18T16:37:02.655618+0000","last_fullsized":"2024-03-18T16:37:02.655618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-03-18T16:37:02.655510+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-03-18T16:37:02.655510+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.655510+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:05.852420+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.555065+0000","last_change":"2024-03-18T16:37:07.555101+0000","last_active":"2024-03-18T16:37:07.555065+0000","last_peered":"2024-03-18T16:37:07.555065+0000","last_clean":"2024-03-18T16:37:07.555065+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:37:07.555065+0000","last_undegraded":"2024-03-18T16:37:07.555065+0000","last_fullsized":"2024-03-18T16:37:07.555065+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.554963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.554963+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.554963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:34.665816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:55.352796+0000","last_change":"2024-03-18T16:36:55.352845+0000","last_active":"2024-03-18T16:36:55.352796+0000","last_peered":"2024-03-18T16:36:55.352796+0000","last_clean":"2024-03-18T16:36:55.352796+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:36:55.352796+0000","last_undegraded":"2024-03-18T16:36:55.352796+0000","last_fullsized":"2024-03-18T16:36:55.352796+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:55.352684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:55.352684+0000","last_clean_scrub_stamp":"2024-03-18T16:36:55.352684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:14:01.365564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'8","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.105679+0000","last_change":"2024-03-18T16:36:58.105742+0000","last_active":"2024-03-18T16:36:58.105679+0000","last_peered":"2024-03-18T16:36:58.105679+0000","last_clean":"2024-03-18T16:36:58.105679+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:36:58.105679+0000","last_undegraded":"2024-03-18T16:36:58.105679+0000","last_fullsized":"2024-03-18T16:36:58.105679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:36:58.105545+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:36:58.105545+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.105545+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:05:38.967022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.518357+0000","last_change":"2024-03-18T16:37:09.518397+0000","last_active":"2024-03-18T16:37:09.518357+0000","last_peered":"2024-03-18T16:37:09.518357+0000","last_clean":"2024-03-18T16:37:09.518357+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:37:09.518357+0000","last_undegraded":"2024-03-18T16:37:09.518357+0000","last_fullsized":"2024-03-18T16:37:09.518357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.518228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.518228+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.518228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:06.937821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.728551+0000","last_change":"2024-03-18T16:36:59.728589+0000","last_active":"2024-03-18T16:36:59.728551+0000","last_peered":"2024-03-18T16:36:59.728551+0000","last_clean":"2024-03-18T16:36:59.728551+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:36:59.728551+0000","last_undegraded":"2024-03-18T16:36:59.728551+0000","last_fullsized":"2024-03-18T16:36:59.728551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:59.728414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:59.728414+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.728414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:14.193672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'5","reported_seq":53,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.325360+0000","last_change":"2024-03-18T16:37:08.325383+0000","last_active":"2024-03-18T16:37:08.325360+0000","last_peered":"2024-03-18T16:37:08.325360+0000","last_clean":"2024-03-18T16:37:08.325360+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:37:08.325360+0000","last_undegraded":"2024-03-18T16:37:08.325360+0000","last_fullsized":"2024-03-18T16:37:08.325360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5","last_scrub_stamp":"2024-03-18T16:37:08.325303+0000","last_deep_scrub":"30'5","last_deep_scrub_stamp":"2024-03-18T16:37:08.325303+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.325303+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:22.790561+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.649133+0000","last_change":"2024-03-18T16:37:08.649189+0000","last_active":"2024-03-18T16:37:08.649133+0000","last_peered":"2024-03-18T16:37:08.649133+0000","last_clean":"2024-03-18T16:37:08.649133+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:37:08.649133+0000","last_undegraded":"2024-03-18T16:37:08.649133+0000","last_fullsized":"2024-03-18T16:37:08.649133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.648874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.648874+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.648874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:50.583000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.209642+0000","last_change":"2024-03-18T16:37:07.209680+0000","last_active":"2024-03-18T16:37:07.209642+0000","last_peered":"2024-03-18T16:37:07.209642+0000","last_clean":"2024-03-18T16:37:07.209642+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:37:07.209642+0000","last_undegraded":"2024-03-18T16:37:07.209642+0000","last_fullsized":"2024-03-18T16:37:07.209642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.209541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.209541+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.209541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:36.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'11","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.673403+0000","last_change":"2024-03-18T16:37:00.673458+0000","last_active":"2024-03-18T16:37:00.673403+0000","last_peered":"2024-03-18T16:37:00.673403+0000","last_clean":"2024-03-18T16:37:00.673403+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:37:00.673403+0000","last_undegraded":"2024-03-18T16:37:00.673403+0000","last_fullsized":"2024-03-18T16:37:00.673403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-03-18T16:37:00.673295+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-03-18T16:37:00.673295+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.673295+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:06.659579+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"30'119","reported_seq":207,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.265455+0000","last_change":"2024-03-18T16:37:02.265455+0000","last_active":"2024-03-18T16:37:02.265455+0000","last_peered":"2024-03-18T16:37:02.265455+0000","last_clean":"2024-03-18T16:37:02.265455+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:37:02.265455+0000","last_undegraded":"2024-03-18T16:37:02.265455+0000","last_fullsized":"2024-03-18T16:37:02.265455+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'119","last_scrub_stamp":"2024-03-18T16:37:02.265291+0000","last_deep_scrub":"30'119","last_deep_scrub_stamp":"2024-03-18T16:37:02.265291+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.265291+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:20:50.601488+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:13.703509+0000","last_change":"2024-03-18T16:37:13.703530+0000","last_active":"2024-03-18T16:37:13.703509+0000","last_peered":"2024-03-18T16:37:13.703509+0000","last_clean":"2024-03-18T16:37:13.703509+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:37:13.703509+0000","last_undegraded":"2024-03-18T16:37:13.703509+0000","last_fullsized":"2024-03-18T16:37:13.703509+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:13.703406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:13.703406+0000","last_clean_scrub_stamp":"2024-03-18T16:37:13.703406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:12.650999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.325231+0000","last_change":"2024-03-18T16:37:10.325252+0000","last_active":"2024-03-18T16:37:10.325231+0000","last_peered":"2024-03-18T16:37:10.325231+0000","last_clean":"2024-03-18T16:37:10.325231+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:37:10.325231+0000","last_undegraded":"2024-03-18T16:37:10.325231+0000","last_fullsized":"2024-03-18T16:37:10.325231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.325165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.325165+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.325165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:14.436209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"30'9","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:56.047926+0000","last_change":"2024-03-18T16:36:56.047926+0000","last_active":"2024-03-18T16:36:56.047926+0000","last_peered":"2024-03-18T16:36:56.047926+0000","last_clean":"2024-03-18T16:36:56.047926+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:36:56.047926+0000","last_undegraded":"2024-03-18T16:36:56.047926+0000","last_fullsized":"2024-03-18T16:36:56.047926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-03-18T16:36:56.047790+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-03-18T16:36:56.047790+0000","last_clean_scrub_stamp":"2024-03-18T16:36:56.047790+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:55.302327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.698518+0000","last_change":"2024-03-18T16:37:12.698557+0000","last_active":"2024-03-18T16:37:12.698518+0000","last_peered":"2024-03-18T16:37:12.698518+0000","last_clean":"2024-03-18T16:37:12.698518+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:37:12.698518+0000","last_undegraded":"2024-03-18T16:37:12.698518+0000","last_fullsized":"2024-03-18T16:37:12.698518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.698398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.698398+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.698398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:44.293587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.662445+0000","last_change":"2024-03-18T16:37:07.662445+0000","last_active":"2024-03-18T16:37:07.662445+0000","last_peered":"2024-03-18T16:37:07.662445+0000","last_clean":"2024-03-18T16:37:07.662445+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:37:07.662445+0000","last_undegraded":"2024-03-18T16:37:07.662445+0000","last_fullsized":"2024-03-18T16:37:07.662445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.662326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.662326+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.662326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:50.438541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"30'5","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.427951+0000","last_change":"2024-03-18T16:36:57.427951+0000","last_active":"2024-03-18T16:36:57.427951+0000","last_peered":"2024-03-18T16:36:57.427951+0000","last_clean":"2024-03-18T16:36:57.427951+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:36:57.427951+0000","last_undegraded":"2024-03-18T16:36:57.427951+0000","last_fullsized":"2024-03-18T16:36:57.427951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5","last_scrub_stamp":"2024-03-18T16:36:57.427798+0000","last_deep_scrub":"30'5","last_deep_scrub_stamp":"2024-03-18T16:36:57.427798+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.427798+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:25.346758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.675835+0000","last_change":"2024-03-18T16:37:05.675835+0000","last_active":"2024-03-18T16:37:05.675835+0000","last_peered":"2024-03-18T16:37:05.675835+0000","last_clean":"2024-03-18T16:37:05.675835+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:37:05.675835+0000","last_undegraded":"2024-03-18T16:37:05.675835+0000","last_fullsized":"2024-03-18T16:37:05.675835+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.675679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.675679+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.675679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:32.786533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:14.227059+0000","last_change":"2024-03-18T16:37:14.227093+0000","last_active":"2024-03-18T16:37:14.227059+0000","last_peered":"2024-03-18T16:37:14.227059+0000","last_clean":"2024-03-18T16:37:14.227059+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:37:14.227059+0000","last_undegraded":"2024-03-18T16:37:14.227059+0000","last_fullsized":"2024-03-18T16:37:14.227059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:14.226948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:14.226948+0000","last_clean_scrub_stamp":"2024-03-18T16:37:14.226948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:41.337801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'8","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.109282+0000","last_change":"2024-03-18T16:36:59.109282+0000","last_active":"2024-03-18T16:36:59.109282+0000","last_peered":"2024-03-18T16:36:59.109282+0000","last_clean":"2024-03-18T16:36:59.109282+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:36:59.109282+0000","last_undegraded":"2024-03-18T16:36:59.109282+0000","last_fullsized":"2024-03-18T16:36:59.109282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:36:59.109184+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:36:59.109184+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.109184+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:02:56.365385+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:13.545858+0000","last_change":"2024-03-18T16:37:13.545915+0000","last_active":"2024-03-18T16:37:13.545858+0000","last_peered":"2024-03-18T16:37:13.545858+0000","last_clean":"2024-03-18T16:37:13.545858+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:37:13.545858+0000","last_undegraded":"2024-03-18T16:37:13.545858+0000","last_fullsized":"2024-03-18T16:37:13.545858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:13.545742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:13.545742+0000","last_clean_scrub_stamp":"2024-03-18T16:37:13.545742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:31.908499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:15.202557+0000","last_change":"2024-03-18T16:37:15.202579+0000","last_active":"2024-03-18T16:37:15.202557+0000","last_peered":"2024-03-18T16:37:15.202557+0000","last_clean":"2024-03-18T16:37:15.202557+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:37:15.202557+0000","last_undegraded":"2024-03-18T16:37:15.202557+0000","last_fullsized":"2024-03-18T16:37:15.202557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:15.202487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:15.202487+0000","last_clean_scrub_stamp":"2024-03-18T16:37:15.202487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:02:30.649558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"30'4","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.336474+0000","last_change":"2024-03-18T16:36:58.336474+0000","last_active":"2024-03-18T16:36:58.336474+0000","last_peered":"2024-03-18T16:36:58.336474+0000","last_clean":"2024-03-18T16:36:58.336474+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:36:58.336474+0000","last_undegraded":"2024-03-18T16:36:58.336474+0000","last_fullsized":"2024-03-18T16:36:58.336474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4","last_scrub_stamp":"2024-03-18T16:36:58.336388+0000","last_deep_scrub":"30'4","last_deep_scrub_stamp":"2024-03-18T16:36:58.336388+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.336388+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:14:04.349269+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.268332+0000","last_change":"2024-03-18T16:37:06.268354+0000","last_active":"2024-03-18T16:37:06.268332+0000","last_peered":"2024-03-18T16:37:06.268332+0000","last_clean":"2024-03-18T16:37:06.268332+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:37:06.268332+0000","last_undegraded":"2024-03-18T16:37:06.268332+0000","last_fullsized":"2024-03-18T16:37:06.268332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.268257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.268257+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.268257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:08.835557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.314526+0000","last_change":"2024-03-18T16:37:12.314561+0000","last_active":"2024-03-18T16:37:12.314526+0000","last_peered":"2024-03-18T16:37:12.314526+0000","last_clean":"2024-03-18T16:37:12.314526+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:37:12.314526+0000","last_undegraded":"2024-03-18T16:37:12.314526+0000","last_fullsized":"2024-03-18T16:37:12.314526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.314443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.314443+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.314443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:41.182517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'26","reported_seq":73,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.064571+0000","last_change":"2024-03-18T16:36:57.064626+0000","last_active":"2024-03-18T16:36:57.064571+0000","last_peered":"2024-03-18T16:36:57.064571+0000","last_clean":"2024-03-18T16:36:57.064571+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:36:57.064571+0000","last_undegraded":"2024-03-18T16:36:57.064571+0000","last_fullsized":"2024-03-18T16:36:57.064571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'26","last_scrub_stamp":"2024-03-18T16:36:57.064452+0000","last_deep_scrub":"30'26","last_deep_scrub_stamp":"2024-03-18T16:36:57.064452+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.064452+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:50.653357+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.669177+0000","last_change":"2024-03-18T16:37:06.669217+0000","last_active":"2024-03-18T16:37:06.669177+0000","last_peered":"2024-03-18T16:37:06.669177+0000","last_clean":"2024-03-18T16:37:06.669177+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:37:06.669177+0000","last_undegraded":"2024-03-18T16:37:06.669177+0000","last_fullsized":"2024-03-18T16:37:06.669177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.669079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.669079+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.669079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:02:22.116151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.717889+0000","last_change":"2024-03-18T16:37:09.717908+0000","last_active":"2024-03-18T16:37:09.717889+0000","last_peered":"2024-03-18T16:37:09.717889+0000","last_clean":"2024-03-18T16:37:09.717889+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:37:09.717889+0000","last_undegraded":"2024-03-18T16:37:09.717889+0000","last_fullsized":"2024-03-18T16:37:09.717889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.717824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.717824+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.717824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:50.052004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.695122+0000","last_change":"2024-03-18T16:37:02.695165+0000","last_active":"2024-03-18T16:37:02.695122+0000","last_peered":"2024-03-18T16:37:02.695122+0000","last_clean":"2024-03-18T16:37:02.695122+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:37:02.695122+0000","last_undegraded":"2024-03-18T16:37:02.695122+0000","last_fullsized":"2024-03-18T16:37:02.695122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:37:02.695054+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:37:02.695054+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.695054+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:30.611362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.110274+0000","last_change":"2024-03-18T16:37:11.110312+0000","last_active":"2024-03-18T16:37:11.110274+0000","last_peered":"2024-03-18T16:37:11.110274+0000","last_clean":"2024-03-18T16:37:11.110274+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:37:11.110274+0000","last_undegraded":"2024-03-18T16:37:11.110274+0000","last_fullsized":"2024-03-18T16:37:11.110274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.110160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.110160+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.110160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:55.397408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.277004+0000","last_change":"2024-03-18T16:37:01.277034+0000","last_active":"2024-03-18T16:37:01.277004+0000","last_peered":"2024-03-18T16:37:01.277004+0000","last_clean":"2024-03-18T16:37:01.277004+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:37:01.277004+0000","last_undegraded":"2024-03-18T16:37:01.277004+0000","last_fullsized":"2024-03-18T16:37:01.277004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:01.276933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:01.276933+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.276933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:00:29.923268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.610468+0000","last_change":"2024-03-18T16:37:03.610532+0000","last_active":"2024-03-18T16:37:03.610468+0000","last_peered":"2024-03-18T16:37:03.610468+0000","last_clean":"2024-03-18T16:37:03.610468+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:37:03.610468+0000","last_undegraded":"2024-03-18T16:37:03.610468+0000","last_fullsized":"2024-03-18T16:37:03.610468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:37:03.610361+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:37:03.610361+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.610361+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:05.826883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:16.016267+0000","last_change":"2024-03-18T16:37:16.016308+0000","last_active":"2024-03-18T16:37:16.016267+0000","last_peered":"2024-03-18T16:37:16.016267+0000","last_clean":"2024-03-18T16:37:16.016267+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:37:16.016267+0000","last_undegraded":"2024-03-18T16:37:16.016267+0000","last_fullsized":"2024-03-18T16:37:16.016267+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:16.016161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:16.016161+0000","last_clean_scrub_stamp":"2024-03-18T16:37:16.016161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:02.104378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.793901+0000","last_change":"2024-03-18T16:37:01.793939+0000","last_active":"2024-03-18T16:37:01.793901+0000","last_peered":"2024-03-18T16:37:01.793901+0000","last_clean":"2024-03-18T16:37:01.793901+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:37:01.793901+0000","last_undegraded":"2024-03-18T16:37:01.793901+0000","last_fullsized":"2024-03-18T16:37:01.793901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:01.793794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:01.793794+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.793794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:05.904838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'15","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:54.791306+0000","last_change":"2024-03-18T16:36:54.791362+0000","last_active":"2024-03-18T16:36:54.791306+0000","last_peered":"2024-03-18T16:36:54.791306+0000","last_clean":"2024-03-18T16:36:54.791306+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:36:54.791306+0000","last_undegraded":"2024-03-18T16:36:54.791306+0000","last_fullsized":"2024-03-18T16:36:54.791306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-03-18T16:36:54.791182+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-03-18T16:36:54.791182+0000","last_clean_scrub_stamp":"2024-03-18T16:36:54.791182+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:14:00.804062+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:14.500754+0000","last_change":"2024-03-18T16:37:14.500795+0000","last_active":"2024-03-18T16:37:14.500754+0000","last_peered":"2024-03-18T16:37:14.500754+0000","last_clean":"2024-03-18T16:37:14.500754+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:37:14.500754+0000","last_undegraded":"2024-03-18T16:37:14.500754+0000","last_fullsized":"2024-03-18T16:37:14.500754+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:14.500645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:14.500645+0000","last_clean_scrub_stamp":"2024-03-18T16:37:14.500645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:23:51.586558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:13.266144+0000","last_change":"2024-03-18T16:37:13.266183+0000","last_active":"2024-03-18T16:37:13.266144+0000","last_peered":"2024-03-18T16:37:13.266144+0000","last_clean":"2024-03-18T16:37:13.266144+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:37:13.266144+0000","last_undegraded":"2024-03-18T16:37:13.266144+0000","last_fullsized":"2024-03-18T16:37:13.266144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:13.266014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:13.266014+0000","last_clean_scrub_stamp":"2024-03-18T16:37:13.266014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:58.243405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'20","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.667065+0000","last_change":"2024-03-18T16:37:03.667098+0000","last_active":"2024-03-18T16:37:03.667065+0000","last_peered":"2024-03-18T16:37:03.667065+0000","last_clean":"2024-03-18T16:37:03.667065+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:37:03.667065+0000","last_undegraded":"2024-03-18T16:37:03.667065+0000","last_fullsized":"2024-03-18T16:37:03.667065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'20","last_scrub_stamp":"2024-03-18T16:37:03.666986+0000","last_deep_scrub":"30'20","last_deep_scrub_stamp":"2024-03-18T16:37:03.666986+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.666986+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:40:17.829200+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.500179+0000","last_change":"2024-03-18T16:37:10.500218+0000","last_active":"2024-03-18T16:37:10.500179+0000","last_peered":"2024-03-18T16:37:10.500179+0000","last_clean":"2024-03-18T16:37:10.500179+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:37:10.500179+0000","last_undegraded":"2024-03-18T16:37:10.500179+0000","last_fullsized":"2024-03-18T16:37:10.500179+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.500072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.500072+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.500072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:52.434197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.314779+0000","last_change":"2024-03-18T16:37:01.314799+0000","last_active":"2024-03-18T16:37:01.314779+0000","last_peered":"2024-03-18T16:37:01.314779+0000","last_clean":"2024-03-18T16:37:01.314779+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:37:01.314779+0000","last_undegraded":"2024-03-18T16:37:01.314779+0000","last_fullsized":"2024-03-18T16:37:01.314779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:01.314706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:01.314706+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.314706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:35.333829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.502759+0000","last_change":"2024-03-18T16:36:59.502807+0000","last_active":"2024-03-18T16:36:59.502759+0000","last_peered":"2024-03-18T16:36:59.502759+0000","last_clean":"2024-03-18T16:36:59.502759+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:36:59.502759+0000","last_undegraded":"2024-03-18T16:36:59.502759+0000","last_fullsized":"2024-03-18T16:36:59.502759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-03-18T16:36:59.502663+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-03-18T16:36:59.502663+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.502663+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:33.521786+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.251361+0000","last_change":"2024-03-18T16:37:07.251382+0000","last_active":"2024-03-18T16:37:07.251361+0000","last_peered":"2024-03-18T16:37:07.251361+0000","last_clean":"2024-03-18T16:37:07.251361+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:37:07.251361+0000","last_undegraded":"2024-03-18T16:37:07.251361+0000","last_fullsized":"2024-03-18T16:37:07.251361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.251292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.251292+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.251292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:36.119366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.774183+0000","last_change":"2024-03-18T16:36:58.774220+0000","last_active":"2024-03-18T16:36:58.774183+0000","last_peered":"2024-03-18T16:36:58.774183+0000","last_clean":"2024-03-18T16:36:58.774183+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:36:58.774183+0000","last_undegraded":"2024-03-18T16:36:58.774183+0000","last_fullsized":"2024-03-18T16:36:58.774183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:58.774078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:58.774078+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.774078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:53.598189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'6","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.673322+0000","last_change":"2024-03-18T16:37:01.673364+0000","last_active":"2024-03-18T16:37:01.673322+0000","last_peered":"2024-03-18T16:37:01.673322+0000","last_clean":"2024-03-18T16:37:01.673322+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:37:01.673322+0000","last_undegraded":"2024-03-18T16:37:01.673322+0000","last_fullsized":"2024-03-18T16:37:01.673322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-03-18T16:37:01.673229+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-03-18T16:37:01.673229+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.673229+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:48.436630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.134457+0000","last_change":"2024-03-18T16:37:10.134495+0000","last_active":"2024-03-18T16:37:10.134457+0000","last_peered":"2024-03-18T16:37:10.134457+0000","last_clean":"2024-03-18T16:37:10.134457+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:37:10.134457+0000","last_undegraded":"2024-03-18T16:37:10.134457+0000","last_fullsized":"2024-03-18T16:37:10.134457+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.134351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.134351+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.134351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:38:03.638453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.625577+0000","last_change":"2024-03-18T16:37:09.625610+0000","last_active":"2024-03-18T16:37:09.625577+0000","last_peered":"2024-03-18T16:37:09.625577+0000","last_clean":"2024-03-18T16:37:09.625577+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:37:09.625577+0000","last_undegraded":"2024-03-18T16:37:09.625577+0000","last_fullsized":"2024-03-18T16:37:09.625577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.625502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.625502+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.625502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:19:33.258443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'7","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.706119+0000","last_change":"2024-03-18T16:37:00.706144+0000","last_active":"2024-03-18T16:37:00.706119+0000","last_peered":"2024-03-18T16:37:00.706119+0000","last_clean":"2024-03-18T16:37:00.706119+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:37:00.706119+0000","last_undegraded":"2024-03-18T16:37:00.706119+0000","last_fullsized":"2024-03-18T16:37:00.706119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-03-18T16:37:00.706035+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-03-18T16:37:00.706035+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.706035+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:55.014574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.089110+0000","last_change":"2024-03-18T16:37:12.089148+0000","last_active":"2024-03-18T16:37:12.089110+0000","last_peered":"2024-03-18T16:37:12.089110+0000","last_clean":"2024-03-18T16:37:12.089110+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:37:12.089110+0000","last_undegraded":"2024-03-18T16:37:12.089110+0000","last_fullsized":"2024-03-18T16:37:12.089110+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.089006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.089006+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.089006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:11.068314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.607377+0000","last_change":"2024-03-18T16:37:09.607417+0000","last_active":"2024-03-18T16:37:09.607377+0000","last_peered":"2024-03-18T16:37:09.607377+0000","last_clean":"2024-03-18T16:37:09.607377+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:37:09.607377+0000","last_undegraded":"2024-03-18T16:37:09.607377+0000","last_fullsized":"2024-03-18T16:37:09.607377+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.607267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.607267+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.607267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.050614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.509351+0000","last_change":"2024-03-18T16:37:05.509386+0000","last_active":"2024-03-18T16:37:05.509351+0000","last_peered":"2024-03-18T16:37:05.509351+0000","last_clean":"2024-03-18T16:37:05.509351+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:37:05.509351+0000","last_undegraded":"2024-03-18T16:37:05.509351+0000","last_fullsized":"2024-03-18T16:37:05.509351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.509242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.509242+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.509242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:34.377316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'4","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.717844+0000","last_change":"2024-03-18T16:36:59.717869+0000","last_active":"2024-03-18T16:36:59.717844+0000","last_peered":"2024-03-18T16:36:59.717844+0000","last_clean":"2024-03-18T16:36:59.717844+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:36:59.717844+0000","last_undegraded":"2024-03-18T16:36:59.717844+0000","last_fullsized":"2024-03-18T16:36:59.717844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4","last_scrub_stamp":"2024-03-18T16:36:59.717783+0000","last_deep_scrub":"30'4","last_deep_scrub_stamp":"2024-03-18T16:36:59.717783+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.717783+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:47:32.312475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.721987+0000","last_change":"2024-03-18T16:37:11.722032+0000","last_active":"2024-03-18T16:37:11.721987+0000","last_peered":"2024-03-18T16:37:11.721987+0000","last_clean":"2024-03-18T16:37:11.721987+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:37:11.721987+0000","last_undegraded":"2024-03-18T16:37:11.721987+0000","last_fullsized":"2024-03-18T16:37:11.721987+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.721872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.721872+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.721872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:54.498086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.267909+0000","last_change":"2024-03-18T16:36:59.267943+0000","last_active":"2024-03-18T16:36:59.267909+0000","last_peered":"2024-03-18T16:36:59.267909+0000","last_clean":"2024-03-18T16:36:59.267909+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:36:59.267909+0000","last_undegraded":"2024-03-18T16:36:59.267909+0000","last_fullsized":"2024-03-18T16:36:59.267909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:59.267795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:59.267795+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.267795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:01.835095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"30'5","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.274927+0000","last_change":"2024-03-18T16:37:02.274969+0000","last_active":"2024-03-18T16:37:02.274927+0000","last_peered":"2024-03-18T16:37:02.274927+0000","last_clean":"2024-03-18T16:37:02.274927+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:37:02.274927+0000","last_undegraded":"2024-03-18T16:37:02.274927+0000","last_fullsized":"2024-03-18T16:37:02.274927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5","last_scrub_stamp":"2024-03-18T16:37:02.274829+0000","last_deep_scrub":"30'5","last_deep_scrub_stamp":"2024-03-18T16:37:02.274829+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.274829+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:29.385683+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"30'10","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.753727+0000","last_change":"2024-03-18T16:36:59.753783+0000","last_active":"2024-03-18T16:36:59.753727+0000","last_peered":"2024-03-18T16:36:59.753727+0000","last_clean":"2024-03-18T16:36:59.753727+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:36:59.753727+0000","last_undegraded":"2024-03-18T16:36:59.753727+0000","last_fullsized":"2024-03-18T16:36:59.753727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-03-18T16:36:59.753619+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-03-18T16:36:59.753619+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.753619+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:24.570326+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.764583+0000","last_change":"2024-03-18T16:37:03.764618+0000","last_active":"2024-03-18T16:37:03.764583+0000","last_peered":"2024-03-18T16:37:03.764583+0000","last_clean":"2024-03-18T16:37:03.764583+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:37:03.764583+0000","last_undegraded":"2024-03-18T16:37:03.764583+0000","last_fullsized":"2024-03-18T16:37:03.764583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:03.764480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:03.764480+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.764480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:32.632554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.030113+0000","last_change":"2024-03-18T16:37:02.030150+0000","last_active":"2024-03-18T16:37:02.030113+0000","last_peered":"2024-03-18T16:37:02.030113+0000","last_clean":"2024-03-18T16:37:02.030113+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:37:02.030113+0000","last_undegraded":"2024-03-18T16:37:02.030113+0000","last_fullsized":"2024-03-18T16:37:02.030113+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:02.030002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:02.030002+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.030002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:26.846709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'7","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.242321+0000","last_change":"2024-03-18T16:37:01.242366+0000","last_active":"2024-03-18T16:37:01.242321+0000","last_peered":"2024-03-18T16:37:01.242321+0000","last_clean":"2024-03-18T16:37:01.242321+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:37:01.242321+0000","last_undegraded":"2024-03-18T16:37:01.242321+0000","last_fullsized":"2024-03-18T16:37:01.242321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-03-18T16:37:01.242229+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-03-18T16:37:01.242229+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.242229+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:46.219619+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.342562+0000","last_change":"2024-03-18T16:37:00.342582+0000","last_active":"2024-03-18T16:37:00.342562+0000","last_peered":"2024-03-18T16:37:00.342562+0000","last_clean":"2024-03-18T16:37:00.342562+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:37:00.342562+0000","last_undegraded":"2024-03-18T16:37:00.342562+0000","last_fullsized":"2024-03-18T16:37:00.342562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:00.342470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:00.342470+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.342470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:55.663770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:15.022833+0000","last_change":"2024-03-18T16:37:15.022872+0000","last_active":"2024-03-18T16:37:15.022833+0000","last_peered":"2024-03-18T16:37:15.022833+0000","last_clean":"2024-03-18T16:37:15.022833+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:37:15.022833+0000","last_undegraded":"2024-03-18T16:37:15.022833+0000","last_fullsized":"2024-03-18T16:37:15.022833+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:15.022712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:15.022712+0000","last_clean_scrub_stamp":"2024-03-18T16:37:15.022712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:24.871279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.699476+0000","last_change":"2024-03-18T16:37:10.699509+0000","last_active":"2024-03-18T16:37:10.699476+0000","last_peered":"2024-03-18T16:37:10.699476+0000","last_clean":"2024-03-18T16:37:10.699476+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:37:10.699476+0000","last_undegraded":"2024-03-18T16:37:10.699476+0000","last_fullsized":"2024-03-18T16:37:10.699476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.699371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.699371+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.699371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:52.794130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.088938+0000","last_change":"2024-03-18T16:37:00.088973+0000","last_active":"2024-03-18T16:37:00.088938+0000","last_peered":"2024-03-18T16:37:00.088938+0000","last_clean":"2024-03-18T16:37:00.088938+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:37:00.088938+0000","last_undegraded":"2024-03-18T16:37:00.088938+0000","last_fullsized":"2024-03-18T16:37:00.088938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:00.088814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:00.088814+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.088814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:11:15.114511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.089324+0000","last_change":"2024-03-18T16:37:04.089360+0000","last_active":"2024-03-18T16:37:04.089324+0000","last_peered":"2024-03-18T16:37:04.089324+0000","last_clean":"2024-03-18T16:37:04.089324+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:37:04.089324+0000","last_undegraded":"2024-03-18T16:37:04.089324+0000","last_fullsized":"2024-03-18T16:37:04.089324+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.089221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.089221+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.089221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:56.126973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.745455+0000","last_change":"2024-03-18T16:37:02.745493+0000","last_active":"2024-03-18T16:37:02.745455+0000","last_peered":"2024-03-18T16:37:02.745455+0000","last_clean":"2024-03-18T16:37:02.745455+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:37:02.745455+0000","last_undegraded":"2024-03-18T16:37:02.745455+0000","last_fullsized":"2024-03-18T16:37:02.745455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:02.745349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:02.745349+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.745349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:05.312649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.244991+0000","last_change":"2024-03-18T16:37:04.245026+0000","last_active":"2024-03-18T16:37:04.244991+0000","last_peered":"2024-03-18T16:37:04.244991+0000","last_clean":"2024-03-18T16:37:04.244991+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:37:04.244991+0000","last_undegraded":"2024-03-18T16:37:04.244991+0000","last_fullsized":"2024-03-18T16:37:04.244991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.244837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.244837+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.244837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:01.664431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.520840+0000","last_change":"2024-03-18T16:37:04.520874+0000","last_active":"2024-03-18T16:37:04.520840+0000","last_peered":"2024-03-18T16:37:04.520840+0000","last_clean":"2024-03-18T16:37:04.520840+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:37:04.520840+0000","last_undegraded":"2024-03-18T16:37:04.520840+0000","last_fullsized":"2024-03-18T16:37:04.520840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.520722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.520722+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.520722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:07.088022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.690340+0000","last_change":"2024-03-18T16:37:07.690360+0000","last_active":"2024-03-18T16:37:07.690340+0000","last_peered":"2024-03-18T16:37:07.690340+0000","last_clean":"2024-03-18T16:37:07.690340+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:37:07.690340+0000","last_undegraded":"2024-03-18T16:37:07.690340+0000","last_fullsized":"2024-03-18T16:37:07.690340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.690271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.690271+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.690271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:09.906793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.229073+0000","last_change":"2024-03-18T16:37:05.229119+0000","last_active":"2024-03-18T16:37:05.229073+0000","last_peered":"2024-03-18T16:37:05.229073+0000","last_clean":"2024-03-18T16:37:05.229073+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:37:05.229073+0000","last_undegraded":"2024-03-18T16:37:05.229073+0000","last_fullsized":"2024-03-18T16:37:05.229073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.228949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.228949+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.228949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:47.163075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.663205+0000","last_change":"2024-03-18T16:37:06.663224+0000","last_active":"2024-03-18T16:37:06.663205+0000","last_peered":"2024-03-18T16:37:06.663205+0000","last_clean":"2024-03-18T16:37:06.663205+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:37:06.663205+0000","last_undegraded":"2024-03-18T16:37:06.663205+0000","last_fullsized":"2024-03-18T16:37:06.663205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.663150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.663150+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.663150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:48.757909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.231593+0000","last_change":"2024-03-18T16:37:06.231593+0000","last_active":"2024-03-18T16:37:06.231593+0000","last_peered":"2024-03-18T16:37:06.231593+0000","last_clean":"2024-03-18T16:37:06.231593+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:37:06.231593+0000","last_undegraded":"2024-03-18T16:37:06.231593+0000","last_fullsized":"2024-03-18T16:37:06.231593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.231456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.231456+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.231456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:02.674803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":27,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.042256+0000","last_change":"2024-03-18T16:37:05.042292+0000","last_active":"2024-03-18T16:37:05.042256+0000","last_peered":"2024-03-18T16:37:05.042256+0000","last_clean":"2024-03-18T16:37:05.042256+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:37:05.042256+0000","last_undegraded":"2024-03-18T16:37:05.042256+0000","last_fullsized":"2024-03-18T16:37:05.042256+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.042153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.042153+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.042153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:32.958462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.723414+0000","last_change":"2024-03-18T16:37:04.723414+0000","last_active":"2024-03-18T16:37:04.723414+0000","last_peered":"2024-03-18T16:37:04.723414+0000","last_clean":"2024-03-18T16:37:04.723414+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:37:04.723414+0000","last_undegraded":"2024-03-18T16:37:04.723414+0000","last_fullsized":"2024-03-18T16:37:04.723414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.723282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.723282+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.723282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:49.700672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.701724+0000","last_change":"2024-03-18T16:37:08.701724+0000","last_active":"2024-03-18T16:37:08.701724+0000","last_peered":"2024-03-18T16:37:08.701724+0000","last_clean":"2024-03-18T16:37:08.701724+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:37:08.701724+0000","last_undegraded":"2024-03-18T16:37:08.701724+0000","last_fullsized":"2024-03-18T16:37:08.701724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.701635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.701635+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.701635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:06:51.779634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":27,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.537132+0000","last_change":"2024-03-18T16:37:11.537166+0000","last_active":"2024-03-18T16:37:11.537132+0000","last_peered":"2024-03-18T16:37:11.537132+0000","last_clean":"2024-03-18T16:37:11.537132+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:37:11.537132+0000","last_undegraded":"2024-03-18T16:37:11.537132+0000","last_fullsized":"2024-03-18T16:37:11.537132+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.537030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.537030+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.537030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:07.980377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.077472+0000","last_change":"2024-03-18T16:37:01.077509+0000","last_active":"2024-03-18T16:37:01.077472+0000","last_peered":"2024-03-18T16:37:01.077472+0000","last_clean":"2024-03-18T16:37:01.077472+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:37:01.077472+0000","last_undegraded":"2024-03-18T16:37:01.077472+0000","last_fullsized":"2024-03-18T16:37:01.077472+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:01.077360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:01.077360+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.077360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:17.832116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.586524+0000","last_change":"2024-03-18T16:37:10.586558+0000","last_active":"2024-03-18T16:37:10.586524+0000","last_peered":"2024-03-18T16:37:10.586524+0000","last_clean":"2024-03-18T16:37:10.586524+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:37:10.586524+0000","last_undegraded":"2024-03-18T16:37:10.586524+0000","last_fullsized":"2024-03-18T16:37:10.586524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.586401+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.586401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:45:22.147206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.581559+0000","last_change":"2024-03-18T16:37:12.581594+0000","last_active":"2024-03-18T16:37:12.581559+0000","last_peered":"2024-03-18T16:37:12.581559+0000","last_clean":"2024-03-18T16:37:12.581559+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:37:12.581559+0000","last_undegraded":"2024-03-18T16:37:12.581559+0000","last_fullsized":"2024-03-18T16:37:12.581559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.581455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.581455+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.581455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:24.957908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.526857+0000","last_change":"2024-03-18T16:37:02.526893+0000","last_active":"2024-03-18T16:37:02.526857+0000","last_peered":"2024-03-18T16:37:02.526857+0000","last_clean":"2024-03-18T16:37:02.526857+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:37:02.526857+0000","last_undegraded":"2024-03-18T16:37:02.526857+0000","last_fullsized":"2024-03-18T16:37:02.526857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:02.526715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:02.526715+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.526715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:03.692015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:53.738279+0000","last_change":"2024-03-18T16:36:53.738313+0000","last_active":"2024-03-18T16:36:53.738279+0000","last_peered":"2024-03-18T16:36:53.738279+0000","last_clean":"2024-03-18T16:36:53.738279+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:36:53.738279+0000","last_undegraded":"2024-03-18T16:36:53.738279+0000","last_fullsized":"2024-03-18T16:36:53.738279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:53.738175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:53.738175+0000","last_clean_scrub_stamp":"2024-03-18T16:36:53.738175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:57.558446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.277459+0000","last_change":"2024-03-18T16:36:57.277509+0000","last_active":"2024-03-18T16:36:57.277459+0000","last_peered":"2024-03-18T16:36:57.277459+0000","last_clean":"2024-03-18T16:36:57.277459+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:36:57.277459+0000","last_undegraded":"2024-03-18T16:36:57.277459+0000","last_fullsized":"2024-03-18T16:36:57.277459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:57.277271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:57.277271+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.277271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:51:58.442571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.295813+0000","last_change":"2024-03-18T16:37:02.295833+0000","last_active":"2024-03-18T16:37:02.295813+0000","last_peered":"2024-03-18T16:37:02.295813+0000","last_clean":"2024-03-18T16:37:02.295813+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:37:02.295813+0000","last_undegraded":"2024-03-18T16:37:02.295813+0000","last_fullsized":"2024-03-18T16:37:02.295813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:02.295733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:02.295733+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.295733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:57.119844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.677882+0000","last_change":"2024-03-18T16:37:04.677904+0000","last_active":"2024-03-18T16:37:04.677882+0000","last_peered":"2024-03-18T16:37:04.677882+0000","last_clean":"2024-03-18T16:37:04.677882+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:37:04.677882+0000","last_undegraded":"2024-03-18T16:37:04.677882+0000","last_fullsized":"2024-03-18T16:37:04.677882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.677810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.677810+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.677810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:47:37.272501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.654558+0000","last_change":"2024-03-18T16:37:06.654579+0000","last_active":"2024-03-18T16:37:06.654558+0000","last_peered":"2024-03-18T16:37:06.654558+0000","last_clean":"2024-03-18T16:37:06.654558+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:37:06.654558+0000","last_undegraded":"2024-03-18T16:37:06.654558+0000","last_fullsized":"2024-03-18T16:37:06.654558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.654475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.654475+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.654475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:52.914426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.361511+0000","last_change":"2024-03-18T16:37:09.361545+0000","last_active":"2024-03-18T16:37:09.361511+0000","last_peered":"2024-03-18T16:37:09.361511+0000","last_clean":"2024-03-18T16:37:09.361511+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:37:09.361511+0000","last_undegraded":"2024-03-18T16:37:09.361511+0000","last_fullsized":"2024-03-18T16:37:09.361511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.361390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.361390+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.361390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:10.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.173994+0000","last_change":"2024-03-18T16:37:08.174029+0000","last_active":"2024-03-18T16:37:08.173994+0000","last_peered":"2024-03-18T16:37:08.173994+0000","last_clean":"2024-03-18T16:37:08.173994+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:37:08.173994+0000","last_undegraded":"2024-03-18T16:37:08.173994+0000","last_fullsized":"2024-03-18T16:37:08.173994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.173886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.173886+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.173886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:32.282777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":37,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:22.026063+0000","last_change":"2024-03-18T16:21:23.049877+0000","last_active":"2024-03-18T16:36:22.026063+0000","last_peered":"2024-03-18T16:36:22.026063+0000","last_clean":"2024-03-18T16:36:22.026063+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:36:22.026063+0000","last_undegraded":"2024-03-18T16:36:22.026063+0000","last_fullsized":"2024-03-18T16:36:22.026063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:21:21.155037+0000","last_clean_scrub_stamp":"2024-03-18T16:21:21.155037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.629366+0000","last_change":"2024-03-18T16:37:11.629405+0000","last_active":"2024-03-18T16:37:11.629366+0000","last_peered":"2024-03-18T16:37:11.629366+0000","last_clean":"2024-03-18T16:37:11.629366+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:37:11.629366+0000","last_undegraded":"2024-03-18T16:37:11.629366+0000","last_fullsized":"2024-03-18T16:37:11.629366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.629262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.629262+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.629262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:29.777022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.282220+0000","last_change":"2024-03-18T16:37:11.282259+0000","last_active":"2024-03-18T16:37:11.282220+0000","last_peered":"2024-03-18T16:37:11.282220+0000","last_clean":"2024-03-18T16:37:11.282220+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:37:11.282220+0000","last_undegraded":"2024-03-18T16:37:11.282220+0000","last_fullsized":"2024-03-18T16:37:11.282220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.282101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.282101+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.282101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:13.849401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.628525+0000","last_change":"2024-03-18T16:37:08.628559+0000","last_active":"2024-03-18T16:37:08.628525+0000","last_peered":"2024-03-18T16:37:08.628525+0000","last_clean":"2024-03-18T16:37:08.628525+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:37:08.628525+0000","last_undegraded":"2024-03-18T16:37:08.628525+0000","last_fullsized":"2024-03-18T16:37:08.628525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.628449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.628449+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.628449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:40.223638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:56.783318+0000","last_change":"2024-03-18T16:36:56.783359+0000","last_active":"2024-03-18T16:36:56.783318+0000","last_peered":"2024-03-18T16:36:56.783318+0000","last_clean":"2024-03-18T16:36:56.783318+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:36:56.783318+0000","last_undegraded":"2024-03-18T16:36:56.783318+0000","last_fullsized":"2024-03-18T16:36:56.783318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:56.783216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:56.783216+0000","last_clean_scrub_stamp":"2024-03-18T16:36:56.783216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:52.104516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.507540+0000","last_change":"2024-03-18T16:37:06.507574+0000","last_active":"2024-03-18T16:37:06.507540+0000","last_peered":"2024-03-18T16:37:06.507540+0000","last_clean":"2024-03-18T16:37:06.507540+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:37:06.507540+0000","last_undegraded":"2024-03-18T16:37:06.507540+0000","last_fullsized":"2024-03-18T16:37:06.507540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.507434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.507434+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.507434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:51.484825+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:16.194824+0000","last_change":"2024-03-18T16:37:16.194860+0000","last_active":"2024-03-18T16:37:16.194824+0000","last_peered":"2024-03-18T16:37:16.194824+0000","last_clean":"2024-03-18T16:37:16.194824+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:37:16.194824+0000","last_undegraded":"2024-03-18T16:37:16.194824+0000","last_fullsized":"2024-03-18T16:37:16.194824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:16.194699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:16.194699+0000","last_clean_scrub_stamp":"2024-03-18T16:37:16.194699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:13.614293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509639,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":421888,"data_stored":729077,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509640,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":344064,"data_stored":137939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509638,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":352256,"data_stored":141275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509640,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":138141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509638,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":413696,"data_stored":727851,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542343,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":352256,"data_stored":138071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575047,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":141241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575047,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":140541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:37:22.130 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-18T16:21:21.155037+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=21, tm_sec=21, tm_wday=0, tm_yday=78, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=18, tm_hour=16, tm_min=36, tm_sec=51, tm_wday=0, tm_yday=78, tm_isdst=0) 2024-03-18T16:37:22.130 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-18T16:37:24.399 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:16:50.658 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:24.405 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:16:50.665 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:24.458 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:16:50.717 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:24.461 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:16:50.806 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:24.464 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:16:50.809 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:24.475 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:50.820 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:24.479 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:50.824 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:24.507 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:16:50.852 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:24.545 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:16:50.890 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:24.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:37:24.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:37:42.131 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:37:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.303+0000 7fa4c7e00640 1 Processor -- start 2024-03-18T16:37:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.303+0000 7fa4c7e00640 1 -- start start 2024-03-18T16:37:42.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.304+0000 7fa4c7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa4c8154520 0x7fa4c80c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.304+0000 7fa4c7e00640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa4c80c3890 0x7fa4c80c8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.304+0000 7fa4c7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c80c3cd0 0x7fa4c80c9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.304+0000 7fa4c7e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c8154900 con 0x7fa4c8154520 2024-03-18T16:37:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.305+0000 7fa4c7e00640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c815a170 con 0x7fa4c80c3890 2024-03-18T16:37:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.305+0000 7fa4c7e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c80c4160 con 0x7fa4c80c3cd0 2024-03-18T16:37:42.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.305+0000 7fa4c7400640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c80c3cd0 0x7fa4c80c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:42.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.305+0000 7fa4c6000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa4c80c3890 0x7fa4c80c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:42.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.305+0000 7fa4c6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa4c8154520 0x7fa4c80c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:42.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.306+0000 7fa4c6000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa4c80c3890 0x7fa4c80c8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:37078/0 (socket says 172.21.15.102:37078) 2024-03-18T16:37:42.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.306+0000 7fa4c6000640 1 -- 172.21.15.102:0/3522524515 learned_addr learned my addr 172.21.15.102:0/3522524515 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:37:42.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.308+0000 7fa4c7400640 1 -- 172.21.15.102:0/3522524515 >> v2:172.21.15.154:3300/0 conn(0x7fa4c80c3890 msgr2=0x7fa4c80c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:37:42.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.308+0000 7fa4c7400640 1 --2- 172.21.15.102:0/3522524515 >> v2:172.21.15.154:3300/0 conn(0x7fa4c80c3890 0x7fa4c80c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.308+0000 7fa4c7400640 1 -- 172.21.15.102:0/3522524515 >> v2:172.21.15.102:3300/0 conn(0x7fa4c8154520 msgr2=0x7fa4c80c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:37:42.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.308+0000 7fa4c7400640 1 --2- 172.21.15.102:0/3522524515 >> v2:172.21.15.102:3300/0 conn(0x7fa4c8154520 0x7fa4c80c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.308+0000 7fa4c7400640 1 -- 172.21.15.102:0/3522524515 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c80c4440 con 0x7fa4c80c3cd0 2024-03-18T16:37:42.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.311+0000 7fa4c7400640 1 --2- 172.21.15.102:0/3522524515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c80c3cd0 0x7fa4c80c9370 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa4c806e4f0 tx=0x7fa4c002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddf79dbd5f4f0573 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:42.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.315+0000 7fa4c5600640 1 -- 172.21.15.102:0/3522524515 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0039070 con 0x7fa4c80c3cd0 2024-03-18T16:37:42.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.315+0000 7fa4c5600640 1 -- 172.21.15.102:0/3522524515 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c002cbd0 con 0x7fa4c80c3cd0 2024-03-18T16:37:42.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.315+0000 7fa4c5600640 1 -- 172.21.15.102:0/3522524515 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0034440 con 0x7fa4c80c3cd0 2024-03-18T16:37:42.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.315+0000 7fa4c7e00640 1 -- 172.21.15.102:0/3522524515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c80c3cd0 msgr2=0x7fa4c80c9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:42.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.315+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/3522524515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c80c3cd0 0x7fa4c80c9370 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa4c806e4f0 tx=0x7fa4c002be20 comp rx=0 tx=0).stop 2024-03-18T16:37:42.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.315+0000 7fa4c7e00640 1 -- 172.21.15.102:0/3522524515 shutdown_connections 2024-03-18T16:37:42.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.315+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/3522524515 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c80c3cd0 0x7fa4c80c9370 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.315+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/3522524515 >> v2:172.21.15.154:3300/0 conn(0x7fa4c80c3890 0x7fa4c80c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.315+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/3522524515 >> v2:172.21.15.102:3300/0 conn(0x7fa4c8154520 0x7fa4c80c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.316+0000 7fa4c7e00640 1 -- 172.21.15.102:0/3522524515 >> 172.21.15.102:0/3522524515 conn(0x7fa4c8066f80 msgr2=0x7fa4c8067360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:42.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.316+0000 7fa4c7e00640 1 -- 172.21.15.102:0/3522524515 shutdown_connections 2024-03-18T16:37:42.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.316+0000 7fa4c7e00640 1 -- 172.21.15.102:0/3522524515 wait complete. 2024-03-18T16:37:42.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.317+0000 7fa4c7e00640 1 Processor -- start 2024-03-18T16:37:42.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.318+0000 7fa4c7e00640 1 -- start start 2024-03-18T16:37:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.318+0000 7fa4c7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4c80c3890 0x7fa4c8148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.318+0000 7fa4c7e00640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c8154520 0x7fa4c81f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.318+0000 7fa4c7e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4c815e1e0 0x7fa4c81f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.318+0000 7fa4c7e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c815a9f0 con 0x7fa4c80c3890 2024-03-18T16:37:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.318+0000 7fa4c7e00640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c815a710 con 0x7fa4c815e1e0 2024-03-18T16:37:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.318+0000 7fa4c7e00640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c80ca3f0 con 0x7fa4c8154520 2024-03-18T16:37:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.319+0000 7fa4c7400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4c815e1e0 0x7fa4c81f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.319+0000 7fa4c6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4c80c3890 0x7fa4c8148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.319+0000 7fa4c6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c8154520 0x7fa4c81f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.320+0000 7fa4c6000640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c8154520 0x7fa4c81f3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:40764/0 (socket says 172.21.15.102:40764) 2024-03-18T16:37:42.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.320+0000 7fa4c6000640 1 -- 172.21.15.102:0/1637237937 learned_addr learned my addr 172.21.15.102:0/1637237937 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:37:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.323+0000 7fa4c6000640 1 -- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4c815e1e0 msgr2=0x7fa4c81f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.323+0000 7fa4c6000640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4c815e1e0 0x7fa4c81f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.323+0000 7fa4c6000640 1 -- 172.21.15.102:0/1637237937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4c80c3890 msgr2=0x7fa4c8148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.323+0000 7fa4c6000640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4c80c3890 0x7fa4c8148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.323+0000 7fa4c6000640 1 -- 172.21.15.102:0/1637237937 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c80ce6c0 con 0x7fa4c8154520 2024-03-18T16:37:42.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.326+0000 7fa4c6000640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c8154520 0x7fa4c81f3080 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009fd0 tx=0x7fa4b400ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:42.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.329+0000 7fa4bea00640 1 -- 172.21.15.102:0/1637237937 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4017070 con 0x7fa4c8154520 2024-03-18T16:37:42.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.329+0000 7fa4bea00640 1 -- 172.21.15.102:0/1637237937 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4b400c940 con 0x7fa4c8154520 2024-03-18T16:37:42.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.329+0000 7fa4bea00640 1 -- 172.21.15.102:0/1637237937 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b400cd10 con 0x7fa4c8154520 2024-03-18T16:37:42.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.329+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c80cde80 con 0x7fa4c8154520 2024-03-18T16:37:42.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.330+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c80ce3e0 con 0x7fa4c8154520 2024-03-18T16:37:42.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.331+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c8074fd0 con 0x7fa4c8154520 2024-03-18T16:37:42.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.351+0000 7fa4bea00640 1 -- 172.21.15.102:0/1637237937 <== mon.2 v2:172.21.15.154:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa4b401a070 con 0x7fa4c8154520 2024-03-18T16:37:42.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.352+0000 7fa4bea00640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa4a806bc00 0x7fa4a806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:42.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.353+0000 7fa4bea00640 1 -- 172.21.15.102:0/1637237937 <== mon.2 v2:172.21.15.154:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7fa4b4057680 con 0x7fa4c8154520 2024-03-18T16:37:42.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.353+0000 7fa4c6a00640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa4a806bc00 0x7fa4a806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:42.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.353+0000 7fa4c6a00640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa4a806bc00 0x7fa4a806e0c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009a20 tx=0x7fa4b8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:42.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.601+0000 7fa4bea00640 1 -- 172.21.15.102:0/1637237937 <== mon.2 v2:172.21.15.154:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa4b40595e0 con 0x7fa4c8154520 2024-03-18T16:37:42.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.816+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 --> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa4c80c8080 con 0x7fa4a806bc00 2024-03-18T16:37:42.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.825+0000 7fa4bea00640 1 -- 172.21.15.102:0/1637237937 <== mgr.4103 v2:172.21.15.154:6800/2819855665 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332843 (secure 0 0 0) 0x7fa4c80c8080 con 0x7fa4a806bc00 2024-03-18T16:37:42.831 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-18T16:37:42.833 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-18T16:37:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa4a806bc00 msgr2=0x7fa4a806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa4a806bc00 0x7fa4a806e0c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8009a20 tx=0x7fa4b8009190 comp rx=0 tx=0).stop 2024-03-18T16:37:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c8154520 msgr2=0x7fa4c81f3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c8154520 0x7fa4c81f3080 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4009fd0 tx=0x7fa4b400ed40 comp rx=0 tx=0).stop 2024-03-18T16:37:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 shutdown_connections 2024-03-18T16:37:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7fa4a806bc00 0x7fa4a806e0c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4c815e1e0 0x7fa4c81f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7fa4c8154520 0x7fa4c81f3080 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 --2- 172.21.15.102:0/1637237937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4c80c3890 0x7fa4c8148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 >> 172.21.15.102:0/1637237937 conn(0x7fa4c8066f80 msgr2=0x7fa4c81585d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 shutdown_connections 2024-03-18T16:37:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:42.830+0000 7fa4c7e00640 1 -- 172.21.15.102:0/1637237937 wait complete. 2024-03-18T16:37:42.857 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-03-18T16:37:41.047304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600632,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":368,"num_write_kb":2483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223168,"kb_used_data":2880,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751552,"statfs":{"total":773094113280,"available":772865589248,"internally_reserved":0,"allocated":2949120,"data_stored":2294136,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003221"},"pg_stats":[{"pgid":"2.1f","version":"30'19","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:55.744172+0000","last_change":"2024-03-18T16:36:55.744239+0000","last_active":"2024-03-18T16:36:55.744172+0000","last_peered":"2024-03-18T16:36:55.744172+0000","last_clean":"2024-03-18T16:36:55.744172+0000","last_became_active":"2024-03-18T16:21:21.401451+0000","last_became_peered":"2024-03-18T16:21:21.401451+0000","last_unstale":"2024-03-18T16:36:55.744172+0000","last_undegraded":"2024-03-18T16:36:55.744172+0000","last_fullsized":"2024-03-18T16:36:55.744172+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-03-18T16:36:55.744050+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-03-18T16:36:55.744050+0000","last_clean_scrub_stamp":"2024-03-18T16:36:55.744050+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:11.820011+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.261782+0000","last_change":"2024-03-18T16:36:58.261782+0000","last_active":"2024-03-18T16:36:58.261782+0000","last_peered":"2024-03-18T16:36:58.261782+0000","last_clean":"2024-03-18T16:36:58.261782+0000","last_became_active":"2024-03-18T16:21:23.044874+0000","last_became_peered":"2024-03-18T16:21:23.044874+0000","last_unstale":"2024-03-18T16:36:58.261782+0000","last_undegraded":"2024-03-18T16:36:58.261782+0000","last_fullsized":"2024-03-18T16:36:58.261782+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:58.261640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:58.261640+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.261640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:12:49.044786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:14.054386+0000","last_change":"2024-03-18T16:37:14.054425+0000","last_active":"2024-03-18T16:37:14.054386+0000","last_peered":"2024-03-18T16:37:14.054386+0000","last_clean":"2024-03-18T16:37:14.054386+0000","last_became_active":"2024-03-18T16:21:29.145842+0000","last_became_peered":"2024-03-18T16:21:29.145842+0000","last_unstale":"2024-03-18T16:37:14.054386+0000","last_undegraded":"2024-03-18T16:37:14.054386+0000","last_fullsized":"2024-03-18T16:37:14.054386+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:14.054277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:14.054277+0000","last_clean_scrub_stamp":"2024-03-18T16:37:14.054277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:21:34.384174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"29'2","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.237356+0000","last_change":"2024-03-18T16:37:03.237356+0000","last_active":"2024-03-18T16:37:03.237356+0000","last_peered":"2024-03-18T16:37:03.237356+0000","last_clean":"2024-03-18T16:37:03.237356+0000","last_became_active":"2024-03-18T16:21:21.412471+0000","last_became_peered":"2024-03-18T16:21:21.412471+0000","last_unstale":"2024-03-18T16:37:03.237356+0000","last_undegraded":"2024-03-18T16:37:03.237356+0000","last_fullsized":"2024-03-18T16:37:03.237356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'2","last_scrub_stamp":"2024-03-18T16:37:03.237215+0000","last_deep_scrub":"29'2","last_deep_scrub_stamp":"2024-03-18T16:37:03.237215+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.237215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:39.476865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.823610+0000","last_change":"2024-03-18T16:36:57.823648+0000","last_active":"2024-03-18T16:36:57.823610+0000","last_peered":"2024-03-18T16:36:57.823610+0000","last_clean":"2024-03-18T16:36:57.823610+0000","last_became_active":"2024-03-18T16:21:23.049372+0000","last_became_peered":"2024-03-18T16:21:23.049372+0000","last_unstale":"2024-03-18T16:36:57.823610+0000","last_undegraded":"2024-03-18T16:36:57.823610+0000","last_fullsized":"2024-03-18T16:36:57.823610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:57.823501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:57.823501+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.823501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:31.842624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.235049+0000","last_change":"2024-03-18T16:37:08.235087+0000","last_active":"2024-03-18T16:37:08.235049+0000","last_peered":"2024-03-18T16:37:08.235049+0000","last_clean":"2024-03-18T16:37:08.235049+0000","last_became_active":"2024-03-18T16:21:29.065392+0000","last_became_peered":"2024-03-18T16:21:29.065392+0000","last_unstale":"2024-03-18T16:37:08.235049+0000","last_undegraded":"2024-03-18T16:37:08.235049+0000","last_fullsized":"2024-03-18T16:37:08.235049+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.234930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.234930+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.234930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:53.212320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"30'19","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.331077+0000","last_change":"2024-03-18T16:37:05.331141+0000","last_active":"2024-03-18T16:37:05.331077+0000","last_peered":"2024-03-18T16:37:05.331077+0000","last_clean":"2024-03-18T16:37:05.331077+0000","last_became_active":"2024-03-18T16:21:21.397492+0000","last_became_peered":"2024-03-18T16:21:21.397492+0000","last_unstale":"2024-03-18T16:37:05.331077+0000","last_undegraded":"2024-03-18T16:37:05.331077+0000","last_fullsized":"2024-03-18T16:37:05.331077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'19","last_scrub_stamp":"2024-03-18T16:37:05.330962+0000","last_deep_scrub":"30'19","last_deep_scrub_stamp":"2024-03-18T16:37:05.330962+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.330962+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:47:00.652262+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.649403+0000","last_change":"2024-03-18T16:37:04.649434+0000","last_active":"2024-03-18T16:37:04.649403+0000","last_peered":"2024-03-18T16:37:04.649403+0000","last_clean":"2024-03-18T16:37:04.649403+0000","last_became_active":"2024-03-18T16:21:22.946054+0000","last_became_peered":"2024-03-18T16:21:22.946054+0000","last_unstale":"2024-03-18T16:37:04.649403+0000","last_undegraded":"2024-03-18T16:37:04.649403+0000","last_fullsized":"2024-03-18T16:37:04.649403+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.649288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.649288+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.649288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:06:47.727287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.263410+0000","last_change":"2024-03-18T16:37:09.263453+0000","last_active":"2024-03-18T16:37:09.263410+0000","last_peered":"2024-03-18T16:37:09.263410+0000","last_clean":"2024-03-18T16:37:09.263410+0000","last_became_active":"2024-03-18T16:21:29.065427+0000","last_became_peered":"2024-03-18T16:21:29.065427+0000","last_unstale":"2024-03-18T16:37:09.263410+0000","last_undegraded":"2024-03-18T16:37:09.263410+0000","last_fullsized":"2024-03-18T16:37:09.263410+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.263275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.263275+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.263275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:36.374129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.287032+0000","last_change":"2024-03-18T16:37:04.287084+0000","last_active":"2024-03-18T16:37:04.287032+0000","last_peered":"2024-03-18T16:37:04.287032+0000","last_clean":"2024-03-18T16:37:04.287032+0000","last_became_active":"2024-03-18T16:21:21.395914+0000","last_became_peered":"2024-03-18T16:21:21.395914+0000","last_unstale":"2024-03-18T16:37:04.287032+0000","last_undegraded":"2024-03-18T16:37:04.287032+0000","last_fullsized":"2024-03-18T16:37:04.287032+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:37:04.286912+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:37:04.286912+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.286912+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:32.205872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.669736+0000","last_change":"2024-03-18T16:37:05.669766+0000","last_active":"2024-03-18T16:37:05.669736+0000","last_peered":"2024-03-18T16:37:05.669736+0000","last_clean":"2024-03-18T16:37:05.669736+0000","last_became_active":"2024-03-18T16:21:22.946138+0000","last_became_peered":"2024-03-18T16:21:22.946138+0000","last_unstale":"2024-03-18T16:37:05.669736+0000","last_undegraded":"2024-03-18T16:37:05.669736+0000","last_fullsized":"2024-03-18T16:37:05.669736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.669668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.669668+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.669668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:46.003848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.297217+0000","last_change":"2024-03-18T16:37:10.297240+0000","last_active":"2024-03-18T16:37:10.297217+0000","last_peered":"2024-03-18T16:37:10.297217+0000","last_clean":"2024-03-18T16:37:10.297217+0000","last_became_active":"2024-03-18T16:21:29.073189+0000","last_became_peered":"2024-03-18T16:21:29.073189+0000","last_unstale":"2024-03-18T16:37:10.297217+0000","last_undegraded":"2024-03-18T16:37:10.297217+0000","last_fullsized":"2024-03-18T16:37:10.297217+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.297151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.297151+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.297151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:46.536801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.629463+0000","last_change":"2024-03-18T16:37:05.629484+0000","last_active":"2024-03-18T16:37:05.629463+0000","last_peered":"2024-03-18T16:37:05.629463+0000","last_clean":"2024-03-18T16:37:05.629463+0000","last_became_active":"2024-03-18T16:21:23.041658+0000","last_became_peered":"2024-03-18T16:21:23.041658+0000","last_unstale":"2024-03-18T16:37:05.629463+0000","last_undegraded":"2024-03-18T16:37:05.629463+0000","last_fullsized":"2024-03-18T16:37:05.629463+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.629391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.629391+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.629391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:17:09.571784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"30'6","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.287493+0000","last_change":"2024-03-18T16:37:03.287493+0000","last_active":"2024-03-18T16:37:03.287493+0000","last_peered":"2024-03-18T16:37:03.287493+0000","last_clean":"2024-03-18T16:37:03.287493+0000","last_became_active":"2024-03-18T16:21:21.394559+0000","last_became_peered":"2024-03-18T16:21:21.394559+0000","last_unstale":"2024-03-18T16:37:03.287493+0000","last_undegraded":"2024-03-18T16:37:03.287493+0000","last_fullsized":"2024-03-18T16:37:03.287493+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-03-18T16:37:03.287337+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-03-18T16:37:03.287337+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.287337+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:56:24.467281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.339313+0000","last_change":"2024-03-18T16:37:11.339313+0000","last_active":"2024-03-18T16:37:11.339313+0000","last_peered":"2024-03-18T16:37:11.339313+0000","last_clean":"2024-03-18T16:37:11.339313+0000","last_became_active":"2024-03-18T16:21:29.073319+0000","last_became_peered":"2024-03-18T16:21:29.073319+0000","last_unstale":"2024-03-18T16:37:11.339313+0000","last_undegraded":"2024-03-18T16:37:11.339313+0000","last_fullsized":"2024-03-18T16:37:11.339313+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.339212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.339212+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.339212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:08.758806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.125587+0000","last_change":"2024-03-18T16:37:09.125622+0000","last_active":"2024-03-18T16:37:09.125587+0000","last_peered":"2024-03-18T16:37:09.125587+0000","last_clean":"2024-03-18T16:37:09.125587+0000","last_became_active":"2024-03-18T16:21:23.059706+0000","last_became_peered":"2024-03-18T16:21:23.059706+0000","last_unstale":"2024-03-18T16:37:09.125587+0000","last_undegraded":"2024-03-18T16:37:09.125587+0000","last_fullsized":"2024-03-18T16:37:09.125587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.125479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.125479+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.125479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:23:01.464282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"30'6","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.506773+0000","last_change":"2024-03-18T16:37:01.506831+0000","last_active":"2024-03-18T16:37:01.506773+0000","last_peered":"2024-03-18T16:37:01.506773+0000","last_clean":"2024-03-18T16:37:01.506773+0000","last_became_active":"2024-03-18T16:21:21.412147+0000","last_became_peered":"2024-03-18T16:21:21.412147+0000","last_unstale":"2024-03-18T16:37:01.506773+0000","last_undegraded":"2024-03-18T16:37:01.506773+0000","last_fullsized":"2024-03-18T16:37:01.506773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-03-18T16:37:01.506679+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-03-18T16:37:01.506679+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.506679+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:15.971938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.329573+0000","last_change":"2024-03-18T16:37:04.329573+0000","last_active":"2024-03-18T16:37:04.329573+0000","last_peered":"2024-03-18T16:37:04.329573+0000","last_clean":"2024-03-18T16:37:04.329573+0000","last_became_active":"2024-03-18T16:21:29.063682+0000","last_became_peered":"2024-03-18T16:21:29.063682+0000","last_unstale":"2024-03-18T16:37:04.329573+0000","last_undegraded":"2024-03-18T16:37:04.329573+0000","last_fullsized":"2024-03-18T16:37:04.329573+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.329478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.329478+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.329478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:05.494778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.055913+0000","last_change":"2024-03-18T16:37:03.055913+0000","last_active":"2024-03-18T16:37:03.055913+0000","last_peered":"2024-03-18T16:37:03.055913+0000","last_clean":"2024-03-18T16:37:03.055913+0000","last_became_active":"2024-03-18T16:21:54.836383+0000","last_became_peered":"2024-03-18T16:21:54.836383+0000","last_unstale":"2024-03-18T16:37:03.055913+0000","last_undegraded":"2024-03-18T16:37:03.055913+0000","last_fullsized":"2024-03-18T16:37:03.055913+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:03.055762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:03.055762+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.055762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:57.364301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"30'11","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.464940+0000","last_change":"2024-03-18T16:37:00.464989+0000","last_active":"2024-03-18T16:37:00.464940+0000","last_peered":"2024-03-18T16:37:00.464940+0000","last_clean":"2024-03-18T16:37:00.464940+0000","last_became_active":"2024-03-18T16:21:21.405466+0000","last_became_peered":"2024-03-18T16:21:21.405466+0000","last_unstale":"2024-03-18T16:37:00.464940+0000","last_undegraded":"2024-03-18T16:37:00.464940+0000","last_fullsized":"2024-03-18T16:37:00.464940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-03-18T16:37:00.464840+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-03-18T16:37:00.464840+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.464840+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:55.288952+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":30,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.060998+0000","last_change":"2024-03-18T16:37:06.060998+0000","last_active":"2024-03-18T16:37:06.060998+0000","last_peered":"2024-03-18T16:37:06.060998+0000","last_clean":"2024-03-18T16:37:06.060998+0000","last_became_active":"2024-03-18T16:21:55.268050+0000","last_became_peered":"2024-03-18T16:21:55.268050+0000","last_unstale":"2024-03-18T16:37:06.060998+0000","last_undegraded":"2024-03-18T16:37:06.060998+0000","last_fullsized":"2024-03-18T16:37:06.060998+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.060854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.060854+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.060854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:40:20.223068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.375469+0000","last_change":"2024-03-18T16:36:59.375515+0000","last_active":"2024-03-18T16:36:59.375469+0000","last_peered":"2024-03-18T16:36:59.375469+0000","last_clean":"2024-03-18T16:36:59.375469+0000","last_became_active":"2024-03-18T16:21:22.840313+0000","last_became_peered":"2024-03-18T16:21:22.840313+0000","last_unstale":"2024-03-18T16:36:59.375469+0000","last_undegraded":"2024-03-18T16:36:59.375469+0000","last_fullsized":"2024-03-18T16:36:59.375469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:59.375329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:59.375329+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.375329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:38:15.451290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"30'3","reported_seq":51,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.357087+0000","last_change":"2024-03-18T16:37:07.357138+0000","last_active":"2024-03-18T16:37:07.357087+0000","last_peered":"2024-03-18T16:37:07.357087+0000","last_clean":"2024-03-18T16:37:07.357087+0000","last_became_active":"2024-03-18T16:21:21.400100+0000","last_became_peered":"2024-03-18T16:21:21.400100+0000","last_unstale":"2024-03-18T16:37:07.357087+0000","last_undegraded":"2024-03-18T16:37:07.357087+0000","last_fullsized":"2024-03-18T16:37:07.357087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'3","last_scrub_stamp":"2024-03-18T16:37:07.356980+0000","last_deep_scrub":"30'3","last_deep_scrub_stamp":"2024-03-18T16:37:07.356980+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.356980+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:02.181091+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.292253+0000","last_change":"2024-03-18T16:37:05.292253+0000","last_active":"2024-03-18T16:37:05.292253+0000","last_peered":"2024-03-18T16:37:05.292253+0000","last_clean":"2024-03-18T16:37:05.292253+0000","last_became_active":"2024-03-18T16:21:29.064256+0000","last_became_peered":"2024-03-18T16:21:29.064256+0000","last_unstale":"2024-03-18T16:37:05.292253+0000","last_undegraded":"2024-03-18T16:37:05.292253+0000","last_fullsized":"2024-03-18T16:37:05.292253+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.292101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.292101+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.292101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:09.403145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.778425+0000","last_change":"2024-03-18T16:37:00.778458+0000","last_active":"2024-03-18T16:37:00.778425+0000","last_peered":"2024-03-18T16:37:00.778425+0000","last_clean":"2024-03-18T16:37:00.778425+0000","last_became_active":"2024-03-18T16:21:23.054493+0000","last_became_peered":"2024-03-18T16:21:23.054493+0000","last_unstale":"2024-03-18T16:37:00.778425+0000","last_undegraded":"2024-03-18T16:37:00.778425+0000","last_fullsized":"2024-03-18T16:37:00.778425+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:00.778351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:00.778351+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.778351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:01.943650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"30'11","reported_seq":60,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.669353+0000","last_change":"2024-03-18T16:37:01.669407+0000","last_active":"2024-03-18T16:37:01.669353+0000","last_peered":"2024-03-18T16:37:01.669353+0000","last_clean":"2024-03-18T16:37:01.669353+0000","last_became_active":"2024-03-18T16:21:21.400592+0000","last_became_peered":"2024-03-18T16:21:21.400592+0000","last_unstale":"2024-03-18T16:37:01.669353+0000","last_undegraded":"2024-03-18T16:37:01.669353+0000","last_fullsized":"2024-03-18T16:37:01.669353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-03-18T16:37:01.669280+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-03-18T16:37:01.669280+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.669280+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:53.707032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.338416+0000","last_change":"2024-03-18T16:37:12.338455+0000","last_active":"2024-03-18T16:37:12.338416+0000","last_peered":"2024-03-18T16:37:12.338416+0000","last_clean":"2024-03-18T16:37:12.338416+0000","last_became_active":"2024-03-18T16:21:29.140549+0000","last_became_peered":"2024-03-18T16:21:29.140549+0000","last_unstale":"2024-03-18T16:37:12.338416+0000","last_undegraded":"2024-03-18T16:37:12.338416+0000","last_fullsized":"2024-03-18T16:37:12.338416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.338340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.338340+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.338340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:54.272465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.272266+0000","last_change":"2024-03-18T16:37:00.272301+0000","last_active":"2024-03-18T16:37:00.272266+0000","last_peered":"2024-03-18T16:37:00.272266+0000","last_clean":"2024-03-18T16:37:00.272266+0000","last_became_active":"2024-03-18T16:21:22.993901+0000","last_became_peered":"2024-03-18T16:21:22.993901+0000","last_unstale":"2024-03-18T16:37:00.272266+0000","last_undegraded":"2024-03-18T16:37:00.272266+0000","last_fullsized":"2024-03-18T16:37:00.272266+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:00.272158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:00.272158+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.272158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:29.140232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"30'11","reported_seq":58,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.472346+0000","last_change":"2024-03-18T16:36:58.472402+0000","last_active":"2024-03-18T16:36:58.472346+0000","last_peered":"2024-03-18T16:36:58.472346+0000","last_clean":"2024-03-18T16:36:58.472346+0000","last_became_active":"2024-03-18T16:21:21.402589+0000","last_became_peered":"2024-03-18T16:21:21.402589+0000","last_unstale":"2024-03-18T16:36:58.472346+0000","last_undegraded":"2024-03-18T16:36:58.472346+0000","last_fullsized":"2024-03-18T16:36:58.472346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-03-18T16:36:58.472229+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-03-18T16:36:58.472229+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.472229+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:53.793529+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:13.079250+0000","last_change":"2024-03-18T16:37:13.079250+0000","last_active":"2024-03-18T16:37:13.079250+0000","last_peered":"2024-03-18T16:37:13.079250+0000","last_clean":"2024-03-18T16:37:13.079250+0000","last_became_active":"2024-03-18T16:21:29.144727+0000","last_became_peered":"2024-03-18T16:21:29.144727+0000","last_unstale":"2024-03-18T16:37:13.079250+0000","last_undegraded":"2024-03-18T16:37:13.079250+0000","last_fullsized":"2024-03-18T16:37:13.079250+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:13.079107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:13.079107+0000","last_clean_scrub_stamp":"2024-03-18T16:37:13.079107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:31:18.445982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.513877+0000","last_change":"2024-03-18T16:37:03.513913+0000","last_active":"2024-03-18T16:37:03.513877+0000","last_peered":"2024-03-18T16:37:03.513877+0000","last_clean":"2024-03-18T16:37:03.513877+0000","last_became_active":"2024-03-18T16:21:22.974108+0000","last_became_peered":"2024-03-18T16:21:22.974108+0000","last_unstale":"2024-03-18T16:37:03.513877+0000","last_undegraded":"2024-03-18T16:37:03.513877+0000","last_fullsized":"2024-03-18T16:37:03.513877+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:03.513739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:03.513739+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.513739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:07.624782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"30'7","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.368680+0000","last_change":"2024-03-18T16:37:06.368707+0000","last_active":"2024-03-18T16:37:06.368680+0000","last_peered":"2024-03-18T16:37:06.368680+0000","last_clean":"2024-03-18T16:37:06.368680+0000","last_became_active":"2024-03-18T16:21:21.399447+0000","last_became_peered":"2024-03-18T16:21:21.399447+0000","last_unstale":"2024-03-18T16:37:06.368680+0000","last_undegraded":"2024-03-18T16:37:06.368680+0000","last_fullsized":"2024-03-18T16:37:06.368680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-03-18T16:37:06.368613+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-03-18T16:37:06.368613+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.368613+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:40.387736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.661783+0000","last_change":"2024-03-18T16:37:07.661835+0000","last_active":"2024-03-18T16:37:07.661783+0000","last_peered":"2024-03-18T16:37:07.661783+0000","last_clean":"2024-03-18T16:37:07.661783+0000","last_became_active":"2024-03-18T16:21:29.141042+0000","last_became_peered":"2024-03-18T16:21:29.141042+0000","last_unstale":"2024-03-18T16:37:07.661783+0000","last_undegraded":"2024-03-18T16:37:07.661783+0000","last_fullsized":"2024-03-18T16:37:07.661783+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.661639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.661639+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.661639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:05.081233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.691194+0000","last_change":"2024-03-18T16:36:58.691214+0000","last_active":"2024-03-18T16:36:58.691194+0000","last_peered":"2024-03-18T16:36:58.691194+0000","last_clean":"2024-03-18T16:36:58.691194+0000","last_became_active":"2024-03-18T16:21:22.947707+0000","last_became_peered":"2024-03-18T16:21:22.947707+0000","last_unstale":"2024-03-18T16:36:58.691194+0000","last_undegraded":"2024-03-18T16:36:58.691194+0000","last_fullsized":"2024-03-18T16:36:58.691194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:58.691130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:58.691130+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.691130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:26.607438+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"30'2","reported_seq":49,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:56.393346+0000","last_change":"2024-03-18T16:36:56.393387+0000","last_active":"2024-03-18T16:36:56.393346+0000","last_peered":"2024-03-18T16:36:56.393346+0000","last_clean":"2024-03-18T16:36:56.393346+0000","last_became_active":"2024-03-18T16:21:21.399763+0000","last_became_peered":"2024-03-18T16:21:21.399763+0000","last_unstale":"2024-03-18T16:36:56.393346+0000","last_undegraded":"2024-03-18T16:36:56.393346+0000","last_fullsized":"2024-03-18T16:36:56.393346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'2","last_scrub_stamp":"2024-03-18T16:36:56.393245+0000","last_deep_scrub":"30'2","last_deep_scrub_stamp":"2024-03-18T16:36:56.393245+0000","last_clean_scrub_stamp":"2024-03-18T16:36:56.393245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:45:10.756772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.544297+0000","last_change":"2024-03-18T16:37:08.544335+0000","last_active":"2024-03-18T16:37:08.544297+0000","last_peered":"2024-03-18T16:37:08.544297+0000","last_clean":"2024-03-18T16:37:08.544297+0000","last_became_active":"2024-03-18T16:21:29.073214+0000","last_became_peered":"2024-03-18T16:21:29.073214+0000","last_unstale":"2024-03-18T16:37:08.544297+0000","last_undegraded":"2024-03-18T16:37:08.544297+0000","last_fullsized":"2024-03-18T16:37:08.544297+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.544193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.544193+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.544193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:28:44.783842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.331718+0000","last_change":"2024-03-18T16:37:03.331737+0000","last_active":"2024-03-18T16:37:03.331718+0000","last_peered":"2024-03-18T16:37:03.331718+0000","last_clean":"2024-03-18T16:37:03.331718+0000","last_became_active":"2024-03-18T16:21:23.055584+0000","last_became_peered":"2024-03-18T16:21:23.055584+0000","last_unstale":"2024-03-18T16:37:03.331718+0000","last_undegraded":"2024-03-18T16:37:03.331718+0000","last_fullsized":"2024-03-18T16:37:03.331718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:03.331651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:03.331651+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.331651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:17.796909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"30'7","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.655618+0000","last_change":"2024-03-18T16:37:02.655663+0000","last_active":"2024-03-18T16:37:02.655618+0000","last_peered":"2024-03-18T16:37:02.655618+0000","last_clean":"2024-03-18T16:37:02.655618+0000","last_became_active":"2024-03-18T16:21:21.398879+0000","last_became_peered":"2024-03-18T16:21:21.398879+0000","last_unstale":"2024-03-18T16:37:02.655618+0000","last_undegraded":"2024-03-18T16:37:02.655618+0000","last_fullsized":"2024-03-18T16:37:02.655618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-03-18T16:37:02.655510+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-03-18T16:37:02.655510+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.655510+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:01:05.852420+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.555065+0000","last_change":"2024-03-18T16:37:07.555101+0000","last_active":"2024-03-18T16:37:07.555065+0000","last_peered":"2024-03-18T16:37:07.555065+0000","last_clean":"2024-03-18T16:37:07.555065+0000","last_became_active":"2024-03-18T16:21:29.066011+0000","last_became_peered":"2024-03-18T16:21:29.066011+0000","last_unstale":"2024-03-18T16:37:07.555065+0000","last_undegraded":"2024-03-18T16:37:07.555065+0000","last_fullsized":"2024-03-18T16:37:07.555065+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.554963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.554963+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.554963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:34.665816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:55.352845+0000","last_change":"2024-03-18T16:36:55.352845+0000","last_active":"2024-03-18T16:36:55.352845+0000","last_peered":"2024-03-18T16:36:55.352845+0000","last_clean":"2024-03-18T16:36:55.352845+0000","last_became_active":"2024-03-18T16:21:22.975063+0000","last_became_peered":"2024-03-18T16:21:22.975063+0000","last_unstale":"2024-03-18T16:36:55.352845+0000","last_undegraded":"2024-03-18T16:36:55.352845+0000","last_fullsized":"2024-03-18T16:36:55.352845+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:55.352684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:55.352684+0000","last_clean_scrub_stamp":"2024-03-18T16:36:55.352684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:14:01.365564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"30'8","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.105679+0000","last_change":"2024-03-18T16:36:58.105742+0000","last_active":"2024-03-18T16:36:58.105679+0000","last_peered":"2024-03-18T16:36:58.105679+0000","last_clean":"2024-03-18T16:36:58.105679+0000","last_became_active":"2024-03-18T16:21:21.407976+0000","last_became_peered":"2024-03-18T16:21:21.407976+0000","last_unstale":"2024-03-18T16:36:58.105679+0000","last_undegraded":"2024-03-18T16:36:58.105679+0000","last_fullsized":"2024-03-18T16:36:58.105679+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:36:58.105545+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:36:58.105545+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.105545+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:05:38.967022+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.518357+0000","last_change":"2024-03-18T16:37:09.518397+0000","last_active":"2024-03-18T16:37:09.518357+0000","last_peered":"2024-03-18T16:37:09.518357+0000","last_clean":"2024-03-18T16:37:09.518357+0000","last_became_active":"2024-03-18T16:21:29.076414+0000","last_became_peered":"2024-03-18T16:21:29.076414+0000","last_unstale":"2024-03-18T16:37:09.518357+0000","last_undegraded":"2024-03-18T16:37:09.518357+0000","last_fullsized":"2024-03-18T16:37:09.518357+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.518228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.518228+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.518228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:06.937821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.728551+0000","last_change":"2024-03-18T16:36:59.728589+0000","last_active":"2024-03-18T16:36:59.728551+0000","last_peered":"2024-03-18T16:36:59.728551+0000","last_clean":"2024-03-18T16:36:59.728551+0000","last_became_active":"2024-03-18T16:21:23.054155+0000","last_became_peered":"2024-03-18T16:21:23.054155+0000","last_unstale":"2024-03-18T16:36:59.728551+0000","last_undegraded":"2024-03-18T16:36:59.728551+0000","last_fullsized":"2024-03-18T16:36:59.728551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:59.728414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:59.728414+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.728414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:14.193672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"30'5","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.325383+0000","last_change":"2024-03-18T16:37:08.325383+0000","last_active":"2024-03-18T16:37:08.325383+0000","last_peered":"2024-03-18T16:37:08.325383+0000","last_clean":"2024-03-18T16:37:08.325383+0000","last_became_active":"2024-03-18T16:21:21.404385+0000","last_became_peered":"2024-03-18T16:21:21.404385+0000","last_unstale":"2024-03-18T16:37:08.325383+0000","last_undegraded":"2024-03-18T16:37:08.325383+0000","last_fullsized":"2024-03-18T16:37:08.325383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5","last_scrub_stamp":"2024-03-18T16:37:08.325303+0000","last_deep_scrub":"30'5","last_deep_scrub_stamp":"2024-03-18T16:37:08.325303+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.325303+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:13:22.790561+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.649133+0000","last_change":"2024-03-18T16:37:08.649189+0000","last_active":"2024-03-18T16:37:08.649133+0000","last_peered":"2024-03-18T16:37:08.649133+0000","last_clean":"2024-03-18T16:37:08.649133+0000","last_became_active":"2024-03-18T16:21:29.146263+0000","last_became_peered":"2024-03-18T16:21:29.146263+0000","last_unstale":"2024-03-18T16:37:08.649133+0000","last_undegraded":"2024-03-18T16:37:08.649133+0000","last_fullsized":"2024-03-18T16:37:08.649133+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.648874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.648874+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.648874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:50.583000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.209642+0000","last_change":"2024-03-18T16:37:07.209680+0000","last_active":"2024-03-18T16:37:07.209642+0000","last_peered":"2024-03-18T16:37:07.209642+0000","last_clean":"2024-03-18T16:37:07.209642+0000","last_became_active":"2024-03-18T16:21:22.947039+0000","last_became_peered":"2024-03-18T16:21:22.947039+0000","last_unstale":"2024-03-18T16:37:07.209642+0000","last_undegraded":"2024-03-18T16:37:07.209642+0000","last_fullsized":"2024-03-18T16:37:07.209642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.209541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.209541+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.209541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:16:36.494321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"30'11","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.673403+0000","last_change":"2024-03-18T16:37:00.673458+0000","last_active":"2024-03-18T16:37:00.673403+0000","last_peered":"2024-03-18T16:37:00.673403+0000","last_clean":"2024-03-18T16:37:00.673403+0000","last_became_active":"2024-03-18T16:21:21.381053+0000","last_became_peered":"2024-03-18T16:21:21.381053+0000","last_unstale":"2024-03-18T16:37:00.673403+0000","last_undegraded":"2024-03-18T16:37:00.673403+0000","last_fullsized":"2024-03-18T16:37:00.673403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'11","last_scrub_stamp":"2024-03-18T16:37:00.673295+0000","last_deep_scrub":"30'11","last_deep_scrub_stamp":"2024-03-18T16:37:00.673295+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.673295+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:06.659579+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"30'119","reported_seq":207,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.265455+0000","last_change":"2024-03-18T16:37:02.265455+0000","last_active":"2024-03-18T16:37:02.265455+0000","last_peered":"2024-03-18T16:37:02.265455+0000","last_clean":"2024-03-18T16:37:02.265455+0000","last_became_active":"2024-03-18T16:21:08.661714+0000","last_became_peered":"2024-03-18T16:21:08.661714+0000","last_unstale":"2024-03-18T16:37:02.265455+0000","last_undegraded":"2024-03-18T16:37:02.265455+0000","last_fullsized":"2024-03-18T16:37:02.265455+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'119","last_scrub_stamp":"2024-03-18T16:37:02.265291+0000","last_deep_scrub":"30'119","last_deep_scrub_stamp":"2024-03-18T16:37:02.265291+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.265291+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:20:50.601488+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:13.703509+0000","last_change":"2024-03-18T16:37:13.703530+0000","last_active":"2024-03-18T16:37:13.703509+0000","last_peered":"2024-03-18T16:37:13.703509+0000","last_clean":"2024-03-18T16:37:13.703509+0000","last_became_active":"2024-03-18T16:21:29.145340+0000","last_became_peered":"2024-03-18T16:21:29.145340+0000","last_unstale":"2024-03-18T16:37:13.703509+0000","last_undegraded":"2024-03-18T16:37:13.703509+0000","last_fullsized":"2024-03-18T16:37:13.703509+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:13.703406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:13.703406+0000","last_clean_scrub_stamp":"2024-03-18T16:37:13.703406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:58:12.650999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.325231+0000","last_change":"2024-03-18T16:37:10.325252+0000","last_active":"2024-03-18T16:37:10.325231+0000","last_peered":"2024-03-18T16:37:10.325231+0000","last_clean":"2024-03-18T16:37:10.325231+0000","last_became_active":"2024-03-18T16:21:22.994911+0000","last_became_peered":"2024-03-18T16:21:22.994911+0000","last_unstale":"2024-03-18T16:37:10.325231+0000","last_undegraded":"2024-03-18T16:37:10.325231+0000","last_fullsized":"2024-03-18T16:37:10.325231+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.325165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.325165+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.325165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:14.436209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"30'9","reported_seq":59,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:56.047926+0000","last_change":"2024-03-18T16:36:56.047926+0000","last_active":"2024-03-18T16:36:56.047926+0000","last_peered":"2024-03-18T16:36:56.047926+0000","last_clean":"2024-03-18T16:36:56.047926+0000","last_became_active":"2024-03-18T16:21:21.381809+0000","last_became_peered":"2024-03-18T16:21:21.381809+0000","last_unstale":"2024-03-18T16:36:56.047926+0000","last_undegraded":"2024-03-18T16:36:56.047926+0000","last_fullsized":"2024-03-18T16:36:56.047926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-03-18T16:36:56.047790+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-03-18T16:36:56.047790+0000","last_clean_scrub_stamp":"2024-03-18T16:36:56.047790+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:20:55.302327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.698518+0000","last_change":"2024-03-18T16:37:12.698557+0000","last_active":"2024-03-18T16:37:12.698518+0000","last_peered":"2024-03-18T16:37:12.698518+0000","last_clean":"2024-03-18T16:37:12.698518+0000","last_became_active":"2024-03-18T16:21:29.139064+0000","last_became_peered":"2024-03-18T16:21:29.139064+0000","last_unstale":"2024-03-18T16:37:12.698518+0000","last_undegraded":"2024-03-18T16:37:12.698518+0000","last_fullsized":"2024-03-18T16:37:12.698518+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.698398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.698398+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.698398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:44.293587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.662445+0000","last_change":"2024-03-18T16:37:07.662445+0000","last_active":"2024-03-18T16:37:07.662445+0000","last_peered":"2024-03-18T16:37:07.662445+0000","last_clean":"2024-03-18T16:37:07.662445+0000","last_became_active":"2024-03-18T16:21:22.949359+0000","last_became_peered":"2024-03-18T16:21:22.949359+0000","last_unstale":"2024-03-18T16:37:07.662445+0000","last_undegraded":"2024-03-18T16:37:07.662445+0000","last_fullsized":"2024-03-18T16:37:07.662445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.662326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.662326+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.662326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:50.438541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"30'5","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.427951+0000","last_change":"2024-03-18T16:36:57.427951+0000","last_active":"2024-03-18T16:36:57.427951+0000","last_peered":"2024-03-18T16:36:57.427951+0000","last_clean":"2024-03-18T16:36:57.427951+0000","last_became_active":"2024-03-18T16:21:21.400597+0000","last_became_peered":"2024-03-18T16:21:21.400597+0000","last_unstale":"2024-03-18T16:36:57.427951+0000","last_undegraded":"2024-03-18T16:36:57.427951+0000","last_fullsized":"2024-03-18T16:36:57.427951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5","last_scrub_stamp":"2024-03-18T16:36:57.427798+0000","last_deep_scrub":"30'5","last_deep_scrub_stamp":"2024-03-18T16:36:57.427798+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.427798+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:52:25.346758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.675835+0000","last_change":"2024-03-18T16:37:05.675835+0000","last_active":"2024-03-18T16:37:05.675835+0000","last_peered":"2024-03-18T16:37:05.675835+0000","last_clean":"2024-03-18T16:37:05.675835+0000","last_became_active":"2024-03-18T16:21:29.075019+0000","last_became_peered":"2024-03-18T16:21:29.075019+0000","last_unstale":"2024-03-18T16:37:05.675835+0000","last_undegraded":"2024-03-18T16:37:05.675835+0000","last_fullsized":"2024-03-18T16:37:05.675835+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.675679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.675679+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.675679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:32.786533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:14.227059+0000","last_change":"2024-03-18T16:37:14.227093+0000","last_active":"2024-03-18T16:37:14.227059+0000","last_peered":"2024-03-18T16:37:14.227059+0000","last_clean":"2024-03-18T16:37:14.227059+0000","last_became_active":"2024-03-18T16:21:22.999730+0000","last_became_peered":"2024-03-18T16:21:22.999730+0000","last_unstale":"2024-03-18T16:37:14.227059+0000","last_undegraded":"2024-03-18T16:37:14.227059+0000","last_fullsized":"2024-03-18T16:37:14.227059+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:14.226948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:14.226948+0000","last_clean_scrub_stamp":"2024-03-18T16:37:14.226948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:41.337801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"30'8","reported_seq":56,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.109282+0000","last_change":"2024-03-18T16:36:59.109282+0000","last_active":"2024-03-18T16:36:59.109282+0000","last_peered":"2024-03-18T16:36:59.109282+0000","last_clean":"2024-03-18T16:36:59.109282+0000","last_became_active":"2024-03-18T16:21:21.408986+0000","last_became_peered":"2024-03-18T16:21:21.408986+0000","last_unstale":"2024-03-18T16:36:59.109282+0000","last_undegraded":"2024-03-18T16:36:59.109282+0000","last_fullsized":"2024-03-18T16:36:59.109282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:36:59.109184+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:36:59.109184+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.109184+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:02:56.365385+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:13.545858+0000","last_change":"2024-03-18T16:37:13.545915+0000","last_active":"2024-03-18T16:37:13.545858+0000","last_peered":"2024-03-18T16:37:13.545858+0000","last_clean":"2024-03-18T16:37:13.545858+0000","last_became_active":"2024-03-18T16:21:29.139233+0000","last_became_peered":"2024-03-18T16:21:29.139233+0000","last_unstale":"2024-03-18T16:37:13.545858+0000","last_undegraded":"2024-03-18T16:37:13.545858+0000","last_fullsized":"2024-03-18T16:37:13.545858+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:13.545742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:13.545742+0000","last_clean_scrub_stamp":"2024-03-18T16:37:13.545742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:40:31.908499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:15.202557+0000","last_change":"2024-03-18T16:37:15.202579+0000","last_active":"2024-03-18T16:37:15.202557+0000","last_peered":"2024-03-18T16:37:15.202557+0000","last_clean":"2024-03-18T16:37:15.202557+0000","last_became_active":"2024-03-18T16:21:22.999743+0000","last_became_peered":"2024-03-18T16:21:22.999743+0000","last_unstale":"2024-03-18T16:37:15.202557+0000","last_undegraded":"2024-03-18T16:37:15.202557+0000","last_fullsized":"2024-03-18T16:37:15.202557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:15.202487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:15.202487+0000","last_clean_scrub_stamp":"2024-03-18T16:37:15.202487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:02:30.649558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"30'4","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.336474+0000","last_change":"2024-03-18T16:36:58.336474+0000","last_active":"2024-03-18T16:36:58.336474+0000","last_peered":"2024-03-18T16:36:58.336474+0000","last_clean":"2024-03-18T16:36:58.336474+0000","last_became_active":"2024-03-18T16:21:21.399041+0000","last_became_peered":"2024-03-18T16:21:21.399041+0000","last_unstale":"2024-03-18T16:36:58.336474+0000","last_undegraded":"2024-03-18T16:36:58.336474+0000","last_fullsized":"2024-03-18T16:36:58.336474+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4","last_scrub_stamp":"2024-03-18T16:36:58.336388+0000","last_deep_scrub":"30'4","last_deep_scrub_stamp":"2024-03-18T16:36:58.336388+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.336388+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:14:04.349269+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.268332+0000","last_change":"2024-03-18T16:37:06.268354+0000","last_active":"2024-03-18T16:37:06.268332+0000","last_peered":"2024-03-18T16:37:06.268332+0000","last_clean":"2024-03-18T16:37:06.268332+0000","last_became_active":"2024-03-18T16:21:29.064891+0000","last_became_peered":"2024-03-18T16:21:29.064891+0000","last_unstale":"2024-03-18T16:37:06.268332+0000","last_undegraded":"2024-03-18T16:37:06.268332+0000","last_fullsized":"2024-03-18T16:37:06.268332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.268257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.268257+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.268257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:08.835557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.314526+0000","last_change":"2024-03-18T16:37:12.314561+0000","last_active":"2024-03-18T16:37:12.314526+0000","last_peered":"2024-03-18T16:37:12.314526+0000","last_clean":"2024-03-18T16:37:12.314526+0000","last_became_active":"2024-03-18T16:21:22.995811+0000","last_became_peered":"2024-03-18T16:21:22.995811+0000","last_unstale":"2024-03-18T16:37:12.314526+0000","last_undegraded":"2024-03-18T16:37:12.314526+0000","last_fullsized":"2024-03-18T16:37:12.314526+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.314443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.314443+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.314443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:41.182517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"30'26","reported_seq":74,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.064626+0000","last_change":"2024-03-18T16:36:57.064626+0000","last_active":"2024-03-18T16:36:57.064626+0000","last_peered":"2024-03-18T16:36:57.064626+0000","last_clean":"2024-03-18T16:36:57.064626+0000","last_became_active":"2024-03-18T16:21:21.403131+0000","last_became_peered":"2024-03-18T16:21:21.403131+0000","last_unstale":"2024-03-18T16:36:57.064626+0000","last_undegraded":"2024-03-18T16:36:57.064626+0000","last_fullsized":"2024-03-18T16:36:57.064626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'26","last_scrub_stamp":"2024-03-18T16:36:57.064452+0000","last_deep_scrub":"30'26","last_deep_scrub_stamp":"2024-03-18T16:36:57.064452+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.064452+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:50.653357+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.669177+0000","last_change":"2024-03-18T16:37:06.669217+0000","last_active":"2024-03-18T16:37:06.669177+0000","last_peered":"2024-03-18T16:37:06.669177+0000","last_clean":"2024-03-18T16:37:06.669177+0000","last_became_active":"2024-03-18T16:21:29.078500+0000","last_became_peered":"2024-03-18T16:21:29.078500+0000","last_unstale":"2024-03-18T16:37:06.669177+0000","last_undegraded":"2024-03-18T16:37:06.669177+0000","last_fullsized":"2024-03-18T16:37:06.669177+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.669079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.669079+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.669079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:02:22.116151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.717889+0000","last_change":"2024-03-18T16:37:09.717908+0000","last_active":"2024-03-18T16:37:09.717889+0000","last_peered":"2024-03-18T16:37:09.717889+0000","last_clean":"2024-03-18T16:37:09.717889+0000","last_became_active":"2024-03-18T16:21:23.056533+0000","last_became_peered":"2024-03-18T16:21:23.056533+0000","last_unstale":"2024-03-18T16:37:09.717889+0000","last_undegraded":"2024-03-18T16:37:09.717889+0000","last_fullsized":"2024-03-18T16:37:09.717889+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.717824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.717824+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.717824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:32:50.052004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.695122+0000","last_change":"2024-03-18T16:37:02.695165+0000","last_active":"2024-03-18T16:37:02.695122+0000","last_peered":"2024-03-18T16:37:02.695122+0000","last_clean":"2024-03-18T16:37:02.695122+0000","last_became_active":"2024-03-18T16:21:21.404125+0000","last_became_peered":"2024-03-18T16:21:21.404125+0000","last_unstale":"2024-03-18T16:37:02.695122+0000","last_undegraded":"2024-03-18T16:37:02.695122+0000","last_fullsized":"2024-03-18T16:37:02.695122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:37:02.695054+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:37:02.695054+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.695054+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:30.611362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.110274+0000","last_change":"2024-03-18T16:37:11.110312+0000","last_active":"2024-03-18T16:37:11.110274+0000","last_peered":"2024-03-18T16:37:11.110274+0000","last_clean":"2024-03-18T16:37:11.110274+0000","last_became_active":"2024-03-18T16:21:29.139387+0000","last_became_peered":"2024-03-18T16:21:29.139387+0000","last_unstale":"2024-03-18T16:37:11.110274+0000","last_undegraded":"2024-03-18T16:37:11.110274+0000","last_fullsized":"2024-03-18T16:37:11.110274+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.110160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.110160+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.110160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:55.397408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.277004+0000","last_change":"2024-03-18T16:37:01.277034+0000","last_active":"2024-03-18T16:37:01.277004+0000","last_peered":"2024-03-18T16:37:01.277004+0000","last_clean":"2024-03-18T16:37:01.277004+0000","last_became_active":"2024-03-18T16:21:22.994728+0000","last_became_peered":"2024-03-18T16:21:22.994728+0000","last_unstale":"2024-03-18T16:37:01.277004+0000","last_undegraded":"2024-03-18T16:37:01.277004+0000","last_fullsized":"2024-03-18T16:37:01.277004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:01.276933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:01.276933+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.276933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:00:29.923268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"30'8","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.610468+0000","last_change":"2024-03-18T16:37:03.610532+0000","last_active":"2024-03-18T16:37:03.610468+0000","last_peered":"2024-03-18T16:37:03.610468+0000","last_clean":"2024-03-18T16:37:03.610468+0000","last_became_active":"2024-03-18T16:21:21.411762+0000","last_became_peered":"2024-03-18T16:21:21.411762+0000","last_unstale":"2024-03-18T16:37:03.610468+0000","last_undegraded":"2024-03-18T16:37:03.610468+0000","last_fullsized":"2024-03-18T16:37:03.610468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'8","last_scrub_stamp":"2024-03-18T16:37:03.610361+0000","last_deep_scrub":"30'8","last_deep_scrub_stamp":"2024-03-18T16:37:03.610361+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.610361+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:05.826883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:16.016267+0000","last_change":"2024-03-18T16:37:16.016308+0000","last_active":"2024-03-18T16:37:16.016267+0000","last_peered":"2024-03-18T16:37:16.016267+0000","last_clean":"2024-03-18T16:37:16.016267+0000","last_became_active":"2024-03-18T16:21:55.268392+0000","last_became_peered":"2024-03-18T16:21:55.268392+0000","last_unstale":"2024-03-18T16:37:16.016267+0000","last_undegraded":"2024-03-18T16:37:16.016267+0000","last_fullsized":"2024-03-18T16:37:16.016267+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:16.016161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:16.016161+0000","last_clean_scrub_stamp":"2024-03-18T16:37:16.016161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:27:02.104378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,2],"acting":[3,5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.793901+0000","last_change":"2024-03-18T16:37:01.793939+0000","last_active":"2024-03-18T16:37:01.793901+0000","last_peered":"2024-03-18T16:37:01.793901+0000","last_clean":"2024-03-18T16:37:01.793901+0000","last_became_active":"2024-03-18T16:21:23.054300+0000","last_became_peered":"2024-03-18T16:21:23.054300+0000","last_unstale":"2024-03-18T16:37:01.793901+0000","last_undegraded":"2024-03-18T16:37:01.793901+0000","last_fullsized":"2024-03-18T16:37:01.793901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:01.793794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:01.793794+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.793794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:19:05.904838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"30'15","reported_seq":63,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:54.791306+0000","last_change":"2024-03-18T16:36:54.791362+0000","last_active":"2024-03-18T16:36:54.791306+0000","last_peered":"2024-03-18T16:36:54.791306+0000","last_clean":"2024-03-18T16:36:54.791306+0000","last_became_active":"2024-03-18T16:21:21.400017+0000","last_became_peered":"2024-03-18T16:21:21.400017+0000","last_unstale":"2024-03-18T16:36:54.791306+0000","last_undegraded":"2024-03-18T16:36:54.791306+0000","last_fullsized":"2024-03-18T16:36:54.791306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'15","last_scrub_stamp":"2024-03-18T16:36:54.791182+0000","last_deep_scrub":"30'15","last_deep_scrub_stamp":"2024-03-18T16:36:54.791182+0000","last_clean_scrub_stamp":"2024-03-18T16:36:54.791182+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:14:00.804062+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:14.500754+0000","last_change":"2024-03-18T16:37:14.500795+0000","last_active":"2024-03-18T16:37:14.500754+0000","last_peered":"2024-03-18T16:37:14.500754+0000","last_clean":"2024-03-18T16:37:14.500754+0000","last_became_active":"2024-03-18T16:21:29.140696+0000","last_became_peered":"2024-03-18T16:21:29.140696+0000","last_unstale":"2024-03-18T16:37:14.500754+0000","last_undegraded":"2024-03-18T16:37:14.500754+0000","last_fullsized":"2024-03-18T16:37:14.500754+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:14.500645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:14.500645+0000","last_clean_scrub_stamp":"2024-03-18T16:37:14.500645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:23:51.586558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:13.266144+0000","last_change":"2024-03-18T16:37:13.266183+0000","last_active":"2024-03-18T16:37:13.266144+0000","last_peered":"2024-03-18T16:37:13.266144+0000","last_clean":"2024-03-18T16:37:13.266144+0000","last_became_active":"2024-03-18T16:21:22.996446+0000","last_became_peered":"2024-03-18T16:21:22.996446+0000","last_unstale":"2024-03-18T16:37:13.266144+0000","last_undegraded":"2024-03-18T16:37:13.266144+0000","last_fullsized":"2024-03-18T16:37:13.266144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:13.266014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:13.266014+0000","last_clean_scrub_stamp":"2024-03-18T16:37:13.266014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:58.243405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"30'20","reported_seq":67,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.667065+0000","last_change":"2024-03-18T16:37:03.667098+0000","last_active":"2024-03-18T16:37:03.667065+0000","last_peered":"2024-03-18T16:37:03.667065+0000","last_clean":"2024-03-18T16:37:03.667065+0000","last_became_active":"2024-03-18T16:21:21.411849+0000","last_became_peered":"2024-03-18T16:21:21.411849+0000","last_unstale":"2024-03-18T16:37:03.667065+0000","last_undegraded":"2024-03-18T16:37:03.667065+0000","last_fullsized":"2024-03-18T16:37:03.667065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'20","last_scrub_stamp":"2024-03-18T16:37:03.666986+0000","last_deep_scrub":"30'20","last_deep_scrub_stamp":"2024-03-18T16:37:03.666986+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.666986+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:40:17.829200+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.500179+0000","last_change":"2024-03-18T16:37:10.500218+0000","last_active":"2024-03-18T16:37:10.500179+0000","last_peered":"2024-03-18T16:37:10.500179+0000","last_clean":"2024-03-18T16:37:10.500179+0000","last_became_active":"2024-03-18T16:21:29.076863+0000","last_became_peered":"2024-03-18T16:21:29.076863+0000","last_unstale":"2024-03-18T16:37:10.500179+0000","last_undegraded":"2024-03-18T16:37:10.500179+0000","last_fullsized":"2024-03-18T16:37:10.500179+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.500072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.500072+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.500072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:52.434197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.314779+0000","last_change":"2024-03-18T16:37:01.314799+0000","last_active":"2024-03-18T16:37:01.314779+0000","last_peered":"2024-03-18T16:37:01.314779+0000","last_clean":"2024-03-18T16:37:01.314779+0000","last_became_active":"2024-03-18T16:21:22.992449+0000","last_became_peered":"2024-03-18T16:21:22.992449+0000","last_unstale":"2024-03-18T16:37:01.314779+0000","last_undegraded":"2024-03-18T16:37:01.314779+0000","last_fullsized":"2024-03-18T16:37:01.314779+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:01.314706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:01.314706+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.314706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:35.333829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"30'9","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.502759+0000","last_change":"2024-03-18T16:36:59.502807+0000","last_active":"2024-03-18T16:36:59.502759+0000","last_peered":"2024-03-18T16:36:59.502759+0000","last_clean":"2024-03-18T16:36:59.502759+0000","last_became_active":"2024-03-18T16:21:21.403022+0000","last_became_peered":"2024-03-18T16:21:21.403022+0000","last_unstale":"2024-03-18T16:36:59.502759+0000","last_undegraded":"2024-03-18T16:36:59.502759+0000","last_fullsized":"2024-03-18T16:36:59.502759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'9","last_scrub_stamp":"2024-03-18T16:36:59.502663+0000","last_deep_scrub":"30'9","last_deep_scrub_stamp":"2024-03-18T16:36:59.502663+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.502663+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:27:33.521786+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.251361+0000","last_change":"2024-03-18T16:37:07.251382+0000","last_active":"2024-03-18T16:37:07.251361+0000","last_peered":"2024-03-18T16:37:07.251361+0000","last_clean":"2024-03-18T16:37:07.251361+0000","last_became_active":"2024-03-18T16:21:29.064829+0000","last_became_peered":"2024-03-18T16:21:29.064829+0000","last_unstale":"2024-03-18T16:37:07.251361+0000","last_undegraded":"2024-03-18T16:37:07.251361+0000","last_fullsized":"2024-03-18T16:37:07.251361+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.251292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.251292+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.251292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:36.119366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:58.774183+0000","last_change":"2024-03-18T16:36:58.774220+0000","last_active":"2024-03-18T16:36:58.774183+0000","last_peered":"2024-03-18T16:36:58.774183+0000","last_clean":"2024-03-18T16:36:58.774183+0000","last_became_active":"2024-03-18T16:21:23.051005+0000","last_became_peered":"2024-03-18T16:21:23.051005+0000","last_unstale":"2024-03-18T16:36:58.774183+0000","last_undegraded":"2024-03-18T16:36:58.774183+0000","last_fullsized":"2024-03-18T16:36:58.774183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:58.774078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:58.774078+0000","last_clean_scrub_stamp":"2024-03-18T16:36:58.774078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:53.598189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"30'6","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.673322+0000","last_change":"2024-03-18T16:37:01.673364+0000","last_active":"2024-03-18T16:37:01.673322+0000","last_peered":"2024-03-18T16:37:01.673322+0000","last_clean":"2024-03-18T16:37:01.673322+0000","last_became_active":"2024-03-18T16:21:21.398034+0000","last_became_peered":"2024-03-18T16:21:21.398034+0000","last_unstale":"2024-03-18T16:37:01.673322+0000","last_undegraded":"2024-03-18T16:37:01.673322+0000","last_fullsized":"2024-03-18T16:37:01.673322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'6","last_scrub_stamp":"2024-03-18T16:37:01.673229+0000","last_deep_scrub":"30'6","last_deep_scrub_stamp":"2024-03-18T16:37:01.673229+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.673229+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:24:48.436630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.134457+0000","last_change":"2024-03-18T16:37:10.134495+0000","last_active":"2024-03-18T16:37:10.134457+0000","last_peered":"2024-03-18T16:37:10.134457+0000","last_clean":"2024-03-18T16:37:10.134457+0000","last_became_active":"2024-03-18T16:21:29.065453+0000","last_became_peered":"2024-03-18T16:21:29.065453+0000","last_unstale":"2024-03-18T16:37:10.134457+0000","last_undegraded":"2024-03-18T16:37:10.134457+0000","last_fullsized":"2024-03-18T16:37:10.134457+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.134351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.134351+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.134351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:38:03.638453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.625577+0000","last_change":"2024-03-18T16:37:09.625610+0000","last_active":"2024-03-18T16:37:09.625577+0000","last_peered":"2024-03-18T16:37:09.625577+0000","last_clean":"2024-03-18T16:37:09.625577+0000","last_became_active":"2024-03-18T16:21:22.993360+0000","last_became_peered":"2024-03-18T16:21:22.993360+0000","last_unstale":"2024-03-18T16:37:09.625577+0000","last_undegraded":"2024-03-18T16:37:09.625577+0000","last_fullsized":"2024-03-18T16:37:09.625577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.625502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.625502+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.625502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:19:33.258443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"30'7","reported_seq":55,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.706119+0000","last_change":"2024-03-18T16:37:00.706144+0000","last_active":"2024-03-18T16:37:00.706119+0000","last_peered":"2024-03-18T16:37:00.706119+0000","last_clean":"2024-03-18T16:37:00.706119+0000","last_became_active":"2024-03-18T16:21:21.399337+0000","last_became_peered":"2024-03-18T16:21:21.399337+0000","last_unstale":"2024-03-18T16:37:00.706119+0000","last_undegraded":"2024-03-18T16:37:00.706119+0000","last_fullsized":"2024-03-18T16:37:00.706119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-03-18T16:37:00.706035+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-03-18T16:37:00.706035+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.706035+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:55.014574+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.089110+0000","last_change":"2024-03-18T16:37:12.089148+0000","last_active":"2024-03-18T16:37:12.089110+0000","last_peered":"2024-03-18T16:37:12.089110+0000","last_clean":"2024-03-18T16:37:12.089110+0000","last_became_active":"2024-03-18T16:21:29.144286+0000","last_became_peered":"2024-03-18T16:21:29.144286+0000","last_unstale":"2024-03-18T16:37:12.089110+0000","last_undegraded":"2024-03-18T16:37:12.089110+0000","last_fullsized":"2024-03-18T16:37:12.089110+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.089006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.089006+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.089006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:06:11.068314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":40,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.607417+0000","last_change":"2024-03-18T16:37:09.607417+0000","last_active":"2024-03-18T16:37:09.607417+0000","last_peered":"2024-03-18T16:37:09.607417+0000","last_clean":"2024-03-18T16:37:09.607417+0000","last_became_active":"2024-03-18T16:21:29.147797+0000","last_became_peered":"2024-03-18T16:21:29.147797+0000","last_unstale":"2024-03-18T16:37:09.607417+0000","last_undegraded":"2024-03-18T16:37:09.607417+0000","last_fullsized":"2024-03-18T16:37:09.607417+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.607267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.607267+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.607267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:06.050614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.509351+0000","last_change":"2024-03-18T16:37:05.509386+0000","last_active":"2024-03-18T16:37:05.509351+0000","last_peered":"2024-03-18T16:37:05.509351+0000","last_clean":"2024-03-18T16:37:05.509351+0000","last_became_active":"2024-03-18T16:21:22.979702+0000","last_became_peered":"2024-03-18T16:21:22.979702+0000","last_unstale":"2024-03-18T16:37:05.509351+0000","last_undegraded":"2024-03-18T16:37:05.509351+0000","last_fullsized":"2024-03-18T16:37:05.509351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.509242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.509242+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.509242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:34.377316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"30'4","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.717844+0000","last_change":"2024-03-18T16:36:59.717869+0000","last_active":"2024-03-18T16:36:59.717844+0000","last_peered":"2024-03-18T16:36:59.717844+0000","last_clean":"2024-03-18T16:36:59.717844+0000","last_became_active":"2024-03-18T16:21:21.329005+0000","last_became_peered":"2024-03-18T16:21:21.329005+0000","last_unstale":"2024-03-18T16:36:59.717844+0000","last_undegraded":"2024-03-18T16:36:59.717844+0000","last_fullsized":"2024-03-18T16:36:59.717844+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'4","last_scrub_stamp":"2024-03-18T16:36:59.717783+0000","last_deep_scrub":"30'4","last_deep_scrub_stamp":"2024-03-18T16:36:59.717783+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.717783+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:47:32.312475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":39,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.721987+0000","last_change":"2024-03-18T16:37:11.722032+0000","last_active":"2024-03-18T16:37:11.721987+0000","last_peered":"2024-03-18T16:37:11.721987+0000","last_clean":"2024-03-18T16:37:11.721987+0000","last_became_active":"2024-03-18T16:21:29.076325+0000","last_became_peered":"2024-03-18T16:21:29.076325+0000","last_unstale":"2024-03-18T16:37:11.721987+0000","last_undegraded":"2024-03-18T16:37:11.721987+0000","last_fullsized":"2024-03-18T16:37:11.721987+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.721872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.721872+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.721872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:48:54.498086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.267909+0000","last_change":"2024-03-18T16:36:59.267943+0000","last_active":"2024-03-18T16:36:59.267909+0000","last_peered":"2024-03-18T16:36:59.267909+0000","last_clean":"2024-03-18T16:36:59.267909+0000","last_became_active":"2024-03-18T16:21:23.052591+0000","last_became_peered":"2024-03-18T16:21:23.052591+0000","last_unstale":"2024-03-18T16:36:59.267909+0000","last_undegraded":"2024-03-18T16:36:59.267909+0000","last_fullsized":"2024-03-18T16:36:59.267909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:59.267795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:59.267795+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.267795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:01.835095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"30'5","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.274927+0000","last_change":"2024-03-18T16:37:02.274969+0000","last_active":"2024-03-18T16:37:02.274927+0000","last_peered":"2024-03-18T16:37:02.274927+0000","last_clean":"2024-03-18T16:37:02.274927+0000","last_became_active":"2024-03-18T16:21:21.412417+0000","last_became_peered":"2024-03-18T16:21:21.412417+0000","last_unstale":"2024-03-18T16:37:02.274927+0000","last_undegraded":"2024-03-18T16:37:02.274927+0000","last_fullsized":"2024-03-18T16:37:02.274927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'5","last_scrub_stamp":"2024-03-18T16:37:02.274829+0000","last_deep_scrub":"30'5","last_deep_scrub_stamp":"2024-03-18T16:37:02.274829+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.274829+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:18:29.385683+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"30'10","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:59.753727+0000","last_change":"2024-03-18T16:36:59.753783+0000","last_active":"2024-03-18T16:36:59.753727+0000","last_peered":"2024-03-18T16:36:59.753727+0000","last_clean":"2024-03-18T16:36:59.753727+0000","last_became_active":"2024-03-18T16:21:21.397100+0000","last_became_peered":"2024-03-18T16:21:21.397100+0000","last_unstale":"2024-03-18T16:36:59.753727+0000","last_undegraded":"2024-03-18T16:36:59.753727+0000","last_fullsized":"2024-03-18T16:36:59.753727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'10","last_scrub_stamp":"2024-03-18T16:36:59.753619+0000","last_deep_scrub":"30'10","last_deep_scrub_stamp":"2024-03-18T16:36:59.753619+0000","last_clean_scrub_stamp":"2024-03-18T16:36:59.753619+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:24.570326+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:03.764583+0000","last_change":"2024-03-18T16:37:03.764618+0000","last_active":"2024-03-18T16:37:03.764583+0000","last_peered":"2024-03-18T16:37:03.764583+0000","last_clean":"2024-03-18T16:37:03.764583+0000","last_became_active":"2024-03-18T16:21:23.058311+0000","last_became_peered":"2024-03-18T16:21:23.058311+0000","last_unstale":"2024-03-18T16:37:03.764583+0000","last_undegraded":"2024-03-18T16:37:03.764583+0000","last_fullsized":"2024-03-18T16:37:03.764583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:03.764480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:03.764480+0000","last_clean_scrub_stamp":"2024-03-18T16:37:03.764480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:32.632554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":38,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.030113+0000","last_change":"2024-03-18T16:37:02.030150+0000","last_active":"2024-03-18T16:37:02.030113+0000","last_peered":"2024-03-18T16:37:02.030113+0000","last_clean":"2024-03-18T16:37:02.030113+0000","last_became_active":"2024-03-18T16:21:29.058413+0000","last_became_peered":"2024-03-18T16:21:29.058413+0000","last_unstale":"2024-03-18T16:37:02.030113+0000","last_undegraded":"2024-03-18T16:37:02.030113+0000","last_fullsized":"2024-03-18T16:37:02.030113+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:02.030002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:02.030002+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.030002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:53:26.846709+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"30'7","reported_seq":54,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.242321+0000","last_change":"2024-03-18T16:37:01.242366+0000","last_active":"2024-03-18T16:37:01.242321+0000","last_peered":"2024-03-18T16:37:01.242321+0000","last_clean":"2024-03-18T16:37:01.242321+0000","last_became_active":"2024-03-18T16:21:21.412212+0000","last_became_peered":"2024-03-18T16:21:21.412212+0000","last_unstale":"2024-03-18T16:37:01.242321+0000","last_undegraded":"2024-03-18T16:37:01.242321+0000","last_fullsized":"2024-03-18T16:37:01.242321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'7","last_scrub_stamp":"2024-03-18T16:37:01.242229+0000","last_deep_scrub":"30'7","last_deep_scrub_stamp":"2024-03-18T16:37:01.242229+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.242229+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:46.219619+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.342562+0000","last_change":"2024-03-18T16:37:00.342582+0000","last_active":"2024-03-18T16:37:00.342562+0000","last_peered":"2024-03-18T16:37:00.342562+0000","last_clean":"2024-03-18T16:37:00.342562+0000","last_became_active":"2024-03-18T16:21:22.974727+0000","last_became_peered":"2024-03-18T16:21:22.974727+0000","last_unstale":"2024-03-18T16:37:00.342562+0000","last_undegraded":"2024-03-18T16:37:00.342562+0000","last_fullsized":"2024-03-18T16:37:00.342562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:00.342470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:00.342470+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.342470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:55.663770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:15.022833+0000","last_change":"2024-03-18T16:37:15.022872+0000","last_active":"2024-03-18T16:37:15.022833+0000","last_peered":"2024-03-18T16:37:15.022833+0000","last_clean":"2024-03-18T16:37:15.022833+0000","last_became_active":"2024-03-18T16:21:55.150144+0000","last_became_peered":"2024-03-18T16:21:55.150144+0000","last_unstale":"2024-03-18T16:37:15.022833+0000","last_undegraded":"2024-03-18T16:37:15.022833+0000","last_fullsized":"2024-03-18T16:37:15.022833+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:15.022712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:15.022712+0000","last_clean_scrub_stamp":"2024-03-18T16:37:15.022712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:35:24.871279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4,2],"acting":[3,5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.699476+0000","last_change":"2024-03-18T16:37:10.699509+0000","last_active":"2024-03-18T16:37:10.699476+0000","last_peered":"2024-03-18T16:37:10.699476+0000","last_clean":"2024-03-18T16:37:10.699476+0000","last_became_active":"2024-03-18T16:21:23.056591+0000","last_became_peered":"2024-03-18T16:21:23.056591+0000","last_unstale":"2024-03-18T16:37:10.699476+0000","last_undegraded":"2024-03-18T16:37:10.699476+0000","last_fullsized":"2024-03-18T16:37:10.699476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.699371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.699371+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.699371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:52.794130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:00.088938+0000","last_change":"2024-03-18T16:37:00.088973+0000","last_active":"2024-03-18T16:37:00.088938+0000","last_peered":"2024-03-18T16:37:00.088938+0000","last_clean":"2024-03-18T16:37:00.088938+0000","last_became_active":"2024-03-18T16:21:22.947762+0000","last_became_peered":"2024-03-18T16:21:22.947762+0000","last_unstale":"2024-03-18T16:37:00.088938+0000","last_undegraded":"2024-03-18T16:37:00.088938+0000","last_fullsized":"2024-03-18T16:37:00.088938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:00.088814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:00.088814+0000","last_clean_scrub_stamp":"2024-03-18T16:37:00.088814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:11:15.114511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.089324+0000","last_change":"2024-03-18T16:37:04.089360+0000","last_active":"2024-03-18T16:37:04.089324+0000","last_peered":"2024-03-18T16:37:04.089324+0000","last_clean":"2024-03-18T16:37:04.089324+0000","last_became_active":"2024-03-18T16:21:54.845177+0000","last_became_peered":"2024-03-18T16:21:54.845177+0000","last_unstale":"2024-03-18T16:37:04.089324+0000","last_undegraded":"2024-03-18T16:37:04.089324+0000","last_fullsized":"2024-03-18T16:37:04.089324+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.089221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.089221+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.089221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:56.126973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.745455+0000","last_change":"2024-03-18T16:37:02.745493+0000","last_active":"2024-03-18T16:37:02.745455+0000","last_peered":"2024-03-18T16:37:02.745455+0000","last_clean":"2024-03-18T16:37:02.745455+0000","last_became_active":"2024-03-18T16:21:23.058918+0000","last_became_peered":"2024-03-18T16:21:23.058918+0000","last_unstale":"2024-03-18T16:37:02.745455+0000","last_undegraded":"2024-03-18T16:37:02.745455+0000","last_fullsized":"2024-03-18T16:37:02.745455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:02.745349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:02.745349+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.745349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:05.312649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.244991+0000","last_change":"2024-03-18T16:37:04.245026+0000","last_active":"2024-03-18T16:37:04.244991+0000","last_peered":"2024-03-18T16:37:04.244991+0000","last_clean":"2024-03-18T16:37:04.244991+0000","last_became_active":"2024-03-18T16:21:22.848546+0000","last_became_peered":"2024-03-18T16:21:22.848546+0000","last_unstale":"2024-03-18T16:37:04.244991+0000","last_undegraded":"2024-03-18T16:37:04.244991+0000","last_fullsized":"2024-03-18T16:37:04.244991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.244837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.244837+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.244837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:01.664431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.520840+0000","last_change":"2024-03-18T16:37:04.520874+0000","last_active":"2024-03-18T16:37:04.520840+0000","last_peered":"2024-03-18T16:37:04.520840+0000","last_clean":"2024-03-18T16:37:04.520840+0000","last_became_active":"2024-03-18T16:21:22.979400+0000","last_became_peered":"2024-03-18T16:21:22.979400+0000","last_unstale":"2024-03-18T16:37:04.520840+0000","last_undegraded":"2024-03-18T16:37:04.520840+0000","last_fullsized":"2024-03-18T16:37:04.520840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.520722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.520722+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.520722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:07.088022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:07.690340+0000","last_change":"2024-03-18T16:37:07.690360+0000","last_active":"2024-03-18T16:37:07.690340+0000","last_peered":"2024-03-18T16:37:07.690340+0000","last_clean":"2024-03-18T16:37:07.690340+0000","last_became_active":"2024-03-18T16:21:23.041625+0000","last_became_peered":"2024-03-18T16:21:23.041625+0000","last_unstale":"2024-03-18T16:37:07.690340+0000","last_undegraded":"2024-03-18T16:37:07.690340+0000","last_fullsized":"2024-03-18T16:37:07.690340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:07.690271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:07.690271+0000","last_clean_scrub_stamp":"2024-03-18T16:37:07.690271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:38:09.906793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.229073+0000","last_change":"2024-03-18T16:37:05.229119+0000","last_active":"2024-03-18T16:37:05.229073+0000","last_peered":"2024-03-18T16:37:05.229073+0000","last_clean":"2024-03-18T16:37:05.229073+0000","last_became_active":"2024-03-18T16:21:22.850495+0000","last_became_peered":"2024-03-18T16:21:22.850495+0000","last_unstale":"2024-03-18T16:37:05.229073+0000","last_undegraded":"2024-03-18T16:37:05.229073+0000","last_fullsized":"2024-03-18T16:37:05.229073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.228949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.228949+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.228949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:47.163075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.663205+0000","last_change":"2024-03-18T16:37:06.663224+0000","last_active":"2024-03-18T16:37:06.663205+0000","last_peered":"2024-03-18T16:37:06.663205+0000","last_clean":"2024-03-18T16:37:06.663205+0000","last_became_active":"2024-03-18T16:21:22.947684+0000","last_became_peered":"2024-03-18T16:21:22.947684+0000","last_unstale":"2024-03-18T16:37:06.663205+0000","last_undegraded":"2024-03-18T16:37:06.663205+0000","last_fullsized":"2024-03-18T16:37:06.663205+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.663150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.663150+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.663150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:48.757909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.231593+0000","last_change":"2024-03-18T16:37:06.231593+0000","last_active":"2024-03-18T16:37:06.231593+0000","last_peered":"2024-03-18T16:37:06.231593+0000","last_clean":"2024-03-18T16:37:06.231593+0000","last_became_active":"2024-03-18T16:21:22.850283+0000","last_became_peered":"2024-03-18T16:21:22.850283+0000","last_unstale":"2024-03-18T16:37:06.231593+0000","last_undegraded":"2024-03-18T16:37:06.231593+0000","last_fullsized":"2024-03-18T16:37:06.231593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.231456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.231456+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.231456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:02.674803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":28,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:05.042292+0000","last_change":"2024-03-18T16:37:05.042292+0000","last_active":"2024-03-18T16:37:05.042292+0000","last_peered":"2024-03-18T16:37:05.042292+0000","last_clean":"2024-03-18T16:37:05.042292+0000","last_became_active":"2024-03-18T16:21:55.267788+0000","last_became_peered":"2024-03-18T16:21:55.267788+0000","last_unstale":"2024-03-18T16:37:05.042292+0000","last_undegraded":"2024-03-18T16:37:05.042292+0000","last_fullsized":"2024-03-18T16:37:05.042292+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:05.042153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:05.042153+0000","last_clean_scrub_stamp":"2024-03-18T16:37:05.042153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:50:32.958462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.723414+0000","last_change":"2024-03-18T16:37:04.723414+0000","last_active":"2024-03-18T16:37:04.723414+0000","last_peered":"2024-03-18T16:37:04.723414+0000","last_clean":"2024-03-18T16:37:04.723414+0000","last_became_active":"2024-03-18T16:21:23.058988+0000","last_became_peered":"2024-03-18T16:21:23.058988+0000","last_unstale":"2024-03-18T16:37:04.723414+0000","last_undegraded":"2024-03-18T16:37:04.723414+0000","last_fullsized":"2024-03-18T16:37:04.723414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.723282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.723282+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.723282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:49.700672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.701724+0000","last_change":"2024-03-18T16:37:08.701724+0000","last_active":"2024-03-18T16:37:08.701724+0000","last_peered":"2024-03-18T16:37:08.701724+0000","last_clean":"2024-03-18T16:37:08.701724+0000","last_became_active":"2024-03-18T16:21:23.041847+0000","last_became_peered":"2024-03-18T16:21:23.041847+0000","last_unstale":"2024-03-18T16:37:08.701724+0000","last_undegraded":"2024-03-18T16:37:08.701724+0000","last_fullsized":"2024-03-18T16:37:08.701724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.701635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.701635+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.701635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:06:51.779634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":27,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.537132+0000","last_change":"2024-03-18T16:37:11.537166+0000","last_active":"2024-03-18T16:37:11.537132+0000","last_peered":"2024-03-18T16:37:11.537132+0000","last_clean":"2024-03-18T16:37:11.537132+0000","last_became_active":"2024-03-18T16:21:54.840090+0000","last_became_peered":"2024-03-18T16:21:54.840090+0000","last_unstale":"2024-03-18T16:37:11.537132+0000","last_undegraded":"2024-03-18T16:37:11.537132+0000","last_fullsized":"2024-03-18T16:37:11.537132+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.537030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.537030+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.537030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:39:07.980377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:01.077472+0000","last_change":"2024-03-18T16:37:01.077509+0000","last_active":"2024-03-18T16:37:01.077472+0000","last_peered":"2024-03-18T16:37:01.077472+0000","last_clean":"2024-03-18T16:37:01.077472+0000","last_became_active":"2024-03-18T16:21:23.042602+0000","last_became_peered":"2024-03-18T16:21:23.042602+0000","last_unstale":"2024-03-18T16:37:01.077472+0000","last_undegraded":"2024-03-18T16:37:01.077472+0000","last_fullsized":"2024-03-18T16:37:01.077472+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:01.077360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:01.077360+0000","last_clean_scrub_stamp":"2024-03-18T16:37:01.077360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:17.832116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:10.586524+0000","last_change":"2024-03-18T16:37:10.586558+0000","last_active":"2024-03-18T16:37:10.586524+0000","last_peered":"2024-03-18T16:37:10.586524+0000","last_clean":"2024-03-18T16:37:10.586524+0000","last_became_active":"2024-03-18T16:21:22.993394+0000","last_became_peered":"2024-03-18T16:21:22.993394+0000","last_unstale":"2024-03-18T16:37:10.586524+0000","last_undegraded":"2024-03-18T16:37:10.586524+0000","last_fullsized":"2024-03-18T16:37:10.586524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:10.586401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:10.586401+0000","last_clean_scrub_stamp":"2024-03-18T16:37:10.586401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:45:22.147206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:12.581559+0000","last_change":"2024-03-18T16:37:12.581594+0000","last_active":"2024-03-18T16:37:12.581559+0000","last_peered":"2024-03-18T16:37:12.581559+0000","last_clean":"2024-03-18T16:37:12.581559+0000","last_became_active":"2024-03-18T16:21:23.056297+0000","last_became_peered":"2024-03-18T16:21:23.056297+0000","last_unstale":"2024-03-18T16:37:12.581559+0000","last_undegraded":"2024-03-18T16:37:12.581559+0000","last_fullsized":"2024-03-18T16:37:12.581559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:12.581455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:12.581455+0000","last_clean_scrub_stamp":"2024-03-18T16:37:12.581455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:36:24.957908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":44,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.526857+0000","last_change":"2024-03-18T16:37:02.526893+0000","last_active":"2024-03-18T16:37:02.526857+0000","last_peered":"2024-03-18T16:37:02.526857+0000","last_clean":"2024-03-18T16:37:02.526857+0000","last_became_active":"2024-03-18T16:21:22.972425+0000","last_became_peered":"2024-03-18T16:21:22.972425+0000","last_unstale":"2024-03-18T16:37:02.526857+0000","last_undegraded":"2024-03-18T16:37:02.526857+0000","last_fullsized":"2024-03-18T16:37:02.526857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:02.526715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:02.526715+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.526715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:03.692015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:53.738279+0000","last_change":"2024-03-18T16:36:53.738313+0000","last_active":"2024-03-18T16:36:53.738279+0000","last_peered":"2024-03-18T16:36:53.738279+0000","last_clean":"2024-03-18T16:36:53.738279+0000","last_became_active":"2024-03-18T16:21:23.056776+0000","last_became_peered":"2024-03-18T16:21:23.056776+0000","last_unstale":"2024-03-18T16:36:53.738279+0000","last_undegraded":"2024-03-18T16:36:53.738279+0000","last_fullsized":"2024-03-18T16:36:53.738279+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:53.738175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:53.738175+0000","last_clean_scrub_stamp":"2024-03-18T16:36:53.738175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:16:57.558446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:57.277459+0000","last_change":"2024-03-18T16:36:57.277509+0000","last_active":"2024-03-18T16:36:57.277459+0000","last_peered":"2024-03-18T16:36:57.277459+0000","last_clean":"2024-03-18T16:36:57.277459+0000","last_became_active":"2024-03-18T16:21:23.044430+0000","last_became_peered":"2024-03-18T16:21:23.044430+0000","last_unstale":"2024-03-18T16:36:57.277459+0000","last_undegraded":"2024-03-18T16:36:57.277459+0000","last_fullsized":"2024-03-18T16:36:57.277459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:57.277271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:57.277271+0000","last_clean_scrub_stamp":"2024-03-18T16:36:57.277271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:51:58.442571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:02.295813+0000","last_change":"2024-03-18T16:37:02.295833+0000","last_active":"2024-03-18T16:37:02.295813+0000","last_peered":"2024-03-18T16:37:02.295813+0000","last_clean":"2024-03-18T16:37:02.295813+0000","last_became_active":"2024-03-18T16:21:23.041538+0000","last_became_peered":"2024-03-18T16:21:23.041538+0000","last_unstale":"2024-03-18T16:37:02.295813+0000","last_undegraded":"2024-03-18T16:37:02.295813+0000","last_fullsized":"2024-03-18T16:37:02.295813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:02.295733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:02.295733+0000","last_clean_scrub_stamp":"2024-03-18T16:37:02.295733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:57.119844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:04.677882+0000","last_change":"2024-03-18T16:37:04.677904+0000","last_active":"2024-03-18T16:37:04.677882+0000","last_peered":"2024-03-18T16:37:04.677882+0000","last_clean":"2024-03-18T16:37:04.677882+0000","last_became_active":"2024-03-18T16:21:23.036678+0000","last_became_peered":"2024-03-18T16:21:23.036678+0000","last_unstale":"2024-03-18T16:37:04.677882+0000","last_undegraded":"2024-03-18T16:37:04.677882+0000","last_fullsized":"2024-03-18T16:37:04.677882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:04.677810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:04.677810+0000","last_clean_scrub_stamp":"2024-03-18T16:37:04.677810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:47:37.272501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.654558+0000","last_change":"2024-03-18T16:37:06.654579+0000","last_active":"2024-03-18T16:37:06.654558+0000","last_peered":"2024-03-18T16:37:06.654558+0000","last_clean":"2024-03-18T16:37:06.654558+0000","last_became_active":"2024-03-18T16:21:23.041883+0000","last_became_peered":"2024-03-18T16:21:23.041883+0000","last_unstale":"2024-03-18T16:37:06.654558+0000","last_undegraded":"2024-03-18T16:37:06.654558+0000","last_fullsized":"2024-03-18T16:37:06.654558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.654475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.654475+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.654475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:47:52.914426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:09.361511+0000","last_change":"2024-03-18T16:37:09.361545+0000","last_active":"2024-03-18T16:37:09.361511+0000","last_peered":"2024-03-18T16:37:09.361511+0000","last_clean":"2024-03-18T16:37:09.361511+0000","last_became_active":"2024-03-18T16:21:22.994847+0000","last_became_peered":"2024-03-18T16:21:22.994847+0000","last_unstale":"2024-03-18T16:37:09.361511+0000","last_undegraded":"2024-03-18T16:37:09.361511+0000","last_fullsized":"2024-03-18T16:37:09.361511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:09.361390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:09.361390+0000","last_clean_scrub_stamp":"2024-03-18T16:37:09.361390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:52:10.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.173994+0000","last_change":"2024-03-18T16:37:08.174029+0000","last_active":"2024-03-18T16:37:08.173994+0000","last_peered":"2024-03-18T16:37:08.173994+0000","last_clean":"2024-03-18T16:37:08.173994+0000","last_became_active":"2024-03-18T16:21:22.947205+0000","last_became_peered":"2024-03-18T16:21:22.947205+0000","last_unstale":"2024-03-18T16:37:08.173994+0000","last_undegraded":"2024-03-18T16:37:08.173994+0000","last_fullsized":"2024-03-18T16:37:08.173994+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.173886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.173886+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.173886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:42:32.282777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:17.215115+0000","last_change":"2024-03-18T16:37:17.215157+0000","last_active":"2024-03-18T16:37:17.215115+0000","last_peered":"2024-03-18T16:37:17.215115+0000","last_clean":"2024-03-18T16:37:17.215115+0000","last_became_active":"2024-03-18T16:21:23.049603+0000","last_became_peered":"2024-03-18T16:21:23.049603+0000","last_unstale":"2024-03-18T16:37:17.215115+0000","last_undegraded":"2024-03-18T16:37:17.215115+0000","last_fullsized":"2024-03-18T16:37:17.215115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:17.215003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:17.215003+0000","last_clean_scrub_stamp":"2024-03-18T16:37:17.215003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:09:59.149129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.629366+0000","last_change":"2024-03-18T16:37:11.629405+0000","last_active":"2024-03-18T16:37:11.629366+0000","last_peered":"2024-03-18T16:37:11.629366+0000","last_clean":"2024-03-18T16:37:11.629366+0000","last_became_active":"2024-03-18T16:21:23.049244+0000","last_became_peered":"2024-03-18T16:21:23.049244+0000","last_unstale":"2024-03-18T16:37:11.629366+0000","last_undegraded":"2024-03-18T16:37:11.629366+0000","last_fullsized":"2024-03-18T16:37:11.629366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.629262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.629262+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.629262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:13:29.777022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:11.282220+0000","last_change":"2024-03-18T16:37:11.282259+0000","last_active":"2024-03-18T16:37:11.282220+0000","last_peered":"2024-03-18T16:37:11.282220+0000","last_clean":"2024-03-18T16:37:11.282220+0000","last_became_active":"2024-03-18T16:21:22.995049+0000","last_became_peered":"2024-03-18T16:21:22.995049+0000","last_unstale":"2024-03-18T16:37:11.282220+0000","last_undegraded":"2024-03-18T16:37:11.282220+0000","last_fullsized":"2024-03-18T16:37:11.282220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:11.282101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:11.282101+0000","last_clean_scrub_stamp":"2024-03-18T16:37:11.282101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:54:13.849401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":47,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:08.628559+0000","last_change":"2024-03-18T16:37:08.628559+0000","last_active":"2024-03-18T16:37:08.628559+0000","last_peered":"2024-03-18T16:37:08.628559+0000","last_clean":"2024-03-18T16:37:08.628559+0000","last_became_active":"2024-03-18T16:21:22.949380+0000","last_became_peered":"2024-03-18T16:21:22.949380+0000","last_unstale":"2024-03-18T16:37:08.628559+0000","last_undegraded":"2024-03-18T16:37:08.628559+0000","last_fullsized":"2024-03-18T16:37:08.628559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:08.628449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:08.628449+0000","last_clean_scrub_stamp":"2024-03-18T16:37:08.628449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:40.223638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:36:56.783318+0000","last_change":"2024-03-18T16:36:56.783359+0000","last_active":"2024-03-18T16:36:56.783318+0000","last_peered":"2024-03-18T16:36:56.783318+0000","last_clean":"2024-03-18T16:36:56.783318+0000","last_became_active":"2024-03-18T16:21:23.048486+0000","last_became_peered":"2024-03-18T16:21:23.048486+0000","last_unstale":"2024-03-18T16:36:56.783318+0000","last_undegraded":"2024-03-18T16:36:56.783318+0000","last_fullsized":"2024-03-18T16:36:56.783318+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:36:56.783216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:36:56.783216+0000","last_clean_scrub_stamp":"2024-03-18T16:36:56.783216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:46:52.104516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":45,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:06.507540+0000","last_change":"2024-03-18T16:37:06.507574+0000","last_active":"2024-03-18T16:37:06.507540+0000","last_peered":"2024-03-18T16:37:06.507540+0000","last_clean":"2024-03-18T16:37:06.507540+0000","last_became_active":"2024-03-18T16:21:23.052480+0000","last_became_peered":"2024-03-18T16:21:23.052480+0000","last_unstale":"2024-03-18T16:37:06.507540+0000","last_undegraded":"2024-03-18T16:37:06.507540+0000","last_fullsized":"2024-03-18T16:37:06.507540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:06.507434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:06.507434+0000","last_clean_scrub_stamp":"2024-03-18T16:37:06.507434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:58:51.484825+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":46,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-18T16:37:16.194824+0000","last_change":"2024-03-18T16:37:16.194860+0000","last_active":"2024-03-18T16:37:16.194824+0000","last_peered":"2024-03-18T16:37:16.194824+0000","last_clean":"2024-03-18T16:37:16.194824+0000","last_became_active":"2024-03-18T16:21:23.041591+0000","last_became_peered":"2024-03-18T16:21:23.041591+0000","last_unstale":"2024-03-18T16:37:16.194824+0000","last_undegraded":"2024-03-18T16:37:16.194824+0000","last_fullsized":"2024-03-18T16:37:16.194824+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:37:16.194699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:37:16.194699+0000","last_clean_scrub_stamp":"2024-03-18T16:37:16.194699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:48:13.614293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10264,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":209,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":20528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64882,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509643,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31588,"kb_used_data":412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340252,"statfs":{"total":96636764160,"available":96604418048,"internally_reserved":0,"allocated":421888,"data_stored":729077,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509644,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":344064,"data_stored":137939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509642,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":352256,"data_stored":141275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509644,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":138141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509642,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":413696,"data_stored":727851,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542347,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":352256,"data_stored":138071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575051,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":348160,"data_stored":141241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575051,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":368640,"data_stored":140541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32441,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:37:42.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-18T16:37:43.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.069+0000 7f59f8200640 1 Processor -- start 2024-03-18T16:37:43.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.069+0000 7f59f8200640 1 -- start start 2024-03-18T16:37:43.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.070+0000 7f59f8200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f59f00723b0 0x7f59f00c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:43.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.070+0000 7f59f8200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00c54f0 0x7f59f015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:43.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.070+0000 7f59f8200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f0158390 con 0x7f59f0141700 2024-03-18T16:37:43.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.070+0000 7f59f8200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 -- 0x7f59f0141ae0 con 0x7f59f00723b0 2024-03-18T16:37:43.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.070+0000 7f59f8200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f59f0158670 con 0x7f59f00c54f0 2024-03-18T16:37:43.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.071+0000 7f59f6200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00c54f0 0x7f59f015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:43.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.072+0000 7f59f6200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00c54f0 0x7f59f015bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3301/0 says I am v2:172.21.15.102:40778/0 (socket says 172.21.15.102:40778) 2024-03-18T16:37:43.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.072+0000 7f59f6200640 1 -- 172.21.15.102:0/2190650552 learned_addr learned my addr 172.21.15.102:0/2190650552 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:37:43.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.075+0000 7f59f6200640 1 -- 172.21.15.102:0/2190650552 >> v2:172.21.15.102:3300/0 conn(0x7f59f00723b0 msgr2=0x7f59f00c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:37:43.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.075+0000 7f59f6200640 1 --2- 172.21.15.102:0/2190650552 >> v2:172.21.15.102:3300/0 conn(0x7f59f00723b0 0x7f59f00c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.075+0000 7f59f6200640 1 -- 172.21.15.102:0/2190650552 >> v1:172.21.15.102:6789/0 conn(0x7f59f0141700 legacy=0x7f59f006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:37:43.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.075+0000 7f59f6200640 1 -- 172.21.15.102:0/2190650552 --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f0158950 con 0x7f59f00c54f0 2024-03-18T16:37:43.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.079+0000 7f59f6200640 1 --2- 172.21.15.102:0/2190650552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00c54f0 0x7f59f015bb50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f59e8009e50 tx=0x7f59e802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a74cc782a658398 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:43.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.083+0000 7f59efe00640 1 -- 172.21.15.102:0/2190650552 <== mon.2 v2:172.21.15.154:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e802cb00 con 0x7f59f00c54f0 2024-03-18T16:37:43.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.083+0000 7f59efe00640 1 -- 172.21.15.102:0/2190650552 <== mon.2 v2:172.21.15.154:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e802cca0 con 0x7f59f00c54f0 2024-03-18T16:37:43.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.083+0000 7f59efe00640 1 -- 172.21.15.102:0/2190650552 <== mon.2 v2:172.21.15.154:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8034610 con 0x7f59f00c54f0 2024-03-18T16:37:43.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.084+0000 7f59f8200640 1 -- 172.21.15.102:0/2190650552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00c54f0 msgr2=0x7f59f015bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:43.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.084+0000 7f59f8200640 1 --2- 172.21.15.102:0/2190650552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00c54f0 0x7f59f015bb50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f59e8009e50 tx=0x7f59e802bf60 comp rx=0 tx=0).stop 2024-03-18T16:37:43.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.084+0000 7f59f8200640 1 -- 172.21.15.102:0/2190650552 shutdown_connections 2024-03-18T16:37:43.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.084+0000 7f59f8200640 1 --2- 172.21.15.102:0/2190650552 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00c54f0 0x7f59f015bb50 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.084+0000 7f59f8200640 1 --2- 172.21.15.102:0/2190650552 >> v2:172.21.15.102:3300/0 conn(0x7f59f00723b0 0x7f59f00c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.084+0000 7f59f8200640 1 -- 172.21.15.102:0/2190650552 >> 172.21.15.102:0/2190650552 conn(0x7f59f00c83a0 msgr2=0x7f59f00c8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:43.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.084+0000 7f59f8200640 1 -- 172.21.15.102:0/2190650552 shutdown_connections 2024-03-18T16:37:43.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.084+0000 7f59f8200640 1 -- 172.21.15.102:0/2190650552 wait complete. 2024-03-18T16:37:43.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.086+0000 7f59f8200640 1 Processor -- start 2024-03-18T16:37:43.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.086+0000 7f59f8200640 1 -- start start 2024-03-18T16:37:43.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.087+0000 7f59f8200640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00723b0 0x7f59f01573c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:43.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.087+0000 7f59f8200640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59f00c54f0 0x7f59f0136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.087+0000 7f59f8200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59f0141700 0x7f59f0137100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.087+0000 7f59f8200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 -- 0x7f59f006c180 con 0x7f59f0141700 2024-03-18T16:37:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.087+0000 7f59f8200640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59f0158f10 con 0x7f59f00c54f0 2024-03-18T16:37:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.087+0000 7f59f8200640 1 -- --> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] -- mon_getmap magic: 0 -- 0x7f59f006c300 con 0x7f59f00723b0 2024-03-18T16:37:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.088+0000 7f59f5800640 1 --2- >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00723b0 0x7f59f01573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:43.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.088+0000 7f59f4e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59f00c54f0 0x7f59f0136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:43.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.088+0000 7f59f4e00640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59f00c54f0 0x7f59f0136bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.102:37092/0 (socket says 172.21.15.102:37092) 2024-03-18T16:37:43.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.088+0000 7f59f4e00640 1 -- 172.21.15.102:0/1871424930 learned_addr learned my addr 172.21.15.102:0/1871424930 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-18T16:37:43.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.091+0000 7f59f4e00640 1 -- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00723b0 msgr2=0x7f59f01573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:43.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.091+0000 7f59f4e00640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00723b0 0x7f59f01573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.091+0000 7f59f4e00640 1 -- 172.21.15.102:0/1871424930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59f0141700 msgr2=0x7f59f0137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:43.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.091+0000 7f59f4e00640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59f0141700 0x7f59f0137100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.091+0000 7f59f4e00640 1 -- 172.21.15.102:0/1871424930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f0154f70 con 0x7f59f00c54f0 2024-03-18T16:37:43.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.095+0000 7f59f4e00640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59f00c54f0 0x7f59f0136bc0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00b410 tx=0x7f59dc00b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:43.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.097+0000 7f59ed600640 1 -- 172.21.15.102:0/1871424930 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc015020 con 0x7f59f00c54f0 2024-03-18T16:37:43.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.098+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f0155250 con 0x7f59f00c54f0 2024-03-18T16:37:43.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.098+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f01597c0 con 0x7f59f00c54f0 2024-03-18T16:37:43.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.099+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f0074fd0 con 0x7f59f00c54f0 2024-03-18T16:37:43.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.100+0000 7f59ed600640 1 -- 172.21.15.102:0/1871424930 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59dc011830 con 0x7f59f00c54f0 2024-03-18T16:37:43.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.100+0000 7f59ed600640 1 -- 172.21.15.102:0/1871424930 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc011c00 con 0x7f59f00c54f0 2024-03-18T16:37:43.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.402+0000 7f59ed600640 1 -- 172.21.15.102:0/1871424930 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f59dc018020 con 0x7f59f00c54f0 2024-03-18T16:37:43.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.403+0000 7f59ed600640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f59d006bbd0 0x7f59d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:37:43.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.404+0000 7f59f5800640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f59d006bbd0 0x7f59d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:37:43.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.404+0000 7f59ed600640 1 -- 172.21.15.102:0/1871424930 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6821+0+0 (secure 0 0 0) 0x7f59dc0573a0 con 0x7f59f00c54f0 2024-03-18T16:37:43.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.404+0000 7f59f5800640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f59d006bbd0 0x7f59d006e090 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f59e40017f0 tx=0x7f59e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:37:43.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.414+0000 7f59ed600640 1 -- 172.21.15.102:0/1871424930 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f59dc01f020 con 0x7f59f00c54f0 2024-03-18T16:37:43.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.611+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f59f00c63a0 con 0x7f59f00c54f0 2024-03-18T16:37:43.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.824+0000 7f59ed600640 1 -- 172.21.15.102:0/1871424930 <== mon.1 v2:172.21.15.154:3300/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f59dc059a00 con 0x7f59f00c54f0 2024-03-18T16:37:43.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.824+0000 7f59ed600640 1 -- 172.21.15.102:0/1871424930 <== mon.1 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7f59dc059300 con 0x7f59f00c54f0 2024-03-18T16:37:43.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f59d006bbd0 msgr2=0x7f59d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f59d006bbd0 0x7f59d006e090 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f59e40017f0 tx=0x7f59e4009040 comp rx=0 tx=0).stop 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59f00c54f0 msgr2=0x7f59f0136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59f00c54f0 0x7f59f0136bc0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00b410 tx=0x7f59dc00b8d0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 shutdown_connections 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:6800/2819855665,v1:172.21.15.154:6801/2819855665] conn(0x7f59d006bbd0 0x7f59d006e090 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59f0141700 0x7f59f0137100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59f00c54f0 0x7f59f0136bc0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 --2- 172.21.15.102:0/1871424930 >> [v2:172.21.15.154:3301/0,v1:172.21.15.154:6790/0] conn(0x7f59f00723b0 0x7f59f01573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 >> 172.21.15.102:0/1871424930 conn(0x7f59f00c83a0 msgr2=0x7f59f00c9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 shutdown_connections 2024-03-18T16:37:43.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-18T16:37:43.828+0000 7f59f8200640 1 -- 172.21.15.102:0/1871424930 wait complete. 2024-03-18T16:37:43.847 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-18T16:37:43.848 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-18T16:37:43.848 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-18T16:37:43.848 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:37:54.800 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:37:54.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:37:59.760 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:17:26.019 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:59.843 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:17:26.103 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:59.849 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:26.195 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:59.853 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:26.198 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:59.854 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:17:26.200 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:59.858 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:17:26.203 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:59.880 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:17:26.226 37582== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:59.883 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:17:26.143 36772== Warning: unimplemented fcntl command: 1036 2024-03-18T16:37:59.889 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:17:26.234 37581== Warning: unimplemented fcntl command: 1036 2024-03-18T16:38:24.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:38:24.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:38:55.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:38:55.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:39:25.223 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:39:25.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:39:55.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:39:55.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:40:25.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:40:25.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:40:55.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:40:55.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:41:25.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:41:25.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:41:55.802 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:41:55.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:42:25.959 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:42:26.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:42:42.916 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-18T16:42:42.917 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-18T16:42:42.917 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:42:42.918 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-18T16:42:42.917+0000 7f6f8ac00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 0 (PID: 37039) UID: 0 2024-03-18T16:42:42.918 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-18T16:42:42.917+0000 7f6f8ac00640 -1 osd.0 30 *** Got signal Terminated *** 2024-03-18T16:42:49.018 INFO:tasks.ceph.osd.0:Stopped 2024-03-18T16:42:49.019 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-18T16:42:49.019 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:42:49.020 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-18T16:42:49.019+0000 7f76efc00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 1 (PID: 37045) UID: 0 2024-03-18T16:42:49.020 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-18T16:42:49.019+0000 7f76efc00640 -1 osd.1 32 *** Got signal Terminated *** 2024-03-18T16:42:52.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-18T16:42:55.121 INFO:tasks.ceph.osd.1:Stopped 2024-03-18T16:42:55.121 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-18T16:42:55.121 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:42:55.122 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-18T16:42:55.121+0000 7fb36ea00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 2 (PID: 37047) UID: 0 2024-03-18T16:42:55.123 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-18T16:42:55.121+0000 7fb36ea00640 -1 osd.2 34 *** Got signal Terminated *** 2024-03-18T16:42:56.121 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:42:56.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:42:59.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-18T16:42:59.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-18T16:43:01.223 INFO:tasks.ceph.osd.2:Stopped 2024-03-18T16:43:01.223 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-18T16:43:01.224 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:01.225 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-18T16:43:01.223+0000 7f3e77c00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 3 (PID: 37050) UID: 0 2024-03-18T16:43:01.225 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-18T16:43:01.223+0000 7f3e77c00640 -1 osd.3 36 *** Got signal Terminated *** 2024-03-18T16:43:05.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-03-18T16:43:05.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-18T16:43:07.326 INFO:tasks.ceph.osd.3:Stopped 2024-03-18T16:43:07.327 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-18T16:43:07.327 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:07.328 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-03-18T16:43:07.326+0000 7f039546a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 4 (PID: 38252) UID: 0 2024-03-18T16:43:07.328 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-03-18T16:43:07.326+0000 7f039546a640 -1 osd.4 38 *** Got signal Terminated *** 2024-03-18T16:43:12.385 INFO:tasks.ceph.mds.a.smithi102.stderr:2024-03-18T16:42:50.971+0000 b5f1640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a (PID: 40801) UID: 0 2024-03-18T16:43:12.386 INFO:tasks.ceph.mds.a.smithi102.stderr:2024-03-18T16:42:50.972+0000 b5f1640 -1 mds.a *** got signal Terminated *** 2024-03-18T16:43:12.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-03-18T16:43:12.396 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-03-18T16:43:13.429 INFO:tasks.ceph.osd.4:Stopped 2024-03-18T16:43:13.429 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-18T16:43:13.429 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:13.430 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-03-18T16:43:13.428+0000 7f6b6e937640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 5 (PID: 38254) UID: 0 2024-03-18T16:43:13.430 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-03-18T16:43:13.428+0000 7f6b6e937640 -1 osd.5 40 *** Got signal Terminated *** 2024-03-18T16:43:18.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-18T16:43:18.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-03-18T16:43:19.530 INFO:tasks.ceph.osd.5:Stopped 2024-03-18T16:43:19.531 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-18T16:43:19.531 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:19.531 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-03-18T16:43:19.530+0000 7f94c54a4640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 6 (PID: 38256) UID: 0 2024-03-18T16:43:19.531 INFO:tasks.ceph.osd.6.smithi154.stderr:2024-03-18T16:43:19.530+0000 7f94c54a4640 -1 osd.6 42 *** Got signal Terminated *** 2024-03-18T16:43:24.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-18T16:43:24.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-18T16:43:25.632 INFO:tasks.ceph.osd.6:Stopped 2024-03-18T16:43:25.633 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-18T16:43:25.633 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:25.634 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-03-18T16:43:25.632+0000 7ff06d036640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 7 (PID: 38258) UID: 0 2024-03-18T16:43:25.634 INFO:tasks.ceph.osd.7.smithi154.stderr:2024-03-18T16:43:25.632+0000 7ff06d036640 -1 osd.7 42 *** Got signal Terminated *** 2024-03-18T16:43:26.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:43:26.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:43:31.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-18T16:43:31.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-03-18T16:43:31.735 INFO:tasks.ceph.osd.7:Stopped 2024-03-18T16:43:31.735 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-18T16:43:31.736 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-18T16:43:31.736 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:31.810 INFO:tasks.ceph.mgr.y:Stopped 2024-03-18T16:43:31.810 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-18T16:43:31.810 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:31.811 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:43:31.810+0000 7f73ba488640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 37579) UID: 0 2024-03-18T16:43:31.811 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-03-18T16:43:31.810+0000 7f73ba488640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-18T16:43:31.836 INFO:tasks.ceph.mgr.x:Stopped 2024-03-18T16:43:31.836 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-18T16:43:31.836 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-18T16:43:31.836 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:31.871 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-18T16:43:31.852+0000 9f9a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 36766) UID: 0 2024-03-18T16:43:31.874 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-18T16:43:31.854+0000 9f9a640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T16:43:32.497 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:22:58.757 36772== 2024-03-18T16:43:32.497 INFO:tasks.ceph.mon.a.smithi102.stderr:==00:00:22:58.757 36772== Exit program on first error (--exit-on-first-error=yes) 2024-03-18T16:43:32.544 INFO:tasks.ceph.mon.a.smithi102.stderr:daemon-helper: command failed with exit status 42 2024-03-18T16:43:36.830 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-18T16:43:36.830 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-18T16:43:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-18T16:43:37.937 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-18T16:43:37.939 INFO:tasks.ceph.mon.a:Stopped 2024-03-18T16:43:37.939 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-18T16:43:37.940 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:37.961 INFO:tasks.ceph.mon.b.smithi154.stderr:2024-03-18T16:43:37.944+0000 9f9a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 37574) UID: 0 2024-03-18T16:43:37.962 INFO:tasks.ceph.mon.b.smithi154.stderr:2024-03-18T16:43:37.946+0000 9f9a640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-18T16:43:38.616 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:04.962 37581== 2024-03-18T16:43:38.616 INFO:tasks.ceph.mon.b.smithi154.stderr:==00:00:23:04.962 37581== Exit program on first error (--exit-on-first-error=yes) 2024-03-18T16:43:38.626 INFO:tasks.ceph.mon.b.smithi154.stderr:daemon-helper: command failed with exit status 42 2024-03-18T16:43:42.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-18T16:43:42.536 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-18T16:43:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-18T16:43:44.042 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi154 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-18T16:43:44.042 INFO:tasks.ceph.mon.b:Stopped 2024-03-18T16:43:44.043 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-18T16:43:44.043 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:43:44.057 INFO:tasks.ceph.mon.c.smithi154.stderr:2024-03-18T16:43:44.046+0000 9f9a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 37575) UID: 0 2024-03-18T16:43:44.058 INFO:tasks.ceph.mon.c.smithi154.stderr:2024-03-18T16:43:44.047+0000 9f9a640 -1 mon.c@2(probing) e1 *** Got Signal Terminated *** 2024-03-18T16:43:44.561 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:23:10.906 37582== 2024-03-18T16:43:44.561 INFO:tasks.ceph.mon.c.smithi154.stderr:==00:00:23:10.906 37582== Exit program on first error (--exit-on-first-error=yes) 2024-03-18T16:43:44.571 INFO:tasks.ceph.mon.c.smithi154.stderr:daemon-helper: command failed with exit status 42 2024-03-18T16:43:48.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-18T16:43:48.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-18T16:43:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-18T16:43:50.145 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi154 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-18T16:43:50.146 INFO:tasks.ceph.mon.c:Stopped 2024-03-18T16:43:50.146 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-18T16:43:50.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v MON_DOWN | head -n 1 2024-03-18T16:43:50.260 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:43:50.260 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-18T16:43:50.419 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:43:50.419 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-18T16:43:50.536 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:43:50.536 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-18T16:43:50.707 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:43:50.707 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-18T16:43:50.818 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:43:50.818 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-18T16:43:50.944 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:43:50.944 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-18T16:43:51.010 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:43:51.010 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-18T16:43:51.075 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:43:51.075 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-18T16:43:51.125 INFO:tasks.ceph:Archiving mon data... 2024-03-18T16:43:51.127 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/data/mon.a.tgz 2024-03-18T16:43:51.127 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-18T16:43:51.169 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.ySjy4xxAwW 2024-03-18T16:43:51.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ySjy4xxAwW 2024-03-18T16:43:51.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.ySjy4xxAwW 2024-03-18T16:43:51.496 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.ySjy4xxAwW is 321KB 2024-03-18T16:43:51.586 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.ySjy4xxAwW 2024-03-18T16:43:51.627 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/data/mon.b.tgz 2024-03-18T16:43:51.627 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-03-18T16:43:51.641 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.jk4E1sT5Ft 2024-03-18T16:43:51.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.jk4E1sT5Ft 2024-03-18T16:43:51.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.jk4E1sT5Ft 2024-03-18T16:43:51.871 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.jk4E1sT5Ft is 349KB 2024-03-18T16:43:51.939 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.jk4E1sT5Ft 2024-03-18T16:43:51.953 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/data/mon.c.tgz 2024-03-18T16:43:51.953 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-03-18T16:43:52.006 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.RyH6SwNSoc 2024-03-18T16:43:52.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.RyH6SwNSoc 2024-03-18T16:43:52.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.RyH6SwNSoc 2024-03-18T16:43:52.231 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.RyH6SwNSoc is 349KB 2024-03-18T16:43:52.325 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.RyH6SwNSoc 2024-03-18T16:43:52.338 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-18T16:43:52.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-18T16:43:52.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-18T16:43:52.532 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-18T16:43:52.572 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-18T16:43:52.613 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-18T16:43:52.687 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-18T16:43:52.759 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-18T16:43:52.776 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-18T16:43:52.822 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-18T16:43:52.861 INFO:tasks.ceph:Archiving crash dumps... 2024-03-18T16:43:52.863 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/remote/smithi102/crash 2024-03-18T16:43:52.864 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-18T16:43:52.934 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/remote/smithi154/crash 2024-03-18T16:43:52.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-18T16:43:52.962 INFO:tasks.ceph:Shutting down logrotate 2024-03-18T16:43:52.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:43:52.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-18T16:43:53.081 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-18T16:43:53.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-18T16:43:53.177 INFO:tasks.ceph:Compressing logs... 2024-03-18T16:43:53.177 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T16:43:53.182 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T16:43:53.243 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-18T16:43:53.243 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37962.log 2024-03-18T16:43:53.251 INFO:teuthology.orchestra.run.smithi154.stderr:gzip/var/log/ceph/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:gzip/var/log/ceph/ceph-client.admin.37962.log: -5 --verbose -- /var/log/ceph/valgrind/mds.f.log 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/valgrind/mon.c.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37962.log.gz 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mds.b.log 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/valgrind/mds.f.log: 55.2% -- replaced with /var/log/ceph/valgrind/mds.f.log.gz 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/valgrind/mds.b.log: 55.2% -- replaced with /var/log/ceph/valgrind/mds.b.log.gz 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.d.log 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37922.log 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/valgrind/mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38161.log 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr: 55.2% -- replaced with /var/log/ceph/valgrind/mds.d.log.gz 2024-03-18T16:43:53.252 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37922.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37922.log.gz 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38161.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.audit.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38161.log.gz 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38117.log 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38006.log 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37773.log 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38117.log.gz 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.38006.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37773.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38006.log.gz 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.37773.log.gz 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-03-18T16:43:53.253 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-18T16:43:53.254 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-18T16:43:53.254 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37831.log 2024-03-18T16:43:53.257 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45147.log 2024-03-18T16:43:53.258 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47821.log 2024-03-18T16:43:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45147.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45147.log.gz 2024-03-18T16:43:53.259 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42491.log 2024-03-18T16:43:53.260 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51816.log 2024-03-18T16:43:53.261 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47821.log: /var/log/ceph/ceph-client.admin.42491.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47821.log.gz 2024-03-18T16:43:53.261 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-18T16:43:53.262 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42491.log.gz 2024-03-18T16:43:53.262 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40694.log 2024-03-18T16:43:53.263 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37831.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37831.log.gz 2024-03-18T16:43:53.263 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59168.log 2024-03-18T16:43:53.263 INFO:teuthology.orchestra.run.smithi102.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51816.log.gz 2024-03-18T16:43:53.264 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49826.log 2024-03-18T16:43:53.264 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40694.log.gz 2024-03-18T16:43:53.265 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59168.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59168.log.gz 2024-03-18T16:43:53.265 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-18T16:43:53.266 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56221.log 2024-03-18T16:43:53.267 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mon.a.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49826.log.gz 2024-03-18T16:43:53.267 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54511.log 2024-03-18T16:43:53.268 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43338.log 2024-03-18T16:43:53.269 INFO:teuthology.orchestra.run.smithi102.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56221.log.gz 2024-03-18T16:43:53.270 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54907.log 2024-03-18T16:43:53.270 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54511.log.gz 2024-03-18T16:43:53.271 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46465.log 2024-03-18T16:43:53.271 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43338.log.gz 2024-03-18T16:43:53.273 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44484.log 2024-03-18T16:43:53.273 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54907.log.gz 2024-03-18T16:43:53.273 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54259.log 2024-03-18T16:43:53.274 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46465.log.gz 2024-03-18T16:43:53.275 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37871.log 2024-03-18T16:43:53.275 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37175.log 2024-03-18T16:43:53.276 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44484.log.gz 2024-03-18T16:43:53.276 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45399.log 2024-03-18T16:43:53.277 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54259.log.gz 2024-03-18T16:43:53.278 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.37175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47779.log 2024-03-18T16:43:53.278 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37175.log.gz 2024-03-18T16:43:53.279 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58208.log 2024-03-18T16:43:53.279 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45399.log.gz 2024-03-18T16:43:53.280 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.log: 93.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-18T16:43:53.280 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-18T16:43:53.281 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.37871.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37871.log.gz 2024-03-18T16:43:53.281 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46525.log 2024-03-18T16:43:53.281 INFO:teuthology.orchestra.run.smithi102.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47779.log.gz 2024-03-18T16:43:53.282 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44331.log 2024-03-18T16:43:53.282 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58208.log.gz 2024-03-18T16:43:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58292.log 2024-03-18T16:43:53.283 INFO:teuthology.orchestra.run.smithi102.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46525.log.gz 2024-03-18T16:43:53.284 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47569.log 2024-03-18T16:43:53.284 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44331.log.gz 2024-03-18T16:43:53.285 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46903.log 2024-03-18T16:43:53.286 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58292.log.gz 2024-03-18T16:43:53.287 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59462.log 2024-03-18T16:43:53.287 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47569.log.gz 2024-03-18T16:43:53.288 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59546.log 2024-03-18T16:43:53.288 INFO:teuthology.orchestra.run.smithi102.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46903.log.gz 2024-03-18T16:43:53.289 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59066.log 2024-03-18T16:43:53.289 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59462.log.gz 2024-03-18T16:43:53.290 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36771.log 2024-03-18T16:43:53.291 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59546.log.gz 2024-03-18T16:43:53.292 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58898.log 2024-03-18T16:43:53.292 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59066.log.gz 2024-03-18T16:43:53.293 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.36771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48554.log 2024-03-18T16:43:53.293 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-18T16:43:53.294 INFO:teuthology.orchestra.run.smithi102.stderr: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.36771.log.gz 2024-03-18T16:43:53.294 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50684.log 2024-03-18T16:43:53.295 INFO:teuthology.orchestra.run.smithi102.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58898.log.gz 2024-03-18T16:43:53.296 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47131.log 2024-03-18T16:43:53.296 INFO:teuthology.orchestra.run.smithi102.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48554.log.gz 2024-03-18T16:43:53.297 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53214.log 2024-03-18T16:43:53.297 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50684.log.gz 2024-03-18T16:43:53.298 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49034.log 2024-03-18T16:43:53.299 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47131.log.gz 2024-03-18T16:43:53.299 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-03-18T16:43:53.300 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44853.log 2024-03-18T16:43:53.300 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53214.log.gz 2024-03-18T16:43:53.301 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48134.log 2024-03-18T16:43:53.301 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49034.log.gz 2024-03-18T16:43:53.302 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56179.log 2024-03-18T16:43:53.302 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44853.log.gz 2024-03-18T16:43:53.303 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58124.log 2024-03-18T16:43:53.303 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48134.log.gz 2024-03-18T16:43:53.304 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55867.log 2024-03-18T16:43:53.305 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56179.log.gz 2024-03-18T16:43:53.306 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49784.log 2024-03-18T16:43:53.307 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58124.log.gz 2024-03-18T16:43:53.307 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-18T16:43:53.307 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43245.log 2024-03-18T16:43:53.307 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55867.log.gz 2024-03-18T16:43:53.308 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52002.log 2024-03-18T16:43:53.308 INFO:teuthology.orchestra.run.smithi102.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49784.log.gz 2024-03-18T16:43:53.309 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58040.log 2024-03-18T16:43:53.310 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43245.log.gz 2024-03-18T16:43:53.311 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52002.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.c.log 2024-03-18T16:43:53.311 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52002.log.gz 2024-03-18T16:43:53.312 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58040.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.a.log 2024-03-18T16:43:53.312 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58040.log.gz 2024-03-18T16:43:53.313 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/mds.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-18T16:43:53.313 INFO:teuthology.orchestra.run.smithi102.stderr: 55.3% -- replaced with /var/log/ceph/valgrind/mds.c.log.gz 2024-03-18T16:43:53.315 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/mds.a.log: gzip 55.3% -5 -- replaced with /var/log/ceph/valgrind/mds.a.log.gz --verbose 2024-03-18T16:43:53.315 INFO:teuthology.orchestra.run.smithi102.stderr: -- /var/log/ceph/valgrind/mds.e.log 2024-03-18T16:43:53.316 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54217.log 2024-03-18T16:43:53.316 INFO:teuthology.orchestra.run.smithi102.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-18T16:43:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/valgrind/mds.e.log: 55.3% -- replaced with /var/log/ceph/valgrind/mds.e.log.gz 2024-03-18T16:43:53.317 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56263.log 2024-03-18T16:43:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51522.log 2024-03-18T16:43:53.324 INFO:teuthology.orchestra.run.smithi102.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54217.log.gz 2024-03-18T16:43:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43994.log 2024-03-18T16:43:53.324 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56263.log.gz 2024-03-18T16:43:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50120.log 2024-03-18T16:43:53.324 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51522.log.gz 2024-03-18T16:43:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44351.log 2024-03-18T16:43:53.324 INFO:teuthology.orchestra.run.smithi102.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43994.log.gz 2024-03-18T16:43:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57559.log 2024-03-18T16:43:53.325 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50120.log.gz 2024-03-18T16:43:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58814.log 2024-03-18T16:43:53.325 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44351.log.gz 2024-03-18T16:43:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57205.log 2024-03-18T16:43:53.326 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57559.log.gz 2024-03-18T16:43:53.328 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51564.log 2024-03-18T16:43:53.328 INFO:teuthology.orchestra.run.smithi102.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58814.log.gz 2024-03-18T16:43:53.329 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57079.log 2024-03-18T16:43:53.329 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57205.log.gz 2024-03-18T16:43:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46945.log 2024-03-18T16:43:53.330 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51564.log.gz 2024-03-18T16:43:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50726.log 2024-03-18T16:43:53.332 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57079.log.gz 2024-03-18T16:43:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51249.log 2024-03-18T16:43:53.333 INFO:teuthology.orchestra.run.smithi102.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46945.log.gz 2024-03-18T16:43:53.334 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43792.log 2024-03-18T16:43:53.334 INFO:teuthology.orchestra.run.smithi102.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50726.log.gz 2024-03-18T16:43:53.334 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: 93.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-18T16:43:53.335 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57788.log 2024-03-18T16:43:53.335 INFO:teuthology.orchestra.run.smithi102.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51249.log.gz 2024-03-18T16:43:53.336 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57686.log 2024-03-18T16:43:53.336 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43792.log.gz 2024-03-18T16:43:53.337 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49910.log 2024-03-18T16:43:53.338 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57788.log.gz 2024-03-18T16:43:53.339 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58436.log 2024-03-18T16:43:53.339 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57686.log.gz 2024-03-18T16:43:53.340 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56827.log 2024-03-18T16:43:53.340 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49910.log.gz 2024-03-18T16:43:53.341 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49346.log 2024-03-18T16:43:53.342 INFO:teuthology.orchestra.run.smithi102.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58436.log.gz 2024-03-18T16:43:53.343 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55219.log 2024-03-18T16:43:53.343 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56827.log.gz 2024-03-18T16:43:53.344 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57602.log 2024-03-18T16:43:53.344 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49346.log.gz 2024-03-18T16:43:53.345 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46255.log 2024-03-18T16:43:53.345 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55219.log.gz 2024-03-18T16:43:53.347 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49262.log 2024-03-18T16:43:53.347 INFO:teuthology.orchestra.run.smithi102.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57602.log.gz 2024-03-18T16:43:53.348 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45105.log 2024-03-18T16:43:53.348 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46255.log.gz 2024-03-18T16:43:53.349 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57956.log 2024-03-18T16:43:53.349 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49262.log.gz 2024-03-18T16:43:53.351 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48782.log 2024-03-18T16:43:53.351 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45105.log.gz 2024-03-18T16:43:53.352 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46651.log 2024-03-18T16:43:53.352 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57956.log.gz 2024-03-18T16:43:53.353 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54174.log 2024-03-18T16:43:53.353 INFO:teuthology.orchestra.run.smithi102.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48782.log.gz 2024-03-18T16:43:53.354 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42701.log 2024-03-18T16:43:53.354 INFO:teuthology.orchestra.run.smithi102.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46651.log.gz 2024-03-18T16:43:53.356 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46003.log 2024-03-18T16:43:53.356 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54174.log.gz 2024-03-18T16:43:53.357 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54865.log 2024-03-18T16:43:53.357 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42701.log.gz 2024-03-18T16:43:53.358 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47863.log 2024-03-18T16:43:53.358 INFO:teuthology.orchestra.run.smithi102.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46003.log.gz 2024-03-18T16:43:53.360 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53904.log 2024-03-18T16:43:53.360 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54865.log.gz 2024-03-18T16:43:53.361 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49160.log 2024-03-18T16:43:53.361 INFO:teuthology.orchestra.run.smithi102.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47863.log.gz 2024-03-18T16:43:53.362 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51165.log 2024-03-18T16:43:53.362 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53904.log.gz 2024-03-18T16:43:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57914.log 2024-03-18T16:43:53.364 INFO:teuthology.orchestra.run.smithi102.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49160.log.gz 2024-03-18T16:43:53.364 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46339.log 2024-03-18T16:43:53.365 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51165.log.gz 2024-03-18T16:43:53.365 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55825.log 2024-03-18T16:43:53.366 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57914.log.gz 2024-03-18T16:43:53.367 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46339.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46339.log.gz 2024-03-18T16:43:53.367 INFO:teuthology.orchestra.run.smithi102.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58856.log 2024-03-18T16:43:53.368 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50390.log 2024-03-18T16:43:53.368 INFO:teuthology.orchestra.run.smithi102.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55825.log.gz 2024-03-18T16:43:53.369 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58856.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.58856.log.gz -5 2024-03-18T16:43:53.369 INFO:teuthology.orchestra.run.smithi102.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51039.log 2024-03-18T16:43:53.370 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48824.log 2024-03-18T16:43:53.370 INFO:teuthology.orchestra.run.smithi102.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50390.log.gz 2024-03-18T16:43:53.371 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51039.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51039.log.gz 2024-03-18T16:43:53.371 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42954.log 2024-03-18T16:43:53.372 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57475.log 2024-03-18T16:43:53.373 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48824.log.gz 2024-03-18T16:43:53.373 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52086.log 2024-03-18T16:43:53.373 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42954.log.gz 2024-03-18T16:43:53.375 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53340.log 2024-03-18T16:43:53.375 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57475.log.gz 2024-03-18T16:43:53.376 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52086.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52086.log.gz 2024-03-18T16:43:53.376 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45567.log 2024-03-18T16:43:53.377 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56473.log 2024-03-18T16:43:53.377 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53340.log.gz 2024-03-18T16:43:53.378 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43560.log 2024-03-18T16:43:53.378 INFO:teuthology.orchestra.run.smithi102.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.45567.log.gz 2024-03-18T16:43:53.379 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45189.log 2024-03-18T16:43:53.380 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56473.log.gz 2024-03-18T16:43:53.380 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43560.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.43560.log.gz --verbose 2024-03-18T16:43:53.381 INFO:teuthology.orchestra.run.smithi102.stderr: -- /var/log/ceph/ceph-client.admin.48512.log 2024-03-18T16:43:53.382 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53778.log 2024-03-18T16:43:53.382 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45189.log.gz 2024-03-18T16:43:53.383 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42575.log 2024-03-18T16:43:53.383 INFO:teuthology.orchestra.run.smithi102.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48512.log.gz 2024-03-18T16:43:53.384 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50078.log 2024-03-18T16:43:53.384 INFO:teuthology.orchestra.run.smithi102.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53778.log.gz 2024-03-18T16:43:53.385 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42575.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42575.log.gz 2024-03-18T16:43:53.385 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42843.log 2024-03-18T16:43:53.386 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50078.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50078.log.gz 2024-03-18T16:43:53.387 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56305.log 2024-03-18T16:43:53.387 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52902.log 2024-03-18T16:43:53.388 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42843.log: /var/log/ceph/ceph-client.admin.56305.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42843.log.gz 2024-03-18T16:43:53.388 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56305.log.gz 2024-03-18T16:43:53.389 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59210.log 2024-03-18T16:43:53.390 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/ceph-client.admin.52902.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56119.log 2024-03-18T16:43:53.390 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52902.log.gz 2024-03-18T16:43:53.391 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59210.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59210.log.gz 2024-03-18T16:43:53.391 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50348.log 2024-03-18T16:43:53.392 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41679.log 2024-03-18T16:43:53.393 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56119.log: /var/log/ceph/ceph-client.admin.50348.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56119.log.gz 2024-03-18T16:43:53.393 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43119.log 2024-03-18T16:43:53.393 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50348.log.gz 2024-03-18T16:43:53.395 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44095.log 2024-03-18T16:43:53.395 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41679.log.gz 2024-03-18T16:43:53.395 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45315.log 2024-03-18T16:43:53.396 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43119.log.gz 2024-03-18T16:43:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49076.log 2024-03-18T16:43:53.397 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44095.log.gz 2024-03-18T16:43:53.398 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45315.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45315.log.gz 2024-03-18T16:43:53.398 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41553.log 2024-03-18T16:43:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47215.log 2024-03-18T16:43:53.399 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49076.log.gz 2024-03-18T16:43:53.400 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41553.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41553.log.gz 2024-03-18T16:43:53.400 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2024-03-18T16:43:53.401 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45961.log 2024-03-18T16:43:53.402 INFO:teuthology.orchestra.run.smithi102.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47215.log.gz 2024-03-18T16:43:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50432.log 2024-03-18T16:43:53.403 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2024-03-18T16:43:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45961.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45961.log.gz 2024-03-18T16:43:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46297.log 2024-03-18T16:43:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50768.log 2024-03-18T16:43:53.406 INFO:teuthology.orchestra.run.smithi102.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50432.log.gz 2024-03-18T16:43:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46297.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46297.log.gz 2024-03-18T16:43:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45901.log 2024-03-18T16:43:53.407 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44604.log 2024-03-18T16:43:53.407 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50768.log: /var/log/ceph/ceph-client.admin.45901.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50768.log.gz 2024-03-18T16:43:53.408 INFO:teuthology.orchestra.run.smithi102.stderr:gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45901.log.gz 2024-03-18T16:43:53.408 INFO:teuthology.orchestra.run.smithi102.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51774.log 2024-03-18T16:43:53.409 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40904.log 2024-03-18T16:43:53.409 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44604.log.gz 2024-03-18T16:43:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51774.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45441.log 2024-03-18T16:43:53.410 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51774.log.gz 2024-03-18T16:43:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40904.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40904.log.gz 2024-03-18T16:43:53.411 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48656.log 2024-03-18T16:43:53.412 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57349.log 2024-03-18T16:43:53.413 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2%/var/log/ceph/ceph-client.admin.48656.log: -- replaced with /var/log/ceph/ceph-client.admin.45441.log.gz 2024-03-18T16:43:53.414 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59730.log 2024-03-18T16:43:53.414 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48656.log.gz 2024-03-18T16:43:53.415 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57349.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57349.log.gz 2024-03-18T16:43:53.415 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55513.log 2024-03-18T16:43:53.416 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59730.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-18T16:43:53.416 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59730.log.gz 2024-03-18T16:43:53.417 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56557.log 2024-03-18T16:43:53.417 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55513.log.gz 2024-03-18T16:43:53.418 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52482.log 2024-03-18T16:43:53.420 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52170.log 2024-03-18T16:43:53.420 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56557.log.gz 2024-03-18T16:43:53.420 INFO:teuthology.orchestra.run.smithi102.stderr: 89.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-18T16:43:53.421 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52482.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52482.log.gz 2024-03-18T16:43:53.421 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53820.log 2024-03-18T16:43:53.422 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58166.log 2024-03-18T16:43:53.423 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53820.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52170.log.gz 2024-03-18T16:43:53.423 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53820.log.gz 2024-03-18T16:43:53.423 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43380.log 2024-03-18T16:43:53.424 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-03-18T16:43:53.425 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58166.log: /var/log/ceph/ceph-client.admin.43380.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58166.log.gz 2024-03-18T16:43:53.425 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43380.log.gz 2024-03-18T16:43:53.425 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51478.log 2024-03-18T16:43:53.426 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46381.log 2024-03-18T16:43:53.427 INFO:teuthology.orchestra.run.smithi102.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-03-18T16:43:53.428 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43778.log 2024-03-18T16:43:53.428 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51478.log.gz 2024-03-18T16:43:53.429 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46381.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46381.log.gz 2024-03-18T16:43:53.429 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49742.log 2024-03-18T16:43:53.430 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50558.log 2024-03-18T16:43:53.430 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43778.log.gz 2024-03-18T16:43:53.431 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41469.log 2024-03-18T16:43:53.431 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49742.log.gz 2024-03-18T16:43:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50558.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50558.log.gz 2024-03-18T16:43:53.432 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56869.log 2024-03-18T16:43:53.433 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52692.log 2024-03-18T16:43:53.434 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41469.log.gz/var/log/ceph/ceph-client.admin.56869.log: 2024-03-18T16:43:53.435 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54655.log 2024-03-18T16:43:53.435 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56869.log.gz 2024-03-18T16:43:53.436 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52692.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52692.log.gz 2024-03-18T16:43:53.436 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49430.log 2024-03-18T16:43:53.437 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55387.log 2024-03-18T16:43:53.437 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54655.log: /var/log/ceph/ceph-client.admin.49430.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54655.log.gz 2024-03-18T16:43:53.438 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49430.log.gz 2024-03-18T16:43:53.438 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43161.log 2024-03-18T16:43:53.439 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/ceph-client.admin.55387.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48260.log 2024-03-18T16:43:53.440 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55387.log.gz 2024-03-18T16:43:53.440 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43161.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43161.log.gz 2024-03-18T16:43:53.440 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44100.log 2024-03-18T16:43:53.441 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59630.log 2024-03-18T16:43:53.442 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48260.log: /var/log/ceph/ceph-client.admin.44100.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48260.log.gz 2024-03-18T16:43:53.442 INFO:teuthology.orchestra.run.smithi102.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44100.log.gz 2024-03-18T16:43:53.443 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54781.log 2024-03-18T16:43:53.444 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/ceph-client.admin.59630.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40809.log 2024-03-18T16:43:53.444 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59630.log.gz 2024-03-18T16:43:53.445 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54781.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54781.log.gz 2024-03-18T16:43:53.445 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58082.log 2024-03-18T16:43:53.446 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57830.log 2024-03-18T16:43:53.447 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40809.log: /var/log/ceph/ceph-client.admin.58082.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40809.log.gz 2024-03-18T16:43:53.447 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58082.log.gz 2024-03-18T16:43:53.447 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58646.log 2024-03-18T16:43:53.448 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/ceph-client.admin.57830.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56431.log 2024-03-18T16:43:53.449 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57830.log.gz 2024-03-18T16:43:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58646.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57391.log 2024-03-18T16:43:53.450 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58646.log.gz 2024-03-18T16:43:53.451 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46861.log 2024-03-18T16:43:53.451 INFO:teuthology.orchestra.run.smithi102.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56431.log.gz 2024-03-18T16:43:53.452 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57391.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57391.log.gz 2024-03-18T16:43:53.452 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55597.log 2024-03-18T16:43:53.453 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56911.log 2024-03-18T16:43:53.453 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46861.log.gz 2024-03-18T16:43:53.454 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58940.log 2024-03-18T16:43:53.454 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55597.log.gz 2024-03-18T16:43:53.456 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54090.log 2024-03-18T16:43:53.456 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56911.log.gz 2024-03-18T16:43:53.457 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58940.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58940.log.gz 2024-03-18T16:43:53.457 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43782.log 2024-03-18T16:43:53.458 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57265.log 2024-03-18T16:43:53.458 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54090.log: /var/log/ceph/ceph-client.admin.43782.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54090.log.gz 2024-03-18T16:43:53.459 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47509.log 2024-03-18T16:43:53.459 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43782.log.gz 2024-03-18T16:43:53.460 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50912.log 2024-03-18T16:43:53.460 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57265.log.gz 2024-03-18T16:43:53.461 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54301.log 2024-03-18T16:43:53.461 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47509.log.gz 2024-03-18T16:43:53.462 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50912.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50912.log.gz 2024-03-18T16:43:53.462 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41829.log 2024-03-18T16:43:53.463 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43779.log 2024-03-18T16:43:53.464 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54301.log.gz 2024-03-18T16:43:53.465 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42904.log 2024-03-18T16:43:53.465 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41829.log.gz 2024-03-18T16:43:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43779.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43779.log.gz 2024-03-18T16:43:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58250.log 2024-03-18T16:43:53.467 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55741.log 2024-03-18T16:43:53.468 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42904.log.gz 2024-03-18T16:43:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48386.log 2024-03-18T16:43:53.469 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58250.log.gz 2024-03-18T16:43:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55741.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55741.log.gz 2024-03-18T16:43:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49952.log 2024-03-18T16:43:53.470 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45483.log 2024-03-18T16:43:53.471 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48386.log: /var/log/ceph/ceph-client.admin.49952.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48386.log.gz 2024-03-18T16:43:53.472 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49952.log.gz 2024-03-18T16:43:53.472 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41070.log 2024-03-18T16:43:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47947.log 2024-03-18T16:43:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45483.log: /var/log/ceph/ceph-client.admin.41070.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45483.log.gz 2024-03-18T16:43:53.474 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41070.log.gz 2024-03-18T16:43:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59690.log 2024-03-18T16:43:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50954.log 2024-03-18T16:43:53.475 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47947.log.gz 2024-03-18T16:43:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58520.log 2024-03-18T16:43:53.477 INFO:teuthology.orchestra.run.smithi102.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59690.log.gz 2024-03-18T16:43:53.478 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41721.log 2024-03-18T16:43:53.478 INFO:teuthology.orchestra.run.smithi102.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50954.log.gz 2024-03-18T16:43:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56599.log 2024-03-18T16:43:53.479 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58520.log.gz 2024-03-18T16:43:53.481 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51393.log 2024-03-18T16:43:53.481 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41721.log.gz 2024-03-18T16:43:53.482 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48428.log 2024-03-18T16:43:53.482 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56599.log.gz 2024-03-18T16:43:53.483 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54469.log 2024-03-18T16:43:53.483 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51393.log.gz 2024-03-18T16:43:53.484 INFO:teuthology.orchestra.run.smithi154.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-18T16:43:53.485 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48428.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-18T16:43:53.485 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48428.log.gz 2024-03-18T16:43:53.486 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42801.log 2024-03-18T16:43:53.486 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54469.log.gz 2024-03-18T16:43:53.487 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48950.log 2024-03-18T16:43:53.488 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50516.log 2024-03-18T16:43:53.489 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42801.log.gz 2024-03-18T16:43:53.490 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41241.log 2024-03-18T16:43:53.490 INFO:teuthology.orchestra.run.smithi102.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48950.log.gz 2024-03-18T16:43:53.491 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45859.log 2024-03-18T16:43:53.491 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50516.log.gz 2024-03-18T16:43:53.492 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44050.log 2024-03-18T16:43:53.493 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41241.log.gz 2024-03-18T16:43:53.494 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59336.log 2024-03-18T16:43:53.494 INFO:teuthology.orchestra.run.smithi102.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45859.log.gz 2024-03-18T16:43:53.495 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50306.log 2024-03-18T16:43:53.495 INFO:teuthology.orchestra.run.smithi102.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44050.log.gz 2024-03-18T16:43:53.497 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59336.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59336.log.gz 2024-03-18T16:43:53.497 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41595.log 2024-03-18T16:43:53.498 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50306.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50306.log.gz 2024-03-18T16:43:53.498 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53526.log 2024-03-18T16:43:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54385.log 2024-03-18T16:43:53.500 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41595.log.gz 2024-03-18T16:43:53.501 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53526.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53526.log.gz 2024-03-18T16:43:53.502 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46819.log 2024-03-18T16:43:53.502 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54385.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54385.log.gz 2024-03-18T16:43:53.503 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46171.log 2024-03-18T16:43:53.504 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54048.log 2024-03-18T16:43:53.504 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46819.log.gz 2024-03-18T16:43:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48302.log 2024-03-18T16:43:53.506 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46171.log.gz 2024-03-18T16:43:53.507 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47047.log 2024-03-18T16:43:53.507 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54048.log.gz 2024-03-18T16:43:53.508 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48302.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48302.log.gz 2024-03-18T16:43:53.509 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45775.log 2024-03-18T16:43:53.510 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47047.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55657.log 2024-03-18T16:43:53.510 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47047.log.gz 2024-03-18T16:43:53.511 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48032.log 2024-03-18T16:43:53.511 INFO:teuthology.orchestra.run.smithi102.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45775.log.gz 2024-03-18T16:43:53.512 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51606.log 2024-03-18T16:43:53.513 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55657.log.gz 2024-03-18T16:43:53.514 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48032.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48032.log.gz 2024-03-18T16:43:53.514 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52254.log 2024-03-18T16:43:53.515 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51606.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52440.log 2024-03-18T16:43:53.515 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51606.log.gz 2024-03-18T16:43:53.517 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44117.log 2024-03-18T16:43:53.517 INFO:teuthology.orchestra.run.smithi102.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52254.log.gz 2024-03-18T16:43:53.518 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52440.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52440.log.gz 2024-03-18T16:43:53.519 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47467.log 2024-03-18T16:43:53.519 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44117.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44117.log.gz 2024-03-18T16:43:53.520 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52524.log 2024-03-18T16:43:53.521 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49118.log 2024-03-18T16:43:53.521 INFO:teuthology.orchestra.run.smithi102.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47467.log.gz 2024-03-18T16:43:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52524.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52524.log.gz 2024-03-18T16:43:53.523 INFO:teuthology.orchestra.run.smithi102.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52734.log 2024-03-18T16:43:53.524 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49118.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49118.log.gz 2024-03-18T16:43:53.525 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57037.log 2024-03-18T16:43:53.526 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57307.log 2024-03-18T16:43:53.526 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52734.log.gz 2024-03-18T16:43:53.527 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51081.log 2024-03-18T16:43:53.527 INFO:teuthology.orchestra.run.smithi102.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57037.log.gz 2024-03-18T16:43:53.528 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57307.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57307.log.gz 2024-03-18T16:43:53.529 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57121.log 2024-03-18T16:43:53.530 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55471.log 2024-03-18T16:43:53.530 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51081.log.gz 2024-03-18T16:43:53.531 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47425.log 2024-03-18T16:43:53.532 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57121.log.gz 2024-03-18T16:43:53.533 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55471.log: gzip 89.1% -5 --verbose -- /var/log/ceph/ceph-client.admin.58604.log 2024-03-18T16:43:53.533 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55471.log.gz 2024-03-18T16:43:53.534 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41157.log 2024-03-18T16:43:53.534 INFO:teuthology.orchestra.run.smithi102.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47425.log.gz 2024-03-18T16:43:53.535 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58604.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58604.log.gz 2024-03-18T16:43:53.536 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56077.log 2024-03-18T16:43:53.537 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41157.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41157.log.gz 2024-03-18T16:43:53.537 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49388.log 2024-03-18T16:43:53.539 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41385.log 2024-03-18T16:43:53.539 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56077.log.gz 2024-03-18T16:43:53.540 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53568.log 2024-03-18T16:43:53.540 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49388.log.gz 2024-03-18T16:43:53.542 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41385.log: gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41385.log.gz 2024-03-18T16:43:53.542 INFO:teuthology.orchestra.run.smithi102.stderr: /var/log/ceph/ceph-client.admin.44666.log 2024-03-18T16:43:53.543 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53568.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53568.log.gz 2024-03-18T16:43:53.543 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54991.log 2024-03-18T16:43:53.545 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52296.log 2024-03-18T16:43:53.545 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44666.log.gz 2024-03-18T16:43:53.546 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54991.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54991.log.gz 2024-03-18T16:43:53.547 INFO:teuthology.orchestra.run.smithi102.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42659.log 2024-03-18T16:43:53.547 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52296.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52296.log.gz 2024-03-18T16:43:53.548 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58394.log 2024-03-18T16:43:53.550 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52212.log 2024-03-18T16:43:53.550 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42659.log.gz 2024-03-18T16:43:53.550 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58394.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58394.log.gz 2024-03-18T16:43:53.551 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47257.log 2024-03-18T16:43:53.552 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49868.log 2024-03-18T16:43:53.552 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52212.log.gz 2024-03-18T16:43:53.554 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50036.log 2024-03-18T16:43:53.554 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47257.log.gz 2024-03-18T16:43:53.555 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48470.log 2024-03-18T16:43:53.555 INFO:teuthology.orchestra.run.smithi102.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49868.log.gz 2024-03-18T16:43:53.557 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50036.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50036.log.gz 2024-03-18T16:43:53.557 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59024.log 2024-03-18T16:43:53.557 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48470.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48470.log.gz 2024-03-18T16:43:53.558 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-18T16:43:53.560 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48908.log 2024-03-18T16:43:53.560 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59024.log.gz 2024-03-18T16:43:53.561 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48074.log 2024-03-18T16:43:53.563 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48908.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48908.log.gz 2024-03-18T16:43:53.563 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45691.log 2024-03-18T16:43:53.564 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48074.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48074.log.gz 2024-03-18T16:43:53.565 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44712.log 2024-03-18T16:43:53.566 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45691.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45691.log.gz 2024-03-18T16:43:53.567 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/ceph-client.admin.44712.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48596.log 2024-03-18T16:43:53.568 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44712.log.gz 2024-03-18T16:43:53.569 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57433.log 2024-03-18T16:43:53.570 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48596.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48596.log.gz 2024-03-18T16:43:53.570 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53694.log 2024-03-18T16:43:53.572 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57433.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57433.log.gz 2024-03-18T16:43:53.573 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54132.log 2024-03-18T16:43:53.574 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53694.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53694.log.gz 2024-03-18T16:43:53.575 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50996.log 2024-03-18T16:43:53.576 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54132.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54132.log.gz 2024-03-18T16:43:53.577 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57728.log 2024-03-18T16:43:53.577 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50996.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50996.log.gz 2024-03-18T16:43:53.578 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53088.log 2024-03-18T16:43:53.580 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57728.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57728.log.gz 2024-03-18T16:43:53.581 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52650.log 2024-03-18T16:43:53.581 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53088.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53088.log.gz 2024-03-18T16:43:53.582 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41427.log 2024-03-18T16:43:53.583 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52650.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52650.log.gz 2024-03-18T16:43:53.584 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55345.log 2024-03-18T16:43:53.585 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41427.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41427.log.gz 2024-03-18T16:43:53.586 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59126.log 2024-03-18T16:43:53.587 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55345.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55345.log.gz 2024-03-18T16:43:53.588 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56641.log 2024-03-18T16:43:53.589 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59126.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59126.log.gz 2024-03-18T16:43:53.590 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45525.log 2024-03-18T16:43:53.590 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56641.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56641.log.gz 2024-03-18T16:43:53.591 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56953.log 2024-03-18T16:43:53.592 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45525.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45525.log.gz 2024-03-18T16:43:53.593 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53736.log 2024-03-18T16:43:53.594 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56953.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56953.log.gz 2024-03-18T16:43:53.595 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55699.log 2024-03-18T16:43:53.596 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53736.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53736.log.gz 2024-03-18T16:43:53.597 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44135.log 2024-03-18T16:43:53.598 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55699.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55699.log.gz 2024-03-18T16:43:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51351.log 2024-03-18T16:43:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44135.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44135.log.gz 2024-03-18T16:43:53.601 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44131.log 2024-03-18T16:43:53.602 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51351.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51351.log.gz 2024-03-18T16:43:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57872.log 2024-03-18T16:43:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44131.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44131.log.gz 2024-03-18T16:43:53.605 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43518.log 2024-03-18T16:43:53.606 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57872.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57872.log.gz 2024-03-18T16:43:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54949.log 2024-03-18T16:43:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43518.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43518.log.gz 2024-03-18T16:43:53.609 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42759.log 2024-03-18T16:43:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54949.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54949.log.gz 2024-03-18T16:43:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48992.log 2024-03-18T16:43:53.611 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42759.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42759.log.gz 2024-03-18T16:43:53.612 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43077.log 2024-03-18T16:43:53.613 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48992.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48992.log.gz 2024-03-18T16:43:53.614 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55093.log 2024-03-18T16:43:53.615 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43077.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43077.log.gz 2024-03-18T16:43:53.616 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55909.log 2024-03-18T16:43:53.617 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55093.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55093.log.gz 2024-03-18T16:43:53.618 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43796.log 2024-03-18T16:43:53.619 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55909.log.gz 2024-03-18T16:43:53.621 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43796.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43796.log.gz 2024-03-18T16:43:53.621 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50260.log 2024-03-18T16:43:53.622 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53862.log 2024-03-18T16:43:53.624 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50260.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50260.log.gz 2024-03-18T16:43:53.625 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/ceph-client.admin.53862.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45063.log 2024-03-18T16:43:53.625 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53862.log.gz 2024-03-18T16:43:53.626 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40854.log 2024-03-18T16:43:53.628 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45063.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45063.log.gz 2024-03-18T16:43:53.628 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55303.log/var/log/ceph/ceph-client.admin.40854.log: 2024-03-18T16:43:53.629 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40854.log.gz 2024-03-18T16:43:53.631 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55303.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55303.log.gz 2024-03-18T16:43:53.632 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47695.log 2024-03-18T16:43:53.633 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51291.log 2024-03-18T16:43:53.635 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47695.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47695.log.gz 2024-03-18T16:43:53.635 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41763.log 2024-03-18T16:43:53.636 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51291.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51291.log.gz 2024-03-18T16:43:53.637 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41763.log: /var/log/ceph/ceph-client.admin.51732.log 2024-03-18T16:43:53.638 INFO:teuthology.orchestra.run.smithi102.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.41763.log.gz 2024-03-18T16:43:53.639 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52566.log 2024-03-18T16:43:53.640 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51732.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51732.log.gz 2024-03-18T16:43:53.641 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41511.log 2024-03-18T16:43:53.642 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52566.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52566.log.gz 2024-03-18T16:43:53.643 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41511.log: /var/log/ceph/ceph-client.admin.51648.log 2024-03-18T16:43:53.644 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41511.log.gz 2024-03-18T16:43:53.645 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46777.log 2024-03-18T16:43:53.646 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51648.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51648.log.gz 2024-03-18T16:43:53.647 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45231.log 2024-03-18T16:43:53.648 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46777.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46777.log.gz 2024-03-18T16:43:53.649 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48344.log 2024-03-18T16:43:53.650 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45231.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45231.log.gz 2024-03-18T16:43:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53610.log 2024-03-18T16:43:53.651 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48344.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48344.log.gz 2024-03-18T16:43:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-03-18T16:43:53.653 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53610.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53610.log.gz 2024-03-18T16:43:53.654 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52398.log 2024-03-18T16:43:53.657 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mds.e.log: /var/log/ceph/ceph-client.admin.52398.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-18T16:43:53.658 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52398.log.gz 2024-03-18T16:43:53.659 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51960.log 2024-03-18T16:43:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5/var/log/ceph/ceph-client.admin.51960.log: --verbose -- /var/log/ceph/ceph-client.admin.52128.log 2024-03-18T16:43:53.662 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51960.log.gz 2024-03-18T16:43:53.664 INFO:teuthology.orchestra.run.smithi102.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-03-18T16:43:53.664 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47383.log 2024-03-18T16:43:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52128.log: /var/log/ceph/ceph-client.admin.47383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59294.log 2024-03-18T16:43:53.669 INFO:teuthology.orchestra.run.smithi102.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52128.log.gz 2024-03-18T16:43:53.670 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48218.log 2024-03-18T16:43:53.672 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2%/var/log/ceph/ceph-client.admin.59294.log: -- replaced with /var/log/ceph/ceph-client.admin.47383.log.gz 2024-03-18T16:43:53.673 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59294.log.gz 2024-03-18T16:43:53.674 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48218.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48218.log.gz 2024-03-18T16:43:53.675 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51876.log 2024-03-18T16:43:53.677 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44753.log 2024-03-18T16:43:53.678 INFO:teuthology.orchestra.run.smithi102.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51876.log.gz 2024-03-18T16:43:53.680 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46129.log 2024-03-18T16:43:53.682 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44753.log: 94.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-18T16:43:53.683 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46129.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44753.log.gz 2024-03-18T16:43:53.684 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43794.log 2024-03-18T16:43:53.684 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46129.log.gz 2024-03-18T16:43:53.685 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43471.log 2024-03-18T16:43:53.686 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43794.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43794.log.gz 2024-03-18T16:43:53.688 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/ceph-client.admin.43471.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56035.log 2024-03-18T16:43:53.688 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43471.log.gz 2024-03-18T16:43:53.689 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55177.log 2024-03-18T16:43:53.690 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56035.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56035.log.gz 2024-03-18T16:43:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54571.log 2024-03-18T16:43:53.691 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55177.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55177.log.gz 2024-03-18T16:43:53.692 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55783.log 2024-03-18T16:43:53.693 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54571.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54571.log.gz 2024-03-18T16:43:53.694 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48866.log 2024-03-18T16:43:53.695 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55783.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55783.log.gz 2024-03-18T16:43:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53046.log 2024-03-18T16:43:53.697 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48866.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48866.log.gz 2024-03-18T16:43:53.698 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55555.log 2024-03-18T16:43:53.699 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53046.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53046.log.gz 2024-03-18T16:43:53.700 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55951.log 2024-03-18T16:43:53.701 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55555.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55555.log.gz 2024-03-18T16:43:53.702 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51123.log 2024-03-18T16:43:53.703 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55951.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55951.log.gz 2024-03-18T16:43:53.704 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/ceph-client.admin.51123.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.58772.log 2024-03-18T16:43:53.705 INFO:teuthology.orchestra.run.smithi102.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51123.log.gz 2024-03-18T16:43:53.706 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52776.log 2024-03-18T16:43:53.707 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58772.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58772.log.gz 2024-03-18T16:43:53.708 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54006.log/var/log/ceph/ceph-client.admin.52776.log: 2024-03-18T16:43:53.708 INFO:teuthology.orchestra.run.smithi102.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52776.log.gz 2024-03-18T16:43:53.709 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57998.log 2024-03-18T16:43:53.711 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54006.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54006.log.gz 2024-03-18T16:43:53.711 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52860.log 2024-03-18T16:43:53.712 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57998.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57998.log.gz 2024-03-18T16:43:53.713 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48698.log 2024-03-18T16:43:53.714 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52860.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52860.log.gz 2024-03-18T16:43:53.715 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40974.log 2024-03-18T16:43:53.716 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48698.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48698.log.gz 2024-03-18T16:43:53.717 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57163.log 2024-03-18T16:43:53.718 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.40974.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40974.log.gz 2024-03-18T16:43:53.719 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52818.log 2024-03-18T16:43:53.720 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57163.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57163.log.gz 2024-03-18T16:43:53.721 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53382.log 2024-03-18T16:43:53.722 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52818.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52818.log.gz 2024-03-18T16:43:53.723 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49220.log 2024-03-18T16:43:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53382.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53382.log.gz 2024-03-18T16:43:53.724 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53172.log 2024-03-18T16:43:53.725 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49220.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49220.log.gz 2024-03-18T16:43:53.726 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58478.log 2024-03-18T16:43:53.732 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53172.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53172.log.gz 2024-03-18T16:43:53.732 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57644.log 2024-03-18T16:43:53.732 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58478.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58478.log.gz 2024-03-18T16:43:53.733 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44895.log 2024-03-18T16:43:53.733 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57644.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57644.log.gz 2024-03-18T16:43:53.733 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41343.log 2024-03-18T16:43:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44895.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44895.log.gz 2024-03-18T16:43:53.734 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47989.log 2024-03-18T16:43:53.735 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41343.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41343.log.gz 2024-03-18T16:43:53.736 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50642.log 2024-03-18T16:43:53.737 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47989.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47989.log.gz 2024-03-18T16:43:53.737 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44550.log 2024-03-18T16:43:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50642.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50642.log.gz 2024-03-18T16:43:53.739 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42266.log 2024-03-18T16:43:53.740 INFO:teuthology.orchestra.run.smithi102.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44550.log.gz 2024-03-18T16:43:53.741 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43203.log 2024-03-18T16:43:53.743 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45817.log 2024-03-18T16:43:53.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-18T16:43:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43203.log: 87.6%gzip -5 --verbose/var/log/ceph/ceph-client.admin.45817.log: -- /var/log/ceph/ceph-osd.0.log 2024-03-18T16:43:53.746 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43203.log.gz 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45817.log.gz 2024-03-18T16:43:53.746 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-18T16:43:53.747 INFO:teuthology.orchestra.run.smithi102.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.42266.log.gz 2024-03-18T16:43:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.0.log: -- /var/log/ceph/ceph-client.admin.48176.log 2024-03-18T16:43:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.48176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55135.log 2024-03-18T16:43:53.752 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48176.log.gz 2024-03-18T16:43:53.753 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59252.log 2024-03-18T16:43:53.756 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55135.log: gzip/var/log/ceph/ceph-client.admin.59252.log: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30752.log 2024-03-18T16:43:53.756 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59252.log.gz 2024-03-18T16:43:53.757 INFO:teuthology.orchestra.run.smithi102.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55135.log.gz 2024-03-18T16:43:53.759 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph.tmp-client.admin.30752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41115.log 2024-03-18T16:43:53.761 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41115.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30752.log.gz 2024-03-18T16:43:53.762 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44766.log 2024-03-18T16:43:53.764 INFO:teuthology.orchestra.run.smithi102.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41115.log.gz 2024-03-18T16:43:53.764 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44979.log 2024-03-18T16:43:53.766 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44766.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46693.log 2024-03-18T16:43:53.768 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44979.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44979.log.gz 2024-03-18T16:43:53.769 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46693.log: -- replaced with /var/log/ceph/ceph-client.admin.44766.log.gz 2024-03-18T16:43:53.771 INFO:teuthology.orchestra.run.smithi102.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46693.log.gz 2024-03-18T16:43:53.771 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50828.log 2024-03-18T16:43:53.774 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49514.log 2024-03-18T16:43:53.775 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50828.log.gz 2024-03-18T16:43:53.777 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53298.log 2024-03-18T16:43:53.779 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49514.log: gzip/var/log/ceph/ceph-client.admin.53298.log: 89.2% -5 --verbose -- /var/log/ceph/ceph-client.admin.46987.log 2024-03-18T16:43:53.779 INFO:teuthology.orchestra.run.smithi102.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53298.log.gz 2024-03-18T16:43:53.780 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49514.log.gz 2024-03-18T16:43:53.781 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52044.log 2024-03-18T16:43:53.785 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46987.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37055.log 2024-03-18T16:43:53.786 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52044.log: -- replaced with /var/log/ceph/ceph-client.admin.46987.log.gz 2024-03-18T16:43:53.787 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52044.log.gz 2024-03-18T16:43:53.789 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.37055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51690.log 2024-03-18T16:43:53.792 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51690.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37055.log.gz 2024-03-18T16:43:53.793 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47173.log 2024-03-18T16:43:53.794 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51690.log.gz 2024-03-18T16:43:53.795 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58562.log 2024-03-18T16:43:53.799 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58562.log: /var/log/ceph/ceph-client.admin.47173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55429.log 2024-03-18T16:43:53.800 INFO:teuthology.orchestra.run.smithi102.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55261.log 2024-03-18T16:43:53.801 INFO:teuthology.orchestra.run.smithi102.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47173.log.gz 2024-03-18T16:43:53.801 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58562.log.gz 2024-03-18T16:43:53.803 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55429.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55429.log.gz 2024-03-18T16:43:53.805 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56515.log 2024-03-18T16:43:53.806 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55261.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55261.log.gz 2024-03-18T16:43:53.809 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56515.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56515.log.gz 2024-03-18T16:43:53.809 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56785.log 2024-03-18T16:43:53.812 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56785.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56785.log.gz 2024-03-18T16:43:53.812 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54823.log 2024-03-18T16:43:53.814 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54823.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54823.log.gz 2024-03-18T16:43:53.815 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45357.log 2024-03-18T16:43:53.816 INFO:teuthology.orchestra.run.smithi154.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-18T16:43:53.818 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45021.log 2024-03-18T16:43:53.819 INFO:teuthology.orchestra.run.smithi102.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45357.log.gz 2024-03-18T16:43:53.821 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-18T16:43:53.825 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.45021.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46735.log 2024-03-18T16:43:53.825 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45021.log.gz 2024-03-18T16:43:53.826 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44444.log 2024-03-18T16:43:53.832 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44444.log: 88.7%/var/log/ceph/ceph-client.admin.46735.log: 88.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-18T16:43:53.833 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44444.log.gz 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46735.log.gz 2024-03-18T16:43:53.833 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-18T16:43:53.833 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44937.log 2024-03-18T16:43:53.835 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44937.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44937.log.gz 2024-03-18T16:43:53.836 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47089.log 2024-03-18T16:43:53.838 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49682.log 2024-03-18T16:43:53.839 INFO:teuthology.orchestra.run.smithi102.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47089.log.gz 2024-03-18T16:43:53.841 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53946.log 2024-03-18T16:43:53.845 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49682.log: /var/log/ceph/ceph-client.admin.53946.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58982.log 2024-03-18T16:43:53.845 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49682.log.gz 2024-03-18T16:43:53.847 INFO:teuthology.orchestra.run.smithi102.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53946.log.gz 2024-03-18T16:43:53.848 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48740.log 2024-03-18T16:43:53.852 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58982.log: /var/log/ceph/ceph-client.admin.48740.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51918.log 2024-03-18T16:43:53.852 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58982.log.gz 2024-03-18T16:43:53.853 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48740.log.gz 2024-03-18T16:43:53.854 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-18T16:43:53.856 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51918.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45273.log 2024-03-18T16:43:53.858 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51918.log.gz 2024-03-18T16:43:53.860 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45273.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45273.log.gz 2024-03-18T16:43:53.860 INFO:teuthology.orchestra.run.smithi102.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47653.log 2024-03-18T16:43:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50870.log 2024-03-18T16:43:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50870.log: /var/log/ceph/ceph-client.admin.47653.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47653.log.gz 87.7% 2024-03-18T16:43:53.867 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50870.log.gz 2024-03-18T16:43:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43790.log 2024-03-18T16:43:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-18T16:43:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43790.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43790.log.gz 2024-03-18T16:43:53.876 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47341.log 2024-03-18T16:43:53.879 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.47341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57517.log 2024-03-18T16:43:53.879 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47341.log.gz 2024-03-18T16:43:53.883 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59504.log 2024-03-18T16:43:53.886 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.57517.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57517.log.gz 2024-03-18T16:43:53.889 INFO:teuthology.orchestra.run.smithi102.stderr: 90.7%/var/log/ceph/ceph-client.admin.59504.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59504.log.gz 2024-03-18T16:43:53.889 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46213.log 2024-03-18T16:43:53.890 INFO:teuthology.orchestra.run.smithi102.stderr: -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-18T16:43:53.890 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46567.log 2024-03-18T16:43:53.896 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46213.log: /var/log/ceph/ceph-client.admin.46567.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46213.log.gz 2024-03-18T16:43:53.896 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46567.log.gz 2024-03-18T16:43:53.896 INFO:teuthology.orchestra.run.smithi102.stderr: /var/log/ceph/ceph-client.admin.49556.log 2024-03-18T16:43:53.899 INFO:teuthology.orchestra.run.smithi154.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-18T16:43:53.901 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50474.log 2024-03-18T16:43:53.902 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49556.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49556.log.gz 2024-03-18T16:43:53.903 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46609.log 2024-03-18T16:43:53.904 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50474.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50474.log.gz 2024-03-18T16:43:53.906 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46609.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46609.log.gz 2024-03-18T16:43:53.908 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55993.log 2024-03-18T16:43:53.910 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52338.log 2024-03-18T16:43:53.911 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54739.log 2024-03-18T16:43:53.911 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55993.log.gz 2024-03-18T16:43:53.913 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52338.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52338.log.gz 2024-03-18T16:43:53.914 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47611.log 2024-03-18T16:43:53.919 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54739.log: gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54739.log.gz 2024-03-18T16:43:53.919 INFO:teuthology.orchestra.run.smithi102.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49472.log 2024-03-18T16:43:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47611.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47611.log.gz 2024-03-18T16:43:53.922 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42998.log 2024-03-18T16:43:53.923 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49472.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49472.log.gz 2024-03-18T16:43:53.928 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42998.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42998.log.gz 2024-03-18T16:43:53.929 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52608.log 2024-03-18T16:43:53.934 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44467.log 2024-03-18T16:43:53.937 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53256.log 2024-03-18T16:43:53.937 INFO:teuthology.orchestra.run.smithi102.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52608.log.gz 2024-03-18T16:43:53.941 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53256.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53256.log.gz 2024-03-18T16:43:53.941 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44467.log.gz 2024-03-18T16:43:53.941 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47905.log 2024-03-18T16:43:53.947 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54427.log 2024-03-18T16:43:53.948 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47905.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47905.log.gz 2024-03-18T16:43:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54613.log 2024-03-18T16:43:53.952 INFO:teuthology.orchestra.run.smithi102.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54427.log.gz 2024-03-18T16:43:53.954 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46045.log 2024-03-18T16:43:53.956 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54613.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54613.log.gz 2024-03-18T16:43:53.961 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54343.log 2024-03-18T16:43:53.962 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46045.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46045.log.gz 2024-03-18T16:43:53.964 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53484.log 2024-03-18T16:43:53.964 INFO:teuthology.orchestra.run.smithi102.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54343.log.gz 2024-03-18T16:43:53.966 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46087.log 2024-03-18T16:43:53.969 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53484.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53484.log.gz 2024-03-18T16:43:53.971 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54697.log 2024-03-18T16:43:53.971 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46087.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46087.log.gz 2024-03-18T16:43:53.976 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.54697.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54697.log.gz 2024-03-18T16:43:53.978 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50204.log 2024-03-18T16:43:53.980 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51435.log 2024-03-18T16:43:53.982 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50204.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50204.log.gz 2024-03-18T16:43:53.986 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58730.log 2024-03-18T16:43:53.988 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51435.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51435.log.gz 2024-03-18T16:43:53.990 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49304.log 2024-03-18T16:43:53.994 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58730.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58730.log.gz 2024-03-18T16:43:53.994 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45733.log 2024-03-18T16:43:53.994 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49304.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49304.log.gz 2024-03-18T16:43:53.995 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56389.log 2024-03-18T16:43:53.996 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.45733.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45733.log.gz 2024-03-18T16:43:53.997 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47737.log 2024-03-18T16:43:53.999 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56389.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56389.log.gz 2024-03-18T16:43:54.002 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47737.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47737.log.gz 2024-03-18T16:43:54.002 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56701.log 2024-03-18T16:43:54.006 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58352.log 2024-03-18T16:43:54.007 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56701.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56701.log.gz 2024-03-18T16:43:54.011 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-03-18T16:43:54.012 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58352.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58352.log.gz 2024-03-18T16:43:54.015 INFO:teuthology.orchestra.run.smithi102.stderr:gzip/var/log/ceph/ceph-client.admin.41637.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50600.log 2024-03-18T16:43:54.015 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2024-03-18T16:43:54.018 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56347.log 2024-03-18T16:43:54.019 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50600.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50600.log.gz 2024-03-18T16:43:54.023 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44429.log 2024-03-18T16:43:54.023 INFO:teuthology.orchestra.run.smithi102.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56347.log.gz 2024-03-18T16:43:54.026 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42617.log 2024-03-18T16:43:54.027 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.44429.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44429.log.gz 2024-03-18T16:43:54.031 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42617.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42617.log.gz 2024-03-18T16:43:54.032 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43296.log 2024-03-18T16:43:54.034 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43786.log 2024-03-18T16:43:54.036 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43296.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43296.log.gz 2024-03-18T16:43:54.036 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43422.log 2024-03-18T16:43:54.038 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43786.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43786.log.gz 2024-03-18T16:43:54.040 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42533.log 2024-03-18T16:43:54.042 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43422.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43422.log.gz 2024-03-18T16:43:54.044 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41301.log 2024-03-18T16:43:54.046 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.42533.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42533.log.gz 2024-03-18T16:43:54.048 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.41301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53130.log 2024-03-18T16:43:54.050 INFO:teuthology.orchestra.run.smithi102.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41301.log.gz 2024-03-18T16:43:54.050 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59420.log 2024-03-18T16:43:54.052 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53130.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53130.log.gz 2024-03-18T16:43:54.055 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59420.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59420.log.gz 2024-03-18T16:43:54.056 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49994.log 2024-03-18T16:43:54.057 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46423.log 2024-03-18T16:43:54.059 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49994.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49994.log.gz 2024-03-18T16:43:54.060 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53004.log 2024-03-18T16:43:54.062 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.46423.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46423.log.gz 2024-03-18T16:43:54.068 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55033.log 2024-03-18T16:43:54.069 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53004.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53004.log.gz 2024-03-18T16:43:54.070 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50162.log 2024-03-18T16:43:54.072 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.55033.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55033.log.gz 2024-03-18T16:43:54.076 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52962.log 2024-03-18T16:43:54.077 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.50162.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50162.log.gz 2024-03-18T16:43:54.079 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49598.log 2024-03-18T16:43:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.52962.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52962.log.gz 2024-03-18T16:43:54.084 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49598.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49598.log.gz 2024-03-18T16:43:54.085 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53652.log 2024-03-18T16:43:54.088 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-18T16:43:54.090 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53652.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53652.log.gz 2024-03-18T16:43:54.092 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49640.log 2024-03-18T16:43:54.095 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56995.log 2024-03-18T16:43:54.097 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.49640.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49640.log.gz 2024-03-18T16:43:54.101 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51207.log 2024-03-18T16:43:54.101 INFO:teuthology.orchestra.run.smithi102.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-18T16:43:54.102 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56995.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56995.log.gz 2024-03-18T16:43:54.106 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.51207.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51207.log.gz 2024-03-18T16:43:54.108 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47299.log 2024-03-18T16:43:54.110 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56743.log 2024-03-18T16:43:54.112 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.47299.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47299.log.gz 2024-03-18T16:43:54.115 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43611.log 2024-03-18T16:43:54.117 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.56743.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56743.log.gz 2024-03-18T16:43:54.119 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59588.log 2024-03-18T16:43:54.120 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.43611.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43611.log.gz 2024-03-18T16:43:54.122 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59378.log 2024-03-18T16:43:54.123 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59588.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59588.log.gz 2024-03-18T16:43:54.126 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58688.log 2024-03-18T16:43:54.127 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.59378.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59378.log.gz 2024-03-18T16:43:54.129 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.58688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53424.log 2024-03-18T16:43:54.132 INFO:teuthology.orchestra.run.smithi102.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58688.log.gz 2024-03-18T16:43:54.133 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/ceph-client.admin.53424.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53424.log.gz 2024-03-18T16:43:54.362 INFO:teuthology.orchestra.run.smithi102.stderr: 89.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-18T16:43:54.469 INFO:teuthology.orchestra.run.smithi154.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-18T16:43:54.650 INFO:teuthology.orchestra.run.smithi154.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-18T16:43:54.669 INFO:teuthology.orchestra.run.smithi154.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-18T16:43:54.683 INFO:teuthology.orchestra.run.smithi154.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-18T16:43:54.686 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-03-18T16:43:54.686 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m1.455s 2024-03-18T16:43:54.686 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m6.499s 2024-03-18T16:43:54.686 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m0.224s 2024-03-18T16:43:54.989 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-18T16:43:55.091 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-18T16:43:55.170 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-18T16:43:55.183 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-18T16:43:55.186 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-18T16:43:55.186 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m1.973s 2024-03-18T16:43:55.186 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m6.344s 2024-03-18T16:43:55.186 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m1.747s 2024-03-18T16:43:55.187 INFO:tasks.ceph:Archiving logs... 2024-03-18T16:43:55.187 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/remote/smithi102/log 2024-03-18T16:43:55.188 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T16:43:56.225 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/remote/smithi154/log 2024-03-18T16:43:56.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T16:43:56.993 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-18T16:44:00.233 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=78d4acaaeac447b2a99b11ba75235c34 Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-18T16:44:00.237 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-18T16:44:00.246 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T16:44:00.283 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T16:44:00.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T16:44:00.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T16:44:00.413 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T16:44:00.421 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T16:44:00.421 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:44:00.429 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:44:00.443 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:44:00.448 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:44:00.448 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-03-18T16:44:00.448 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 +2696us[+2689us] +/- 91ms 2024-03-18T16:44:00.448 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv02.front.sepia.ceph.com 3 7 377 10 -1185us[-1192us] +/- 58ms 2024-03-18T16:44:00.448 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 -518us[ -525us] +/- 84ms 2024-03-18T16:44:00.448 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:44:00.462 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:44:00.469 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:44:00.469 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-18T16:44:00.469 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +2707us[+2707us] +/- 91ms 2024-03-18T16:44:00.469 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -1154us[-1154us] +/- 58ms 2024-03-18T16:44:00.469 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 -497us[ -497us] +/- 84ms 2024-03-18T16:44:00.469 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:44:00.470 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T16:44:00.479 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T16:44:00.479 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T16:44:00.509 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T16:44:00.659 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T16:44:00.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T16:44:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:00.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T16:44:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:00.833 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T16:44:00.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T16:44:00.892 INFO:teuthology.task.internal:Duration was 2762.765422 seconds 2024-03-18T16:44:00.893 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T16:44:00.942 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T16:44:00.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T16:44:00.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T16:44:00.993 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T16:44:01.055 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T16:44:01.435 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T16:44:01.435 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-03-18T16:44:01.436 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T16:44:01.498 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-03-18T16:44:01.498 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T16:44:01.523 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T16:44:01.524 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T16:44:01.529 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T16:44:01.608 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T16:44:01.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T16:44:01.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T16:44:01.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T16:44:01.793 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T16:44:01.793 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T16:44:01.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T16:44:01.888 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T16:44:01.898 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:01.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:01.953 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-03-18T16:44:01.983 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-03-18T16:44:02.023 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:02.059 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:02.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T16:44:02.083 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T16:44:02.084 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/remote/smithi102 2024-03-18T16:44:02.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T16:44:02.178 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908/remote/smithi154 2024-03-18T16:44:02.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T16:44:02.213 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T16:44:02.213 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T16:44:02.219 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T16:44:02.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T16:44:02.272 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T16:44:02.272 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T16:44:02.293 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T16:44:02.293 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T16:44:02.300 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T16:44:02.315 INFO:teuthology.orchestra.run.smithi154.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 16:44 /home/ubuntu/cephtest 2024-03-18T16:44:02.334 INFO:teuthology.orchestra.run.smithi102.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 16:44 /home/ubuntu/cephtest 2024-03-18T16:44:02.336 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T16:44:02.346 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T16:44:02.444 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T16:44:02.470 DEBUG:teuthology.nuke:shortname: smithi102 2024-03-18T16:44:02.470 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T16:44:02.492 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:48:14.324969', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCIVywnGKMz4dbv6DuAZKnwUEr2ps9arusWyrnGMF6hbhtInfttAU9qNgdINIypmvSPQBuV58PzDs5kqECCYahA='} 2024-03-18T16:44:02.496 DEBUG:teuthology.nuke:shortname: smithi154 2024-03-18T16:44:02.496 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T16:44:02.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608908', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:48:14.326898', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUN4kDHCkrh0k3Tv2BDt0sRWbFTJMwli02mlbrynLiEHbZLewUIAqhOS4zfPYrE85fDkkH6Malb9EVm7+BSETI='} 2024-03-18T16:44:02.541 INFO:teuthology.orchestra.console.smithi102:Power off 2024-03-18T16:44:02.541 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T16:44:02.565 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-03-18T16:44:02.566 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:44:02.581 INFO:teuthology.orchestra.console.smithi154:Power off 2024-03-18T16:44:02.581 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T16:44:02.590 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-18T16:44:02.605 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2024-03-18T16:44:02.605 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:44:02.632 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-18T16:44:06.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-18T16:44:06.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-18T16:44:06.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2024-03-18T16:44:06.591 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:44:06.633 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:44:06.703 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-18T16:44:06.743 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-18T16:44:10.706 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:44:10.744 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:44:10.818 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-03-18T16:44:10.854 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2024-03-18T16:44:10.919 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-03-18T16:44:10.955 INFO:teuthology.orchestra.console.smithi154:Power off completed 2024-03-18T16:44:11.060 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3845b7ac715a3ff198582638d199a90e30e254da description: fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} duration: 2762.765422105789 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=78d4acaaeac447b2a99b11ba75235c34 status: fail success: false 2024-03-18T16:44:11.060 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T16:44:11.142 INFO:teuthology.run:FAIL